07: Introducing sed

To this point you've been using grep (and ugrep on Windows) to only search for patterns in text. This is useful but it'd be more useful if you could replace patterns with other text. In this exercise you'll learn to use the command line tool sed to use regex patterns to "search and replace."

What Is Sed

The tool sed stands for "stream editor" and it's purpose is to input a "stream" of text (usually a file), scan each line for a regex pattern, and then replace any matches with alternative text.

The simplest way to run sed is like this:

sed s/pattern/replace/g

If you run this command in your Terminal you can actually enter text, hit ENTER and sed will do its job:

$ sed s/pattern/replace/g
This is a pattern
This is a replace
That I pattern
That I replace
Nobody needs to pattern
Nobody needs to replace

To exit this hold ctrl and hit d so ctrl-d.

The first line is one I typed, then next line is the one sed prints and you can see it replaced pattern with replace. This continues with each line, and be sure you understand I only type every odd line, sed outputs the even ones.

The format of this sed command is like this:

s

substitute this is the sed command to run.

/

bounding character This starts the regex, and you can actually use any character so another popular one is ,. You need this because sometimes the text you're searching has / in it so as long as each bounding character matches you can use what you want.

pattern

regex You can use any regex here to match in lines.

/

bounding Again, this is your bounding character, and if you chose , you'd use that instead.

replace

replacement This is what you want the regex to be replaced with.

/

bounding One more time to end the pattern, but you need one more little thing.

g

modifiers You can add special modifiers to your s// command, and this one says "global" which means replace all occurrences on each line. If you don't have this sed will only replace the first occurrence of pattern.

Be sure you understand how this works and try a few more of your own. The more time you spend playing with basic input and regex the more prepared you'll be for later.

Replacing in Files

For this part we'll need a new file to work with so here's a snippet from hipster ipsum:

Biodiesel irony subway tile banh mi kombucha, whatever praxis selvage chia drinking vinegar pickled. Direct trade paleo thundercats knausgaard swag. Fit kinfolk same trust fund, helvetica fingerstache master cleanse pinterest flannel Brooklyn tbh keytar. Cold-pressed actually marxism, whatever narwhal celiac lo-fi blackbird spyplane air plant Brooklyn letterpress irony beard crucifix cred. IPhone pabst four loko thundercats drinking vinegar. Hoodie master cleanse paleo jean shorts chicharrones chia gastropub chambray freegan literally williamsburg marfa asymmetrical la croix.

Feel free to grab more text from the site, but for this one we'll start replacing some of the text. Place it in a text file named ex07.txt

You already know how to do the majority of regex you'll need, so now you just have to practice running them on this text file. To do that simply add the file's name as a 2nd parameter like this:

$ sed "s/subway/NYC BABY/g" ex07.txt

This will replace every occurrence of "subway" with "NYC BABY". You'll notice that I put the entire command in " (double-quote) characters. This is to "protect" it from the shell. If you don't then the shell reads the space in "NYC BABY" as separating a parameter, so it thinks you typed:

sed
s/subway/NYC
BABY/g
ex07.txt

You should always add " around your command to make sure it's interpreted as a single argument to sed.

Challenge Time

You'll now complete some challenges to learn how to use sed. You can use any file but I'll be using the sample above to describe the challenges:

1. Replace very capital letter with the number 8.
2. Replace all "ing" with nothing. See if you can figure out how to do this. Basically, "delete ing."
3. Replace "Brooklyn" with your home town or favorite city. If your favorite city is Brooklyn then you are now required to visit the Brooklyn Museum the next time they show John Singer Sargent's watercolor collection. Bonus points if you go when they have the entire collection including the paintings from Boston. You should also read the story about why the paintings are split between the two cities. It's hilarious.
4. Fix the name of IPhone.
5. Remove any word ending in "ar".
6. Remove all the spaces.
7. Replace all the spaces with a newline. You can use \n in your replace to create a newline.

You should spend some time doing more practice with this before moving on.

Further Study

• Find out what ed is and how it's related to sed.
• Read through the sed manual. Keep in mind this is the GNU sed instructions so if you're on OSX your sed is different.
• Practice using bounding chars different from /.