Linux Essentials: Sed (Stream Editor)

It's sometimes said that find, sed and awk are all you need when it comes to Linux command-line utilities. The last two are Turing complete, just like any other programming language. Having already covered grep, we move on to its stream-wrangling cousin, sed. Its purpose is filtering and transforming streams of text, which is a very loose and general concept in Linux. However, some of its functionality might not be all that obvious without reading the entire manual, but that's where this article jumps into the picture. Fire up your terminal and type along.

To start working on a file, you can pipe (send) it into sed:

cat example.txt | sed s/test/Test/

or have it operate directly on a file:

sed s/test/Test/ example.txt

And if you'd like to save changes in place (as sed outputs to stdout by default), you'd add the -i switch:

sed -i s/TEST/Test/ example.txt

The operation we're performing up there is called string substitution, and the letter s stands for "substitute". The first string is the one we'd like to substitute, and the second one is what we're substituting it with. The forward slashes are a separator, and though you may use a different one (for example, you could write sed s#one#1# example.file), it'll be easier if you stick to this convention.

If you want to substitute more than one word, you have to use quotes:

sed 's/cowardly old world/brave new world/' example2.txt

This is quite useful by itself, but the real power here lies in using regular expressions to perform substitutions. However, they can be fairly arcane, and easily deserve an article of their own, so we won't cover them in depth here. Suffice to say they consist of regular and meta characters, and if you wish to use meta characters as normal characters, you have to escape them using the backslash (\).

For example, to capitalize the letter A, but only if it's the beginning of a line:

sed 's/^a/A/' example3.txt

But if you'd like to replace the ^ character, you'd have to escape it:

sed 's/\^/caret/' example4.txt

Another important thing to note about substitution is that it operates on the first occurrence of the string in the line. So if you'd like to replace, let's say, the third one:

sed 's/\^/caret/3' example4.txt

If you wish to replace all of them, replace 3 with g (stands for global) in the example above.

Substitution is merely one out of twenty available sed commands, and though it is by far the most commonly used one, it's not the only useful one. Let's take a look at how deleting works.

Deleting a line with a certain pattern in it:

sed '/delete me/d'

Deleting all commented lines (lines beginning with # are comments in Bash scripts):

sed '/^\#/d'

To delete lines 4 through 9, type:

sed '4,9d'

As you have seen, you can use sed to perform various operations on the text - replace, delete or add content. Just like grep, it's available by default on nearly every Linux distribution, and there's even a Windows port (version). However, everything shown here is only a fraction of sed's true power, as it can be used to modify files in many different ways, even edit multiple files at once. If you'd like to pursue mastery of this powerful tool, there are many tutorials and FAQs available on the Web.

 

Image credit: 1