sed is an extremely powerful stream editor. The web is full of useful sed one liners.

1. Sed Command Cheat Sheet
2. Sed Tutorial Pdf
3. Linux Sed Cheat Sheet
4. Ed Text Editor
5. Sed Command Line Cheat Sheet

Getting started Introduction. This is a quick reference to getting started with Bash scripting. Learn bash in y minutes (learnxinyminutes.com); Bash Guide (mywiki.wooledge.org).

This post will be an ever growing list of sed one liners I have found useful.

Find and Replace

Find and remove all instances of TEXT:

1. The full form of `sed` is Steam Editor, and it is mainly used to read and convert the text in different ways by using a regular expression. How this command can be used to replace the multiple lines of a file in different ways are explained in this tutorial. Commonly used `sed` Cheat Sheet.
2. FILE SPACING: # double space a file sed G # double space a file which already has blank lines in it. Output file # should contain no more than one blank line between lines of text. Sed ‘/^$/d;G’ # triple space a file sed ‘G;G’ # undo double-spacing (assumes even-numbered lines are always blank) sed Continue reading Sed cheat sheet / one-line sed.
3. Sed Cheat Sheet Edit Cheat Sheet. Advanced use of sed In-place Editing. To edit file use the -i option this safely changes the file contents without any output.

Find and remove all spaces:

Delete all blank lines from a file

Delete leading whitespace from front of each line

Delete trailing whitespace from end of each line

Delete leading and trailing whitespace from each line

Insert a line of text before a line

Change configuration file values

Change BRIDGE_HOTPLUG=yes to BRIDGE_HOTPLUG=no no matter what it is already set to:

Change PermitRootLogin no to PermitRootLogin yes no matter what it is already set to:

Wrap Value in Double Quotes

Change date=2015 to date='2015”:

Remove word wrap

Display links between href tags

Lower case all href=”*” tags

Lower case all img src=”*” tags

Lower case all href=”” tags but exclude href='http” tags

Lower case all img src=”” tags but exclude img src='http” tags

Remove ‘%20’ only from lines matching href=”*”

References

by Alex Harvey

I enjoyed reading the GNU AWK manual so much that I decided to read the GNU sed manual too. Full disclaimer. sed is harder. Much harder!

This post began as a companion to my earlier post, Observations of AWK, and ended up as a full tutorial and reference.

• Command-line options
• The structure of a sed script
• Sed commands
  • The s command (substitute)
• Grouping commands
• Address specifications
• Using the hold space
• Multiline techniques
• Branching and flow control
• The GNU sed debugger

Conventions

Throughout this post, I use sed in examples for a script that should work in any version of sed, such as the BSD sed that ships with Mac OS X; and I use gsed where GNU sed is actually required.¹ All sed scripts are unquoted unless single or double quotes are reqired to protect characters from interpretation by the shell.

Scope and target audience

In this post, I cover most of the features of GNU sed, nearly all of its commands, but not regular expressions, only the command-line options I consider useful, and my treatment of branching and flow control is cursory. The target audience is a Bash programmer who knows the basics of sed and wants to learn the rest.

What is sed

Sed stands for stream editor. It is used to perform transformations on an input stream, either a file or input from a pipeline. It was written in 1973-4 by Lee E. McMahon and first appeared in Unix version 7.

When to use sed

According to the sed FAQ, you should use sed when you need a small, fast program to modify words, lines, or blocks of lines in a text file. Conversely, you should not use sed if grep, tr, awk, or perl do the job better. Use the right tool for the job. Note that almost any sed one-liner can be rewritten as a Perl one-liner, although the question of Bash idiom and performance should be considered. That is, these have the same effect:

and:

But the sed version may be more idiomatic, easier to remember, easier to read and write - and also faster.

Command-line options

Command-line options summary

GNU sed has a lot of command-line options. Most of them I have never used, and after carefully reading the manual, the following table lists the only ones I consider useful.

For the remainder of the command-line options, see the manual.

Note about -e

I see shell scripts everywhere that specify the -e option, and, I think, most of the time unnecessarily, and probably because almost no one understands what the -e option is really for. Perhaps this is because the documentation is not all that clear. The man page says:

-e script
–expression=script

Add the commands in script to the set of commands to be run while processing the input.

In fact, the following commands are identical:

So, most of the time, you do not actually need to specify -e. In particular, there is never a good reason to specify just one -e; such code can and should always be refactored to remove the redundant -e. Use of -e makes sense when splitting up a command into multiple sections via multiple -e improves readability.

Note about -i

The -i option allows editing of files in-place.

Be aware that in GNU sed only, the suffix passed to -i is optional. In BSD sed, to avoid a backup file being created, it is necessary to explicitly pass an empty string to -i. Thus:

BSD sed

Is the same on BSD as:

GNU sed

The structure of a sed script

The structure of a command

All sed commands have the following basic structure:

Where:

• [ADDR] is an optional address specification
• X is a single-letter command
• [OPTIONS] are optional options (or flags) accepted by the command.

The single-letter commands are called functions in the BSD manual. [ADDR] is an optional address specification. If [ADDR] is specified, the command X will be executed only for lines matching [ADDR]. [ADDR] can be a single line number, a regular expression, or a range of lines. There is more on this below.

The structure of a script

A sed script is a sequence of commands:

Note that instead of single-letter sed commands, commands can be grouped instead, using { X; Y; Z }:

Example 1

The first example is one of the most familiar:

In that case, the optional [ADDR] is not used, the s/// command follows, with a regexp alexander, a replacement alex and for [OPTIONS], g. This script replaces all occurences of the string alexander with alex.

Example 2

Another example:

Here the [ADDR] is 1,1000, which specifies lines 1 to 1000; the single-letter command is d (delete); and there are no [OPTIONS]. This script would delete the first 1000 lines from a log file.

Example 3

An example with a block:

This time the [ADDR] is /roo/ meaning any line matching /roo/, and instead of a command there is a block {p;p;q;} - print, print, quit. So this script finds the first line matching /roo/, prints it twice, and then exits.

Sed Command Cheat Sheet

Under the hood

So what is sed doing under the hood?

Sed operates by performing a cycle on each line of a file: first, it reads one line from the input stream, removes any trailing newline, and places it in the pattern space (see below). The commands are then executed. The optional [ADDR] (address specification) is a kind of conditional; the command is only executed if the condition is verified before the command is to be executed.

When the end of the script is reached, unless the -n option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed. Then the next cycle starts for the next input line.

The pattern space and hold space

Sed maintains two buffers: the pattern space and the hold space. Both are initially empty.

The pattern space buffers each line that is read in from the input stream, although the N command can grow the pattern space by additional input lines. Unless special commands like D are used, the pattern space is deleted between cycles.

The hold space, on the other hand, keeps its data between cycles. I’ll have more to say about the hold space (see below), but, for now, just remember that there are two buffers and remember their names.

Sed commands

Overview

Understanding sed, in my opinion, requires learning nearly all of its single-letter commands. The GNU sed manual groups the commands in this way:

• s/// - the “s” command.
• q, d, p, n - “commonly-used” commands.
• y///, a, i, c, =, l, r, w, d, n, P, h, H, g, G, x - “less frequently-used” commands.
• :, b, t - “commands for sed gurus”.
• e, F, Q, R, T, v, W, z - GNU sed extensions.

A troubling implication of this breakdown, of course, is that learning GNU sed at all beyond the “commonly-used” commands - and it is beyond these that the difficulty lies - might be a waste of time, in the sense that any sed program requiring the others could be better written in AWK or Perl.

In this post, however, I have used a different breakdown:

• =, a, c, d, e, i, l, n, p, q, Q, s///, y///, z - “easy to use” commands, and I have included discussion and examples of their usage in this section.
• :, b, h, H, g, G, P, t, T, x - commands for multiline techniques, the hold space, branching, and flow control, all are for sed gurus, and with examples in subsequent sections.
• F, r, R, v, w, W - commands that are mostly GNU extensions and that I don’t regard as sufficiently useful to discuss in this post.

Finally, it can be seen that sed is a tiny language. As initimidating as all of its cryptic commands are, it is actually easy to learn.

Sed command cheat sheet

This section presents a cheat sheet, and when I say “cheat sheet”, I mean every command in sed in alphabetical order, mostly lifted word for word from the docs.

The s command (substitute)

Overview

Some say the s and p commands are the only commands that sed should ever be used for. I disagree with those people but I thought I should mention it.

The s command has the form:

The s command attempts to match the pattern space against the supplied regular expression REGEXP; if the match is successful, then the portion of the pattern space that matched is replaced with REPLACEMENT.

Using a different delimiter

If the regular expression itself contains the / character, it is typical to use a different delimiter. Sed accepts any character as a replacement delimiter. For example:

Which would change everyone’s shell to tcsh, because, why not?

Back references

The replacement can contain back references, 1, 2 .. 9. (If you need 10 or more back references, you should consider using Perl².)

For example:

Back reference &

Also, the replacement can contain unescaped & characters which reference the whole matched portion of the pattern space.

For example, double all spaces:

Case conversions (GNU only)

In GNU sed there are extensions for converting text to upper and lower case:

For example, convert all instances of a string to uppercase:

For another example, convert only the first letter of a string to uppercase:

Flags

Here is the full list of flags accepted by s///:

Example 1

Here are some examples. Replace all instances of a pattern from the second onwards:

Example 2

Print only lines where replacements are made:

The q and Q commands (quit)

The q and Q commands are useful if for whatever reason you want sed to quit and stop printing. Some examples:

Emulate the head command:

The difference between q and Q is that q prints the line then quits, whereas Q quits without printing. Here’s an example of Q:

Print all lines between (the first instance of) 2 patterns, exclusive of the patterns:

The d command (delete)

The d command deletes the pattern space, and also immediately begins the next cycle.

Some examples:

Delete all lines in a file matching a pattern:

Delete the first 10 lines in a file:

There is more on the d command in relation to its branching behaviour below.

The p command (print)

The p command prints out the pattern space to STDOUT. The p command is mostly only used in conjunction with the -n option to sed, because, otherwise, printing each line is sed’s default behaviour.

Using p and -n is another way to emulate the head command. Print only lines 1 to 3:

The n command (next)

The n command behaves differently depending on whether -n is enabled:

• If -n is specified, just replace the pattern space with the next line of input.
• If -n is not specified, print the pattern space, then replace it with the next line of input.

Some examples. Emulate grep -A2 3:

Print only the line after a line matching 3:

Perform a substitution only every 3rd line:

The y command

The y/SRC/DST/ is occasionally used and occasionally useful. It transliterates characters in the pattern space which match any of the SRC with the corresponding character in DST. For example, convert all upper case characters to lower case:

The i, a and c commands (insert, append, change)

Using i, a and c we can insert before, append after, and replace (change) lines matching patterns or otherwise satisfying a condition. For example, given a file:

Replace XX with the missing lines of the pattern, aa, bb, cc, dd, ee.

The l command

The l command prints text in an unambiguous way, revealing hidden and unprintable characters:

Compare to:

Using a GNU extension a line wrap can also be specified:

The = command

The = command can be used to print the line numbers. A bit like an alternative to cat -n:

The z command (zap)

While not often required, it’s also not terribly complicated. The z (zap) command, in GNU sed only, can be used as a more reliable and efficient alternative to s/.*//, to simply empty the pattern space.

The e command (exec)

While I don’t necessarily recommend doing this, the GNU sed e command can be used to pipe the pattern space into an external Unix command. For example, print lines after a replacement, piping the response into the Unix column command:

Meanwhile, the e command with an optional command following it simply executes that command and sends it output into the output stream. I’ll update this post if I ever find a use for it!

Grouping commands

Brace notation

Commands can be grouped in a block as in other programming languages.

For example, perform substitution then print the second input line:

Or a real life example, suppose I have a script:

And I want to update the name of the variable foo, but only inside the function myfunction only, and print the new version of the function after the substitutions. I can do this:

Which is a bit more readable written out like this:

A note about semi-colons

Grouped sed commands can be separated on a single line by the semi-colon ; or separated on multiple lines where the semi-colon is not required. That is:

Is the same as:

Is the same as:

Be aware that BSD sed and GNU sed have different syntactic requirements inside a block. This script:

Works fine, but in BSD sed, an error is emitted:

To write code that works in both GNU and BSD sed, terminate the last character in a group with a semi-colon, if followed by a closing brace:

A note about comments

As already seen in passing, sed scripts can be commented using # as in most other languages.

Address specifications

This section lists all the ways you can select lines in sed.

Select by number

One way is to simply specify the actual line number you want in a file. For example, delete the second line from a file:

Select the last line

The last line is specified as $, as in Vim. Delete the last line in a file:

Select every second, third etc line

A GNU extension, for the sake of completeness, allows you to select consecutive lines. For example, print even numbered lines:

Print odd numbered lines:

Print every third line:

Select lines matching a pattern

To select lines matching a pattern, use /regexp/. For example, print lines beginning with 1:

To select by a pattern using a different delimiter, use %regexp%, !regexp!. For example, find all files in /usr/local matching /usr/local/Cellar:

Select lines by range

Lines can be selected by a range. For example, to print the first 3 lines of a stream:

To select from the 8th to end of file:

To select between two patterns inclusive of the patterns:

Select from a line or pattern to an offset

In GNU sed only, it is also possible to select an offset. For example, to select the next 2 lines after a line matching a pattern:

Select from a line to the next line divisible by N

Not that I have any idea why a feature like this would exist, it is also possible to select from a line matching a pattern to a line number divisible by N. For example:

Negation !

Appending the ! character to the end of an address specification (before the command letter) negates the sense of the match. That is, if the ! character follows an address or an address range, then only lines which do not match the addresses will be selected.

For example, delete all lines other than the last one:

Exclude all lines between two patterns:

Using the hold space

The h, H (hold), g, G (get), and x (exchange) commands

If you have made it this far, congratulations! But be warned, beyond this point is seriously into the territory of where you should consider other programming languages, most of the time.

Recall that sed has two buffers, the pattern space and the hold space. Both are initially empty. The following commands manipulate the hold space:

There is also more on the H and G commands in the multiline section below.

Example 1

This is a classic sed one-liner, to double-space a file. It’s very simple:

Because the hold space is initially empty, the G command appends a newline followed by the contents of hold buffer to pattern space. Thus, this one character script just adds a newline before every line.

Example 2

One example is from the O’Reilly sed and AWK book. Suppose we have a file:

And we want to reverse the order of the lines beginning with 1 and the lines beginning with 2. This script will do it:

Testing:

Multiline techniques

The D, G, H, N and P commands

Multiple lines can be processed using the capital letter commands D, G, H, N, P. These are all similar to their corresponding lowercase commands d, g, h, n, p except that they also respect newlines in the strings, allowing manipulation of multiline patterns in the pattern and hold spaces.

They operate as follows:

Example 1

Here is a contrived example from the docs to illustrate N and D:

Testing:

Example 2

Print paragraphs only if they match a pattern. A paragraph here is a sequence of lines that aren’t empty. This script is a solution:

Testing:

Example 3

Delete preceding line and line matching a pattern:

Branching and flow control

The :, b, t and T commands

Seriously, don’t do this. But if you really want to, read on.

The following table lists all of sed’s flow control contructs:

Note well that some of the basic commands like d and D also have side effects that alter the program flow. This can be confusing at first. Notice also that an address specification is like an if/then and that s/// in conjunction with t and T also can conditionally control flow.

But some examples will have to suffice.

Example 1

This is a classic sed script that sets up a sliding window to emulate tail -5. It illustrates use of N and D, and also a loop using : and b.:

In this way, the script maintains a stable buffer of the last 5 lines throughout all cycles. Also, and confusingly, the script depends on a GNU-specific feature of the N command, as documented here:

Most versions of sed exit without printing anything when the N command is issued on the last line of a file. GNU sed prints pattern space before exiting unless of course the -n command switch has been specified. This choice is by design.

Testing:

Note that the script doesn’t work if POSIX-conforming behaviour is requested:

(No output.)

Example 2

Another classic example illustrating branching and the P command: Append a line to the previous one if it starts with “=”:

Testing:

This code is further explained here.

Example 3

Implementing s///3g on BSD or other non-GNU sed. This illustrates use of t again.

Testing:

Note that BSD sed, unlike GNU sed, requires each label to be line-break terminated thus the requirement to use -e.

This version also makes it easier to understand how it works:

The GNU sed debugger

GNU sed has a debugger, activated by running the script with --debug.

Example program

Suppose I want to debug this very simple script:

Debugger output

With the debugger on, I see this:

As can be seen, the debugger tells us everything that happens in each cycle.

At the beginning, we are told what the sed program itself is:

Sed Tutorial Pdf

Cycle 1

In cycle 1 (i.e. the first line):

We have:

• INPUT: 'STDIN' line 1 tells us the file name (STDIN) and line number.
• PATTERN: 1 tells us the (new) contents of the pattern space.
• COMMAND: /2/ { is a regexp that does not match the pattern space.
• COMMAND: } sed has moved ahead to find the closing brace.
• COMMAND: /4/ { is another regexp that does not match the pattern space.
• COMMAND: } sed has again moved ahead to find the closing brace.
• END-OF-CYCLE: shows us what actually gets printed (if anything) at the end of the cycle.
• 1 is the actual output from the sed script.

Cycle 2

Cycle 2 is more interesting:

This time:

• INPUT: 'STDIN' line 2 tells us the file name (STDIN) and line number again.
• PATTERN: 2 again tells us the new contents of the pattern space.
• COMMAND: /2/ { is a command that this time does match the pattern space.
• COMMAND: h is the hold command.
• HOLD: 2 shows us the new contents of the hold space.
• COMMAND: d deletes the pattern space - and branches to immediately end this cycle.
• END-OF-CYCLE: shows us nothing being printed at the end of this cycle.

Cycle 3

Cycle 3 is more or less the same as cycle 1:

Cycle 4

Cycle 4 is also interesting.

• INPUT: 'STDIN' line 4 we are on line 4.
• PATTERN: 4 the pattern space is now 4.
• COMMAND: /2/ { a command that does not match.
• COMMAND: } closing brace.
• COMMAND: /4/ { a command that does match the pattern space.
• COMMAND: x exchanges pattern space and hold space:
• PATTERN: 2 the new pattern space.
• HOLD: 4 the new hold space.
• COMMAND: } closing brace.
• END-OF-CYCLE: shows us what is printed at the end of the cycle.
• 2

Cycle 5

Cycle 5 is more or less the same as cycles 1 and 3:

Exit status

An exit status of zero indicates success, and a nonzero value indicates failure. GNU sed returns the following exit status error values:

Linux Sed Cheat Sheet

Also, the q and Q commands, via a GNU extension, can be used to exit with a custom exit status:

Summary

This completes an almost complete overview of the GNU sed programming language. I have covered most of sed’s features and illustrated them with examples, with the exception of regular expressions, which I regarded as documented elsewhere and not strictly speaking a feature of the sed language. I have omitted some of the command line options, undocumented behaviours, some of the most advanced commands, and my treatment of branching and flow control is cursory.

Ed Text Editor

Please let me know if you find any errors or have any suggestions for improvement!

Sed Command Line Cheat Sheet

See also

• GNU sed manual.
• sed & AWK, 2nd ed O’Reilly.
• sed FAQ.
• Sed One-liners Explained, Part I: File Spacing, Numbering and Text Conversion and Substitution.
• Useful One-line Scripts for Sed.
• Sed - An Introduction and Tutorial by Bruce Barnett (similar scope to the present post).

¹ Be aware that I am only familiar with the BSD version of sed that ships on Mac OS X, and GNU sed 4.6. All code in this post is tested on one of these two implementations.
² For example,