Save time with text editing one-liners

Fast editing examples using cat, ed, and sed

Take a quick look at some essential editing one-liners that can save you time and effort. Text-editing operations are normally done interactively, inside a text editor application. Some tasks, however, can be accomplished quickly and easily, right from the UNIX® command line. What's more, these one-liners can be used in scripts to automate various editing procedures.

Michael Stutz, Author, Consultant

Photo of Michael StutzMichael Stutz is author of The Linux Cookbook, which he also designed and typeset using only open source software. His research interests include digital publishing and the future of the book. He has used various UNIX operating systems for 20 years.



16 January 2007

Also available in Chinese Russian

Introduction

Most UNIX® developers settle on Emacs, vi, or one of the many variants, offshoots, and clones of these two text-editing applications. Files are normally opened in the editor of choice, and changes are interactively specified and applied to the file by the operator.

But you can often do an editing job at the command line more quickly than it takes to open the file in a text editor. A complex editing procedure can be programmed and specified from the command line and executed across multiple files, eliminating all unnecessary screen display, cursor motion, and manual interaction with the files. A good tactic is to keep a cache of relevant one-liners on hand to do common editing jobs. Not only do they save you time, especially in batch operations involving multiple files, but you can also use them in scripts.

One-liners for editing and processing text are a famous tradition in the Perl and AWK (and lately Ruby) languages and, of course, the shell. This article demonstrates basic text-editing techniques with three of the most primary command-line editing tools readily available on all systems: cat, ed, and sed. The editing examples that follow start with the simplest and most common constructs and work up to the more complex.

Edits with cat

Use cat, whose name stands for together, to concatenate files and standard input streams, as in Listing 1. The slackers of the world also use it as a general pager (cat file) and a complete text-editing environment (cat > file). Its syntax is unrivaled in its simplicity and, for text editing one-liners, it gives you quick ways to append or insert text without an editor.

Listing 1. Using cat to concatenate files and standard input streams
$ (cat - input1 - input2 - input3 - input4) | mailx ted
Ted,

Take a look at these example files.

This is the first file ...
Ctrl-D

This is the second file ...
Ctrl-D

This is the third file -- note the fourth paragraph below ...
Ctrl-D

And here's the last file ...
Ctrl-D
$

Add text to the end of a file

The slackers are on to something, though. When you need to append text to the end of a file, there's nothing quicker than cat:

$ cat >> file
> line
> line
> line
Ctrl-D $

While you're adding lines, pressing Ctrl-U erases the current line, Ctrl-Z suspends the process, and Ctrl-C aborts everything. When you're done, press Ctrl-D on a line of its own. (These are some of the default Korn shell control keys, but they work for most shells and editing modes.)

If the data you're entering is an X selection that you're pasting from another window, this one-liner is generally quicker to use than calling up an editor, opening the target file, moving to the end of the file, pasting the selection, saving the file, and exiting the editor. It can also be more useful when you're pasting formatted or specially formatted text, and you want to keep the formatting because some text editors and editing modes reformat the X selection when you paste it.

Although this operation is a common, everyday practice, you always have to be careful that you use the shell operator for appending redirection (>>) and not the regular redirection operator (>); if you mistakenly use the latter, you'll overwrite the contents of the file with the text you mean to append.

To add the entire contents of one file to the end of another file, give the filename:

$ cat footnotes.txt >> file

If you're appending only a single line instead of multiple lines or an entire file, you can use echo instead of cat:

$ echo "192.255.255.255     bigblue" >> /etc/hosts

To append lines of text that are itemized beginning with 1, use cat's -n option; lines are preceded with the line number (offset with up to five space characters) and a tab character. Add the -b option to suppress the numbering of blank lines:

$ cat -nb > fileThis line is numberedAnd so is thisAnother numbered lineCtrl-D
$ cat file
     1  This line is numbered
     2  And so is this

     3  Another numbered line

$

Insert text at the beginning of a file

You can insert text at the beginning of a file with cat by specifying the standard input with a hyphen (-) and writing to a new file:

$ cat - file > newfile
This is the beginning of the file
And then the old file is inserted
Below this line:
Ctrl-D
$

Although it's simple, the disadvantage of this one-liner is that it creates a new file. If you want to insert text into the original file, the renaming shuffle you have to do makes this almost more trouble than it's worth. Better ways are just ahead with ed.

Show nonprinting characters

cat has several useful options. Some of them control the way it outputs nonprinting characters, such as tabs and control characters. To determine whether a file or a group of text files has embedded control characters, use these options. For instance, if a file has trailing blanks, you can see them:

$ cat -vet input.txt

This line has trailing blanks.    $
This line does not.$
$

These options differ according to your UNIX implementation; Table 1 gives the standard IBM AIX® operating system options.

Table 1. Options for control of output in AIX cat
OptionDescription
-bDon't number blank lines.
-eShow the end of line with a $ character.
-nNumber all lines of output, beginning with 1.
-qUse quiet operation (suppress error messages).
-rReplace all multiple empty lines with a single line ("squeeze" blanks).
-SSqueeze multiple blank lines together into a single line (same as -r).
-sSuppress error messages (quiet operation).
-tShow tab characters as ^I.
-uDon't buffer output.
-vShow nonprinting control characters visibly.

Edits with ed

As its name implies, the line editor, ed, performs edits on lines of input files. It reads entire files into its own buffer, makes the specified operations on that copy, and optionally writes the buffer to disk. You can specify any number of lines in an editing operation, and these operations can be combined and given in a series. These facts make ed an excellent choice for use in scripts. Specify an operation in this format:

[address]command [text]

The address specifies the line or lines to work on (the default is the current line), and it can be given in a number of ways. The single-character command is the action to perform on the specified lines. For ad hoc one-liners in scripts, use ed noninteractively by piping a group of commands and text to it with echo:

( echo 'OPERATION'; echo 'OPERATION';
... echo 'wq' ) | ed -s FILENAME

If text is input in an operation, a period (.) should be echoed to indicate the end of input. The final wq writes the file and quits. The -s option makes ed behave silently and suppresses all normal output.

Fortunately, ed's basic addressing methods and commands are fairly standardized. The major forms of addressing are described in Table 2. Commands are given in Table 3.

Table 2. Addressing lines in ed
OptionDescription
.This option addresses the current line (the default address).
numberThis option addresses the line number. Lines might be addresses in a range, separated by a comma (first,last). 0 addresses the beginning of the buffer (before the first line).
-numberThis option addresses the line number lines before the current line. Without number, the minus addresses the line immediately preceding the current line.
+numberThis option addresses the line number lines after the current line. Without number, the plus addresses the line immediately after the current line.
$This option addresses the last line.
,This option addresses the first through the last line, inclusive (same as 1,$).
;This option addresses the current line through the last line.
/pattern/This option addresses the next line containing the text that matches pattern.
?pattern?This option addresses the previous line containing the text that matches pattern.
Table 3. Major ed commands
CommandDescription
aThis command appends text after the specified address.
cThis command changes the specified address to the given text.
dThis command deletes the addressed lines.
iThis command inserts text before the specified address.
qThis command quits the program and exits, as long as the buffer is saved to disk.
r fileThis command reads the contents of filespec and inserts it after the specified address.
s/pattern/replacement/This command substitutes text matched by pattern, with replacement text in the specified address.
w fileThis command writes the specified address to file. With no address, this command uses a default of the entire buffer.

Insert text at the beginning of a file, take two

You can easily insert text at the beginning of a file with a scriptable ed one-liner. The insertion is made with ed appending the given text at line 0 (the beginning of the file), with the a command:

$ cat file
This is the end.
$ (echo '0a'; echo 'This is the beginning.'; echo '.'; echo 'wq') | ed -s file
$ cat file
This is the beginning.
This is the end.
$

You can do the same thing interactively:

$ cat file
This is the end.
$ ed -s file
> 0a
> This is the beginning.
> .
> wq
$ cat file
This is the beginning.
This is the end.
$

To insert the contents of another file at the beginning of the file, use the r command:

$ (echo '0r headnotes'; echo 'wq') | ed file

Insert text after a given string

You can use ed to insert any number of lines of text before or after any line in the file. To insert after the first line containing a given string, enclose the string in slashes and follow it with the a command to append the text that follows. As before, end with a period on a line of its own, and use wq to write the file and quit.

This technique comes in handy when you want to append a block of text at a particular place in a file:

$ ( echo '/begin/a'; echo 'This is the middle.'; \
> echo '.'; echo 'wq') | ed -s file
$ cat file
This is the beginning.
This is the middle.
This is the end.
$

It's also useful when you're making a multiline text insertion into a group of files. If you have a lot of lines to insert, use a here document, which is a document that's specified inline with << and a limit string, redirecting all input that follows it until the limit string is reached (see Resources):

$ for i in *.xml
> { ed -s $i << EOF
> /<records>/a \
> <record> \
>   <name>johnnycomelately</name> \
>   <step>10</step> \
>   <dur>4</dur> \
> </record>\
> .
> wq
> EOF
> }
$

You can insert a file after a given string:

$ (echo '/END OF PART I/r footnotes.txt'; echo 'wq') | ed file

Delete lines

Use the d command to delete lines from the file. As with all commands discussed here, you can specify any kind of valid address, such as a particular line or range of lines. In practice, this one-liner is generally most useful in conjunction with at least one matched pattern, such as deleting lines beginning with the first line matches some pattern to the end of the file:

$ ( echo '/FOOTNOTES/,$/d'; echo 'wq' ) | ed -s file

You can do the inverse and delete everything from the first line of the file until the first line matches some pattern:

$ ( echo '1,/\.\.\./d'; echo 'wq' ) | ed -s file

Remove trailing blanks

You can remove trailing blanks from a file by using the s command and substituting a null replacement:

$ cat -vet input.txt

This line has trailing blanks.    $
This line does not.$
$ (echo ',s/ *$//'; echo 'wq') | ed -s input.txt
$ cat -vet input.txt

This line has trailing blanks.$
This line does not.$
$

Edits with sed

The most complex and powerful of editing tools discussed here is sed (Stream EDitor). It's a text editor, but unlike file editors such as ed, it edits the input stream and writes to the output stream. Therefore, it's useful for editing command output or for when you're preprocessing a file with other tools -- you can then pipe that text output straight to sed for quick editing. But sed can also operate on files, and its scripting language has advanced pattern-matching capabilities, so it's a great choice for performing any kind of fast text edits -- such as a quick search and replace across a group of files. In fact, it's one of the most popular command-line tools in existence for text editing operations.

sed takes a script of any number of commands followed by an optional input filespec; by default, it reads standard input. Some versions of sed have an -i option, which specifies that the input file should be edited. (Without the option, the input file is read from, but it is not written to.) If your installed version supports this option, you should use it -- it lets you perform swift editing operations on any given filespec with a single command:

sed -i script filespec

The following examples assume your sed has -i. Otherwise, you have to do the temp-file shuffle using shell redirection to save the output to a new file and renaming the new to the old in another step:

sed script file > newfile; mv newfile file

And with multiple files, you then have to loop:

for i in *; { sed script $i > $i.new; mv $i.new $i; }

Replace text in a file

You can use the s/searchstring/replacestring/ construct to replace a given string with another. To replace the first occurrence of old on every line in a file, use the following:

$ sed -i 's/old/new/' file

To replace every occurrence, append g to the search. This technique is good for fixing a typo or otherwise replacing a recurring word, phrase, or other string in a file or group of files:

$ sed -i 's/Esclipse/Eclipse/g' *.xml

You can bracket characters in the input expression but, if you use brackets in the replacement text, they're treated as regular characters:

$ cat file
This is the beginning.
This is the middle.
This is the end.
$ sed 's/[Tt]h/[Tt]h/g' file
[Tt]his is [Tt]he beginning.
[Tt]his is [Tt]he middle.
[Tt]his is [Tt]he end.

When the slash character is part of the phrase to search or replace, define a new delimiter by using it:

$ sed -i 's,/usr/local/websphere,/usr/websphere,' file

You can also replace an entire line that contains a pattern with some new text:

$ sed -i 's/.*pattern.*/LINE DELETED/' file

Edit a matched pattern

Recall the example of brackets grouping characters together in a pattern and how they were taken as literal characters in the replacement text. What if you want to include the literal matched pattern in the replacement text? You can do this with an ampersand (&). This approach is useful for editing a matched pattern by putting text before or after it:

$ cat file
This is the beginning.
This is the middle.
This is the end.
$ sed 's/[Tt]h/>&</g' file
>Th<is is >th<e beginning.
>Th<is is >th<e middle.
>Th<is is >th<e end.

Insert text after a matched pattern

Use the a command to add a line of text after a given matched pattern:

$ sed -i '/pattern/a text' file

This doesn't replace the text matched by the pattern -- it only adds the text after the first line that contains that pattern.

Insert text at the beginning of lines

To insert text at the beginning of each line, match the caret metacharacter and supply the text to insert. Here's how to add e-mail style quotations to all lines in a file:

$ sed 's/^/> /' input.txt
> This line has trailing blanks.
> This line does not.
$

Insert text at the end of lines

The same principle applies to inserting text at the end of each line -- match the dollar-sign metacharacter and supply the text to insert. Here's how to emulate AIX cat's -vet options to mark trailing blanks:

$ sed 's/$/$/' file
This line has trailing blanks.   $
This line does not.$
$

Delete lines from a file

The d command deletes given lines. You can precede it with a line number, range, pattern to match, or enclosed in slashes.

To delete the first line in a file:

$ sed -i 1d file

To delete lines 1 through 10:

$ sed -i 1,10d file

To delete all the lines between the first instance of the "BEGIN QUOTE" string and the first instance of the "END QUOTE" string:

$ sed -i '/BEGIN QUOTE/,/END QUOTE/d' file

To delete all sections of text whose first line contains "<record>" and last line contains "</record>" in all files with the .xml extension in the current directory:

$ sed -i '/<record>/,/<\/record>/d' *.xml

To delete from the first line until the first blank line:

$ sed -i '/^> /d' file

(When used on e-mail messages or Usenet articles, the preceding one-liner strips out all headers.)

To delete all lines beginning with e-mail style quotations:

$ sed -i /^$/d file

To delete the last line of a file:

$ sed -i '$d' file

Delete trailing spaces

If lines of a file contain trailing space characters that you need to clean up, it can be cumbersome to manually find and edit them out in a text editor -- but with sed, doing so is a swift one-line operation. You search for a literal space character that occurs one or more times before the end of a line, and replace it with the empty set:

$ cat -vet input.txt

This line has trailing blanks.    $
This line does not.$
$ sed -i 's/ *$//' input.txt
$ cat -vet input.txt

This line has trailing blanks.$
This line does not.$
$

Summary

Text files can be edited in significant and complex ways without an editor by running one-liners from the UNIX command line. There are many good reasons why you would want to do so: for speed and convenience, for scriptability in cases where an interactive edit isn't possible or desirable, and sometimes to do complex edits on a single file or a group of files that would be difficult or even impossible to accomplish in your interactive application. This article demonstrated the concept with many simple text editing one-liners using three of the most ubiquitous of editing tools: cat, ed, and sed.

Resources

Learn

Get products and technologies

  • IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=187869
ArticleTitle=Save time with text editing one-liners
publish-date=01162007