Introduction

Most UNIX® developers settle on Emacs, vi, or one of the many variants, offshoots, and clones of these two text-editing applications. Files are normally opened in the editor of choice, and changes are interactively specified and applied to the file by the operator.

But you can often do an editing job at the command line more quickly than it takes to open the file in a text editor. A complex editing procedure can be programmed and specified from the command line and executed across multiple files, eliminating all unnecessary screen display, cursor motion, and manual interaction with the files. A good tactic is to keep a cache of relevant one-liners on hand to do common editing jobs. Not only do they save you time, especially in batch operations involving multiple files, but you can also use them in scripts.

One-liners for editing and processing text are a famous tradition in the Perl and AWK (and lately Ruby) languages and, of course, the shell. This article demonstrates basic text-editing techniques with three of the most primary command-line editing tools readily available on all systems: cat, ed, and sed. The editing examples that follow start with the simplest and most common constructs and work up to the more complex.

Edits with cat

Use cat, whose name stands for together, to concatenate files and standard input streams, as in Listing 1. The slackers of the world also use it as a general pager (cat file) and a complete text-editing environment (cat > file). Its syntax is unrivaled in its simplicity and, for text editing one-liners, it gives you quick ways to append or insert text without an editor.

$ (cat - input1 - input2 - input3 - input4) | mailx ted
Ted,

Take a look at these example files.

This is the first file ...
Ctrl-D

This is the second file ...
Ctrl-D

This is the third file -- note the fourth paragraph below ...
Ctrl-D

And here's the last file ...
Ctrl-D
$ 

Add text to the end of a file

The slackers are on to something, though. When you need to append text to the end of a file, there's nothing quicker than cat:

$ cat >> file
> line
> line
> line
Ctrl-D
$

While you're adding lines, pressing Ctrl-U erases the current line, Ctrl-Z suspends the process, and Ctrl-C aborts everything. When you're done, press Ctrl-D on a line of its own. (These are some of the default Korn shell control keys, but they work for most shells and editing modes.)

If the data you're entering is an X selection that you're pasting from another window, this one-liner is generally quicker to use than calling up an editor, opening the target file, moving to the end of the file, pasting the selection, saving the file, and exiting the editor. It can also be more useful when you're pasting formatted or specially formatted text, and you want to keep the formatting because some text editors and editing modes reformat the X selection when you paste it.

Although this operation is a common, everyday practice, you always have to be careful that you use the shell operator for appending redirection (>>) and not the regular redirection operator (>); if you mistakenly use the latter, you'll overwrite the contents of the file with the text you mean to append.

To add the entire contents of one file to the end of another file, give the filename:

$ cat footnotes.txt >> file

If you're appending only a single line instead of multiple lines or an entire file, you can use echo instead of cat:

$ echo "192.255.255.255     bigblue" >> /etc/hosts

To append lines of text that are itemized beginning with 1, use cat's -n option; lines are preceded with the line number (offset with up to five space characters) and a tab character. Add the -b option to suppress the numbering of blank lines:

$ cat -nb > fileThis line is numberedAnd so is thisAnother numbered lineCtrl-D
$ cat file
     1  This line is numbered
     2  And so is this

     3  Another numbered line

$

Insert text at the beginning of a file

You can insert text at the beginning of a file with cat by specifying the standard input with a hyphen (-) and writing to a new file:

$ cat - file > newfile
This is the beginning of the file
And then the old file is inserted
Below this line:
Ctrl-D
$

Although it's simple, the disadvantage of this one-liner is that it creates a new file. If you want to insert text into the original file, the renaming shuffle you have to do makes this almost more trouble than it's worth. Better ways are just ahead with ed.

Show nonprinting characters

cat has several useful options. Some of them control the way it outputs nonprinting characters, such as tabs and control characters. To determine whether a file or a group of text files has embedded control characters, use these options. For instance, if a file has trailing blanks, you can see them:

$ cat -vet input.txt

This line has trailing blanks.    $
This line does not.$
$

These options differ according to your UNIX implementation; Table 1 gives the standard IBM AIX® operating system options.

Edits with ed

As its name implies, the line editor, ed, performs edits on lines of input files. It reads entire files into its own buffer, makes the specified operations on that copy, and optionally writes the buffer to disk. You can specify any number of lines in an editing operation, and these operations can be combined and given in a series. These facts make ed an excellent choice for use in scripts. Specify an operation in this format:

[address]command [text]

The address specifies the line or lines to work on (the default is the current line), and it can be given in a number of ways. The single-character command is the action to perform on the specified lines. For ad hoc one-liners in scripts, use ed noninteractively by piping a group of commands and text to it with echo:

( echo 'OPERATION'; echo 'OPERATION';
... echo 'wq' ) | ed -s FILENAME

If text is input in an operation, a period (.) should be echoed to indicate the end of input. The final wq writes the file and quits. The -s option makes ed behave silently and suppresses all normal output.

Fortunately, ed's basic addressing methods and commands are fairly standardized. The major forms of addressing are described in Table 2. Commands are given in Table 3.

Insert text at the beginning of a file, take two

You can easily insert text at the beginning of a file with a scriptable ed one-liner. The insertion is made with ed appending the given text at line 0 (the beginning of the file), with the a command:

$ cat file
This is the end.
$ (echo '0a'; echo 'This is the beginning.'; echo '.'; echo 'wq') | ed -s file
$ cat file
This is the beginning.
This is the end.
$

You can do the same thing interactively:

$ cat file
This is the end.
$ ed -s file
> 0a
> This is the beginning.
> .
> wq
$ cat file
This is the beginning.
This is the end.
$

To insert the contents of another file at the beginning of the file, use the r command:

$ (echo '0r headnotes'; echo 'wq') | ed file

Insert text after a given string

You can use ed to insert any number of lines of text before or after any line in the file. To insert after the first line containing a given string, enclose the string in slashes and follow it with the a command to append the text that follows. As before, end with a period on a line of its own, and use wq to write the file and quit.

This technique comes in handy when you want to append a block of text at a particular place in a file:

$ ( echo '/begin/a'; echo 'This is the middle.'; \
> echo '.'; echo 'wq') | ed -s file
$ cat file
This is the beginning.
This is the middle.
This is the end.
$

It's also useful when you're making a multiline text insertion into a group of files. If you have a lot of lines to insert, use a here document, which is a document that's specified inline with << and a limit string, redirecting all input that follows it until the limit string is reached (see Resources):

$ for i in *.xml
> { ed -s $i << EOF
> /<records>/a \
> <record> \
>   <name>johnnycomelately</name> \
>   <step>10</step> \
>   <dur>4</dur> \
> </record>\
> .
> wq
> EOF
> }
$

You can insert a file after a given string:

$ (echo '/END OF PART I/r footnotes.txt'; echo 'wq') | ed file

Delete lines

Use the d command to delete lines from the file. As with all commands discussed here, you can specify any kind of valid address, such as a particular line or range of lines. In practice, this one-liner is generally most useful in conjunction with at least one matched pattern, such as deleting lines beginning with the first line matches some pattern to the end of the file:

$ ( echo '/FOOTNOTES/,$/d'; echo 'wq' ) | ed -s file

You can do the inverse and delete everything from the first line of the file until the first line matches some pattern:

$ ( echo '1,/\.\.\./d'; echo 'wq' ) | ed -s file

Remove trailing blanks

You can remove trailing blanks from a file by using the s command and substituting a null replacement:

$ cat -vet input.txt

This line has trailing blanks.    $
This line does not.$
$ (echo ',s/ *$//'; echo 'wq') | ed -s input.txt
$ cat -vet input.txt

This line has trailing blanks.$
This line does not.$
$ 

Back to top

Edits with sed

The most complex and powerful of editing tools discussed here is sed (Stream EDitor). It's a text editor, but unlike file editors such as ed, it edits the input stream and writes to the output stream. Therefore, it's useful for editing command output or for when you're preprocessing a file with other tools -- you can then pipe that text output straight to sed for quick editing. But sed can also operate on files, and its scripting language has advanced pattern-matching capabilities, so it's a great choice for performing any kind of fast text edits -- such as a quick search and replace across a group of files. In fact, it's one of the most popular command-line tools in existence for text editing operations.

sed takes a script of any number of commands followed by an optional input filespec; by default, it reads standard input. Some versions of sed have an -i option, which specifies that the input file should be edited. (Without the option, the input file is read from, but it is not written to.) If your installed version supports this option, you should use it -- it lets you perform swift editing operations on any given filespec with a single command:

sed -i script filespec

The following examples assume your sed has -i. Otherwise, you have to do the temp-file shuffle using shell redirection to save the output to a new file and renaming the new to the old in another step:

sed script file > newfile; mv newfile file

And with multiple files, you then have to loop:

for i in *; { sed script $i > $i.new; mv $i.new $i; }

Replace text in a file

You can use the s/searchstring/replacestring/ construct to replace a given string with another. To replace the first occurrence of old on every line in a file, use the following:

$ sed -i 's/old/new/' file

To replace every occurrence, append g to the search. This technique is good for fixing a typo or otherwise replacing a recurring word, phrase, or other string in a file or group of files:

$ sed -i 's/Esclipse/Eclipse/g' *.xml

You can bracket characters in the input expression but, if you use brackets in the replacement text, they're treated as regular characters:

$ cat file
This is the beginning.
This is the middle.
This is the end.
$ sed 's/[Tt]h/[Tt]h/g' file
[Tt]his is [Tt]he beginning.
[Tt]his is [Tt]he middle.
[Tt]his is [Tt]he end.

When the slash character is part of the phrase to search or replace, define a new delimiter by using it:

$ sed -i 's,/usr/local/websphere,/usr/websphere,' file

You can also replace an entire line that contains a pattern with some new text:

$ sed -i 's/.*pattern.*/LINE DELETED/' file

Edit a matched pattern

Recall the example of brackets grouping characters together in a pattern and how they were taken as literal characters in the replacement text. What if you want to include the literal matched pattern in the replacement text? You can do this with an ampersand (&). This approach is useful for editing a matched pattern by putting text before or after it:

$ cat file
This is the beginning.
This is the middle.
This is the end.
$ sed 's/[Tt]h/>&</g' file
>Th<is is >th<e beginning.
>Th<is is >th<e middle.
>Th<is is >th<e end.

Insert text after a matched pattern

Use the a command to add a line of text after a given matched pattern:

$ sed -i '/pattern/a text' file

This doesn't replace the text matched by the pattern -- it only adds the text after the first line that contains that pattern.

Insert text at the beginning of lines

To insert text at the beginning of each line, match the caret metacharacter and supply the text to insert. Here's how to add e-mail style quotations to all lines in a file:

$ sed 's/^/> /' input.txt
> This line has trailing blanks.
> This line does not.
$

Insert text at the end of lines

The same principle applies to inserting text at the end of each line -- match the dollar-sign metacharacter and supply the text to insert. Here's how to emulate AIX cat's -vet options to mark trailing blanks:

$ sed 's/$/$/' file
This line has trailing blanks.   $
This line does not.$
$

Delete lines from a file

The d command deletes given lines. You can precede it with a line number, range, pattern to match, or enclosed in slashes.

To delete the first line in a file:

$ sed -i 1d file

To delete lines 1 through 10:

$ sed -i 1,10d file

To delete all the lines between the first instance of the "BEGIN QUOTE" string and the first instance of the "END QUOTE" string:

$ sed -i '/BEGIN QUOTE/,/END QUOTE/d' file

To delete all sections of text whose first line contains "<record>" and last line contains "</record>" in all files with the .xml extension in the current directory:

$ sed -i '/<record>/,/<\/record>/d' *.xml

To delete from the first line until the first blank line:

$ sed -i '/^> /d' file

(When used on e-mail messages or Usenet articles, the preceding one-liner strips out all headers.)

To delete all lines beginning with e-mail style quotations:

$ sed -i /^$/d file

To delete the last line of a file:

$ sed -i '$d' file

Delete trailing spaces

If lines of a file contain trailing space characters that you need to clean up, it can be cumbersome to manually find and edit them out in a text editor -- but with sed, doing so is a swift one-line operation. You search for a literal space character that occurs one or more times before the end of a line, and replace it with the empty set:

$ cat -vet input.txt

This line has trailing blanks.    $
This line does not.$
$ sed -i 's/ *$//' input.txt
$ cat -vet input.txt

This line has trailing blanks.$
This line does not.$
$

Back to top

Summary

Text files can be edited in significant and complex ways without an editor by running one-liners from the UNIX command line. There are many good reasons why you would want to do so: for speed and convenience, for scriptability in cases where an interactive edit isn't possible or desirable, and sometimes to do complex edits on a single file or a group of files that would be difficult or even impossible to accomplish in your interactive application. This article demonstrated the concept with many simple text editing one-liners using three of the most ubiquitous of editing tools: cat, ed, and sed.

Resources

Learn

• Consult the man pages for the AIX 5L Version 5.3 versions of:
  • cat
  • ed
  • sed
  
  
• "Text processing with UNIX" (developerWorks, Aug 2006): This article shows how you can use sed in conjunction with other command-line tools for powerful text processing.
  
  
• "Speaking UNIX, Part 1: Command the power of the command line" (developerWorks, Mar 2006): Learn how redirection and streams work in the shell.
  
  
• Teodor Zlatanov's Cultured Perl column has featured Perl one-liners. Read about them in "One-liners 101" (developerWorks, Apr 2001) and "One-liners 102" (developerWorks, Mar 2003).
  
  
• IBM System p™ and AIX Information Center: Visit this site for documents on AIX and System p.
  
  
• AIX and UNIX articles: Check out other articles written by Michael Stutz.
  
  
• Search the AIX and UNIX library by topic:
  • System administration
  • Application development
  • Performance
  • Porting
  • Security
  • Tips
  • Tools and utilities
  • Java™ technology
  • Linux®
  • Open source
  
  
• AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
  
  
• AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.
  
  
• Safari bookstore: Visit this e-reference library to find specific technical resources.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  
• Podcasts: Tune in and catch up with IBM technical experts.
  
  

Get products and technologies

• IBM trial software: Build your next development project with software for download directly from developerWorks.
  
  

Discuss

• Participate in the developerWorks blogs and get involved in the developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX 5L -- technical forum
  • AIX for Developers Forum
  • Cluster Systems Management
  • IBM Support Assistant
  • Performance Tools -- technical
  • Virtualization -- technical
  • More AIX and UNIX forums
  
  

About the author

Michael Stutz is author of The Linux Cookbook, which he also designed and typeset using only open source software. His research interests include digital publishing and the future of the book. He has used various UNIX operating systems for 20 years.

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Edits with cat
• Edits with ed
• Edits with sed
• Summary
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers