System Administration Toolkit: Get the most out of zsh

Examine key parts of the Z shell (zsh) and how to use it's features to ease your UNIX® system administration tasks. zsh is a popular alternative to the original Bourne and Korn shells. It provides an impressive range of additional functionality, including improvements for completing different commands, files, and paths automatically, and for binding keys to functions and operations.

Martin Brown (mc@mcslp.com), Freelance Writer, Freelance Developer

Martin Brown has been a professional writer for more than seven years. He is the author of numerous books and articles across a range of topics. His expertise spans myriad development languages and platforms—Perl, Python, Java™, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk, Shellscript, Windows®, Solaris, Linux, BeOS, Mac OS X and more—as well as Web programming, systems management, and integration. He is a Subject Matter Expert (SME) for Microsoft® and regular contributor to ServerWatch.com, LinuxToday.com, and IBM developerWorks. He is also a regular blogger at Computerworld, The Apple Blog, and other sites. You can contact him through his Web site.



19 December 2006

Also available in Chinese Russian

About this series

The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.


zsh background

Shells under UNIX and Linux® typically fall into one of two categories based on the original shells included with the earliest versions of UNIX. The two types are the Bourne shell and the C shell; the latter being distinctive because its format and structure is like that of the C programming language.

The Bourne shell is easier to use and understand than the C shell, but it is less practical for the complex script programming that you might want to achieve within the shell programming environment. The Korn shell provides ease of use of the Bourne shell and added extensions for job control (allowing you to easily manage multiple background jobs), command-line editing and history, and added elements of the C shell to make programming easier.

The Z shell (zsh) was designed with interactive use in mind rather than programming, so it incorporates a lot of functionality that makes using and running commands much easier. Examples include more extensive filename matching (globbing), multiple I/O streams for redirecting input and output, and a fully customizable command-line completion system.


Filename generation

Filename globbing is the process behind turning a filename or file specification into a list of files for use on the command line, for example, when copying or moving files. Basic filename globbing includes the use of the ? for a single character, or * for one or more characters.

For example, to list all of the C source files, you might use Listing 1.

Listing 1. Listing all of the C source files
$ ls *.c
barney.c        betty.c         fred.c          wilma.c

And you can use letter collections (as you might in a regular expression), for example, to list the files with a "c" or "o" extension, as shown in Listing 2.

Listing 2. Listing the files with a "c" or "o" extension
$ ls *.[co]
barney.c        betty.c         fred.o
barney.o        fred.c          wilma.c

With zsh, the same wildcards work, but you can also use extended globbing for additional options. To switch on extended globbing, set the EXTENDEDGLOB environment variable, or use: $ setopt extendedglob.

You can now use constructs like the ^ character to display all of the files that don't match a given file specification. For example, to list all of the files that don't match the *.c specification, use Listing 3.

Listing 3. Listing files that don't match *.c
zsh$ ls ^*.c            
barney.o        betty.h         fred.h          fred.o

The expression <x-y> matches a range of integers. For example, if you have filed your access logs using dates, you can then select all of the files within a specific range of dates. To achieve this, name the files with the year, month, and date, in that order, using zeros to pad out the values. For example, to list the logs between the third and tenth of November 2006, you might use Listing 4.

Listing 4. Listing the logs between the third and tenth of November 2006
zsh$ ls access<20061103-20061110>.log
access20061103.log      access20061106.log      access20061109.log
access20061104.log      access20061107.log      access20061110.log
access20061105.log      access20061108.log

You can also use regular expression-like groups to match files. For example, use Listing 5 to list all the files called fred and barney.

Listing 5. Listing all the fred and barney files
zsh$ ls (fred|barney)*
barney.c        fred.c          fred.o
barney.o        fred.h

Subdirectories can also be searched by using **/; the process is recursive, such as the find command, so that you can produce the equivalent of a find command, such as $ find . -name "*.c".

In zsh (with extended globbing), this can be extended to: zsh$ ls **/*.c.

Or you can combine further examples, such as find all of the C source and compiled object files in subdirectories using the command in Listing 6.

Listing 6. Finding all C source and compiled object files
zsh$ ls **/*.(c|o)
glob/barney.c   glob/betty.c    glob/fred.o
glob/barney.o   glob/fred.c     glob/wilma.c

Finally, zsh supports a number of postfix qualifiers. To get a list of executable files, use the postfix qualifier (*) at the end of the globbing expression (see Listing 7).

Listing 7. Getting a list of executable files
zsh$ ls *(*)
barney  fred

You can also get a list of executable files using (x) at the end of the globbing expression (see Listing 8).

Listing 8. Using (x) to get a list of executable files
zsh$ ls *(x) 
barney  fred

The above list files are executable by the file's owner. A capital X selects files that are executable by others. The same also applies with the R/r (readable) and W/w (writable) mode on the file.


Command or process substitution

Within most shell environments, you can use basic process substitution to embed the output of one command into the input or arguments of another. You can do this with the backtick operator (see Listing 9).

Listing 9. Using the backtick operator to perform process substitution
$ emacs `find . -name "*.html"`

There are a number of options available to you within zsh. The primary method is to use the $() construct. This provides a direct alternative to the backticks, and it has the benefit of being more easily embeddable and nestable in certain command combinations. For example, you could rewrite Listing 9 as Listing 10.

Listing 10. Alternative to using backticks to perform process substitution
zsh$ emacs $(ls **/*.html)

Here the process substitution runs the directory listing components and returns a list of filenames, which is in turn supplied to the argument list of the emacs command.

Another useful construct is the =(list) structure. When using this feature, the element in the parentheses generates a temporary file, and the name of this file is returned. For example, you can generate a text file using Listing 11.

Listing 11. Generating a text file
zsh$ cat =(print -l tom dick harry)                            
tom
dick
harry

More usefully, you can combine it with other elements in order to support more complex output and filtering. For example, you can get a list of processes matching imapd and httpd (IMAP mail service and Apache http service) using the following command (see Listing 12).

Listing 12. Getting a list of processes matching imapd and httpd
zsh$ ps -ax |fgrep -f =(print -l httpd imapd)
 406 ?? Ss  0:02.05 /usr/sbin/httpd
 426 ?? S   0:01.32 /usr/sbin/httpd
 429 ?? S   0:06.42 imapd: sulaco.mcslp.pri [192.168.0.101] appleblog user.appleblog
 434 ?? S   0:57.81 imapd: sulaco.mcslp.pri [192.168.0.101] mcarc user.mcarc
 435 ?? S   0:00.14 imapd: sulaco.mcslp.pri [192.168.0.101] mlists user.mlists
 436 ?? S   0:00.12 imapd: sulaco.mcslp.pri [192.168.0.101] play user.play
 437 ?? S   0:01.16 imapd: sulaco.mcslp.pri [192.168.0.101] mc user.mc
 507 ?? S   0:01.25 /usr/sbin/httpd

In Listing 12, the print command prints each argument on an individual line to a temporary file. The file is then used by fgrep as the list of matches against the process list. The temporary file is deleted once the command has been finished.

Because the feature creates temporary files from the output, you can use it to perform functions and sequences that would have previously required you to create a temporary file and delete it yourself. For example, to compare the list of files between two different directories, you could output a list of files in each directory to a temporary file, and then use the diff command to compare the output. With zsh, the same result can be achieved on a single command line (see Listing 13).

Listing 13. Compare the list of files between two different directories
zsh$ diff =(ls new) =(ls old)
3d2
< barney.o
9d7
< fred.o

In this example, the two substituted ls commands generate a temporary text file that is then compared inline by the diff command.


Multiple I/O streams

Related to the process substitution system is a more extensive system of redirection. In most shells, you are used to redirecting to and from files (see Listing 14).

Listing 14. Redirecting to and from files
$ crontab <newtab
$ ls > filelist

You can redirect to multiple outputs simultaneously with zsh (see Listing 15).

Listing 15. Redirecting to multiple outputs simultaneously
zsh$ ls >listone >listtwo

You can also input from multiple streams (see Listing 16).

Listing 16. Inputting from multiple streams
zsh$ sort <listone <listtwo

The pipe is an implicit redirect, so it works in much the same way. For example, within any shell, you can use the tee command to redirect a file and the standard output (see Listing 17 ).

Listing 17. Using the tee command to redirect a file and the standard output
$ ls | tee listone
barney
barney.c
barney.o
betty.c
betty.h
fred
fred.c
fred.h
fred.o
wilma.c

You can use Listing 18 within zsh.

Listing 18. Redirecting to a file and standard output in zsh
zsh$ ls >fileone |cat
barney
barney.c
barney.o
betty.c
betty.h
fileone
fred
fred.c
fred.h
fred.o
wilma.c

As an extension of the process substitution, you can also redirect to and from another command by using the <(list) and >(list) constructs (see Listing 19).

Listing 19. Redirecting to and from another command using < and >
zsh$ sort <(ls) <(ls /usr)      
X11R6
barney
barney.c
barney.o
betty.c
betty.h
bin
fileone
fred
fred.c
fred.h
fred.o
include
lib
libexec
local
sbin
share
standalone
wilma.c

In Listing 19, you combine the output from the two ls commands as the standard input for the sort command, which outputs an amalgamated and sorted list of files from two different directories.

A typical use for this is where you are using cut and paste to extract and recombine fields from a file into another. You would probably use a number of temporary files with a normal shell (see Listing 20).

Listing 20. Using temporary files to extract and recombine fields from one file into another
$ cut -f1 fileone >t1
$ cut -f5 filetone >t5
$ paste t1 t5

In zsh, you can accomplish this by skipping the temporary files, as shown in Listing 21.

Listing 21. Cut and paste to extract and recombine fields from a file within zsh
zsh$ paste -d: <(cut -d: -f1 /etc/passwd) <(cut -d: -f5 /etc/passwd)

Furthermore, because the redirected substitution can be easily nested, you can create complex structures, such as combining two fields from the passwd file in a different order from source, removing the comments, and then sorting them (see Listing 22).

Listing 22. Creating complex structures in zsh
zsh$ sort <(egrep -v '^#' <(paste -d: <(cut -d: -f5 /etc/passwd) <(cut -d: -f1 
/etc/passwd) ) )
:# mode.  At other times this information is handled by one or more of
Amavisd User:amavisd
Apple Events User:eppc
Application Owner:appowner
Application Server:appserver
...
Xgrid Agent:xgridagent
Xgrid Controller:xgridcontroller
sshd Privilege separation:sshd

You can simplify Listing 22 by looking at each element as in Listing 23.

Listing 23. Simplified process
-	<(cut -d: -f1 /etc/passwd) - Get the first field
-	<(cut -d: -f5 /etc/passwd) - Get the fifth field
-	<(paste -d: <(f5) <(f1) ) - Recombine them in a different order
-	<(egrep -v '^#' <(paste...) - Remove the comments
-	sort <(egrep ...) - Sort the standard input

File and command completion

Within some shells, zsh included, you can complete a file or command by pressing the TAB key. Let's use your current directory as an example (see Listing 24).

Listing 24. Listing of the current directory
zsh$ ls
barney          betty.c         fred            fred.o
barney.c        betty.h         fred.c          wilma.c
barney.o        fileone         fred.h

With completion, you can type the start of a filename: zsh$ cd bar. Then press the TAB key and get a full or partial completion: zsh$ cd barney.

Press TAB a second time, and you are presented with a list of available files (see Listing 25).

Listing 25. Getting a list of available files
zsh$ cd barney
barney*   barney.c  barney.o

The same completion process also works with directories and paths. zsh has a further trick up its sleeve with respect to completion.


Customizing completion

The limitation of completion in its standard form is that it is only able to complete files and paths (including commands) while you are entering them on the command line. There are, however, many additional areas where you might want to be able to complete the command without having to complete all of the typing. For example, Subversion, the source code control system, provides a number of secondary commands that you type in addition to other arguments. For example, to commit changes back to a Subversion repository, you use the commit command: $ svn commit. Or, to update, you use the update command: $ svn update.

But you have to type this manually. By using the custom completion control in zsh, you can add these subcommands to svn as part of the completion process. Completion control is a very complex (and sometimes confusing) system, but the fundamentals are fairly easy to understand.

Completion is controlled through a number of commands, but the main command is compctl. This provides a simple interface for basic completion. Completion can be applied globally (in other words, like the files and paths), or it can be applied to specific commands.

There are a range of options and formats available, but to continue with the Subversion idea, you can use the -k option to provide an array of words that act as a potential completion to the svn command (see Listing 26).

Listing 26. Using the -k option
zsh$ compctl -k '(commit checkout update status)' svn

Now when you type svn at the start of the command, you need to press TAB to complete the command (see Listing 27).

Listing 27. Pressing TAB to complete command
zsh$ svn com <TAB>
zsh$ svn commit

The completion system includes standard functions and completions -- for example, lookups of valid users on the system, home directories, hosts, networks, and so on. You can also add and extend these yourself to produce customized completion results.

For example, you can create a custom function called activeusers that returns the output from the users command: zsh$ function activeusers { reply =(`users`) }.

You can now use this as a completion for another command, for example, chat: zsh$ compctl -K activeusers chat.

Now when you type chat on the command line, you can obtain a completion list that only returns a list of the users currently logged in.

The available options and possibilities with the custom completion system are too numerous and extensive to cover here. See Resources for the official documentation on the available options.


Summary

zsh incorporates a range of functionality designed to make the interaction between the user and shell environment easier. Extensions include better ways of substituting commands and redirecting information in and out to different processes. These options alone enable you to convert the many commands that would be needed in another shell into a single command-line entry within zsh.

The real differentiator between zsh and other shells, even the recent improvements provided in shells like bash, is the ability to customize the auto-completion system to work with more than just filenames and paths. Extending the functionality to support additional arguments to your existing commands is just one example, but the system is so flexible that you could almost complete any command or command-line element.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=184761
ArticleTitle=System Administration Toolkit: Get the most out of zsh
publish-date=12192006