Automate, automate, automate!
Mechanize personal and system chores with shell scripts
This content is part # of # in the series: Speaking UNIX, Part 6
This content is part of the series:Speaking UNIX, Part 6
Stay tuned for additional content in this series.
If you peer over a longtime UNIX® user's shoulder while he or she works, you might be mesmerized by the strange incantations being murmured at the command line. If you've read any of the previous articles in the Speaking UNIX series (see Related topics), at least some of the mystical runes being typed -- such as tilde (~), pipe (|), variables, and redirection (
>) -- will look familiar. You might also recognize certain UNIX command names and combinations, or realize when an alias is being used as a sorcerer's shorthand.
Still, other command-line conjurations might elude you, because it's typical for an experienced UNIX user to amass a large arsenal of small, highly specialized spells in the form of shell scripts to simplify or automate oft-repeated tasks. Rather than type and re-type a (potentially) complex series of commands to accomplish a chore, a shell script mechanizes the work.
In Part 6 of the Speaking UNIX series (see Related topics), you'll learn how to write shell scripts and more command-line tricks.
Ben, just one word: "automation"
Some shell scripts run exactly the same commands, processing the same set of files time and again. For instance, a Z shell script to propagate the entire contents of your home directory to three remote computers could be as simple as Listing 1.
Listing 1. A simple shell script to synchronize your home directory across many remote machines
#! /bin/zsh for each machine (groucho chico harpo) rsync -e ssh --times --perms --recursive --delete $HOME $machine: end
To use Listing 1 as a shell script, save the contents above to a file -- say, simpleprop.zsh -- and run
chmod +x simpleprop.zsh to make the file executable. You can the run the script by typing
If you'd like to see how Z shell expands each command, add the
-x option to the end of the
#! (the octothorp-exclamation pair is commonly referred to as shuh-bang) line of the script, like so:
#! /bin/zsh -x
For each computer,
harpo, the script runs the
rsync command, replacing
$HOME with your home directory (for example, /home/joe) and
$machine with a computer name.
As demonstrated in Listing 1, variables and script control structures, such as loops, make scripts easier to write and simpler to maintain. If you'd like to include a fourth computer, such as zeppo, to your pool, simply add it to the list. If you must change the
rsync command, say, to add another option, there's only one instance to edit. As in traditional programming, you should strive to avoid cut-and-paste in shell scripts, too.
Making a good argument
Other shell scripts require arguments, or a dynamic list of things -- files, directories, computer names -- to process. As an example, consider Listing 2, a variation of the previous example that allows you to use the command line to name the computers you'd like to synchronize to.
Listing 2. A variation of Listing 1 that allows you to name which computers to process
#! /bin/zsh for each machine rsync -e ssh --times --perms --recursive --delete $HOME $machine: end
Assuming that you save Listing 2 in a file called synch.zsh, you'd invoke the script as
zsh synch.zsh moe larry curly to copy your home directory to the computers moe, larry, and curly.
The missing list on the
foreach line isn't a typo: If you omit a list, the
foreach structure processes the list of arguments given on the command line. Command-line arguments are also called positional parameters, because the position of an argument on the command line is usually semantically important.
As an example, Listing 2 can leverage the existence or non-existence of positional parameters to provide a helpful usage message if you specify no arguments. The enhanced script is shown in Listing 3.
Listing 3. Many scripts provide helpful messages if no arguments are provided
#! /bin/zsh if [[ -z $1 || $1 == "--help" ]] then echo "usage: $0 machine [machine ...] fi foreach machine rsync -e ssh --times --perms --recursive --delete $HOME $machine: end
Each space-delimited string on the command line becomes a positional parameter, including the name of the script being invoked. Hence, the command
synch.zsh has only one positional parameter,
synch.zsh --help command has two:
$1 is the string
So, Listing 3 says, "If the first positional parameter is empty (the
-z operator tests for an empty string) or (denoted by
||) if the first parameter is equal to '--help', then print a usage message." (If you start writing scripts, consider providing a usage message in each one as a hint. It reminds others -- and even you, if you forget -- how to use the script.)
[[ -z $1 || $1 == "--help" ]] is the condition of the
if statement, but you can also use the same conditional as a command and combine it with other commands to control flow through your script. Take a look at Listing 4. It enumerates all the executable commands in your
$PATH, and it uses conditions in combination with other commands to perform suitable work.
Listing 4. List the commands in your $PATH
#! /bin/zsh directories=(`echo $PATH | column -s ':' -t`) for directory in $directories do [[ -d $directory ]] || continue pushd "$directory" for file in * do [[ -x $file && ! -d $file ]] || continue echo $file done popd done | sort | uniq
There's quite a bit going on in the script, so let's break it down into pieces:
- The first actual line of the script --
directories=(`echo $PATH | column -s ':' -t`)-- creates an array of named directories. You create an array in
zshby placing parentheses around your arguments, as in
). In this case, the elements of the array are generated by splitting
$PATHat each colon (
column -s ':') to yield a space-delimited list (the
column) of directories.
- For each directory in the list, the script attempts to enumerate the executable files in the directory. Steps 3 through 6 describe the process.
[[ -d $directory ]] || continueline is an example of a so-called
short-circuitingcommand terminates "as soon as" its logical conditions yield a definitive result.
For instance, the
[[ -d $directory ]] || continuephrase uses a logical OR (
||) -- it executes the first command and executes the second command if -- and only if -- the first command fails. So, if the entry in
$directoryexists and is a directory (the
-doperator), the test succeeds, evaluation ends, and the
continuecommand, which skips processing of the current element, never executes.
However, if the first test fails, the next condition of the logical or
continuealways succeeds, so it typically appears last in a
Short-circuitingbased on logical AND (
&&) executes the first command, and then executes the second command if, and only if, the first command succeeds.
popdare used to change to a new directory before processing and change to the previous directory after processing, respectively. Using the directory stack is a good scripting technique to maintain your place in the file system.
- The inner
forloop enumerates all the files in the current working directory -- the wild card
*(asterisk) matches everything -- and then tests whether each entry is a file. The line
[[ -x $file && ! -d $file ]] || continuesays, "If
$fileexists and is executable and isn't a directory, then process it; otherwise,
- Finally, if all the former conditions are met, the name of the file is printed with
- Did you catch the last line of the script? You can send the output of most control structures to another UNIX command -- after all, the shell treats the control structure as a command. Therefore, the output of the entire script is piped through
sort, and then
uniqto yield an alphabetized list of unique commands found in your
If you save Listing 4 to an executable file named listcmds.zsh, the output might look like this:
$ ./listcmds.zsh [ a2p ab ac accept accton aclocal
short-circuiting command is very useful in scripts. It combines a conditional and an operation in one. And because every UNIX command returns a status code reflecting success or failure, you can use any command as a
conditional -- not just the test operators. By convention, UNIX commands return zero (0) for success and non-zero for failure, where the non-zero value reflects the kind of error that occurred.
popd could have been eliminated from Listing 4 if the line
[[ -d $directory ]] || continue was replaced with
cd $directory || continue. If the
cd command succeeds, it returns 0 and evaluation of the logical OR can end immediately. However, if
cd fails, it returns non-zero, evaluation proceeds, and
Don't remove. Archive!
Modern UNIX shells --
zsh -- offer many control structures and operations to create complex scripts. Because you can call upon all the UNIX commands to massage data from one form to another, shell scripting is nearly as rich as programming in a complete language, such as
C or Perl.
You can use scripts to mechanize virtually any personal or system task. Scripts can monitor, archive, update, upload, download, and transform data. A script can be a single line or an enormous subsystem. No job is too small or too great (almost) for a shell script. Indeed, if you look at your /etc/init.d directory, you'll find a variety of shell scripts that launch services each time you start your computer. If you create a very useful script, you can even deploy it as a system-wide utility. Just drop it into a directory on
$PATH of users.
Let's create a utility to exercise your newfound mojo. The script, myrm, is a replacement for the system's own rm. Rather than deleting a file outright, myrm copies the file to an archive, names it uniquely so you can find it later, and then removes the original file. The myrm script is functional but simple, and you can add many bells and whistles. You can also write an extensive unrm ("un-remove") script as a companion. (You can search the Internet to find a variety of implementations.)
The myrm script is shown in Listing 5.
Listing 5. A simple utility to back up a file before it's removed from the file system
#! /bin/zsh backupdir=$HOME/.tomb systemrm=/bin/rm if [[ -z $1 || $1 == "--help" ]] then exec $systemrm fi if [[ ! -d $backupdir ]] then mkdir -m 0700 $backupdir || echo "$0: Cannot create $backupdir"; exit fi args$=$( getopt dfiPRrvw $* ) || exec $systemrm count=0 flags = "" foreach argument in $args do case $argument in --) break; ;; *) flags="$flags $argument"; (( count=$count + 1 )); ;; esac done shift $(( $count )) for file do [[ -e $file ]] || continue copyfile=$backupdir/$(basename $file).$(date "+%m.%d.%y.%H.%M.%S") /bin/cp -R $file $copyfile done exec $systemrm $=flags "$@"
You should find the shell script readable, although there are a few new things that haven't been discussed before. Let's cover those, and then review the entire script.
- When a shell launches a command, such as
ls, it spawns a new process for the command, and then waits for the (sub)process to finish before proceeding. The
execcommand also launches a command, but instead of spawning a new process,
exec"replaces" the task of the current process -- that is, the shell (or script) process -- with the new command. In other words,
execreuses the same process to start a new task. In the context of the script, an
execimmediately "terminates" the script and starts the specified task.
getoptUNIX utility scans the positional parameters for the named arguments you specify. Here, the
dfiPRrvwlist looks for
-w. If another option appears,
getoptreturns a string of the options ending with the special string,
shiftcommand removes positional parameters from left to right. For example, if the command line were
myrm, -r -f -P file1 file2 file3,
shift 3would remove
file3are renumbered as the new
casestatement works like its counterparts in traditional programming languages: It compares its argument to each pattern in a list; when a match is found, the corresponding code executes. Much like in the shell,
*matches anything and can be used as the default action if no other match is found.
- The sigil,
$@, expands to all the (remaining) positional parameters.
$=, splits words at whitespace boundaries.
$=is useful when you have a long string and want to split the string into individual arguments. For instance, if the variable
xcontains the string
'-r -f'-- which is one word with five characters --
$=xbecomes two separate words,
Given those illuminations, you should now be able to dissect the script fully. Let's look at the code in blocks:
- The first block sets variables that are used throughout the script.
- The next block should look familiar: It prints a usage message if no arguments are provided. Why does it
execthe real rm utility? If you name this script "rm" and place it earlier in your
$PATH, it can act as a surrogate for /bin/rm. A bad option to the script is also a bad option to /bin/rm, so the script lets /bin/rm provide the usage message.
- The next block creates the backup directory if it does not exist. If the
mkdirfails, the script dies with an appropriate error message.
- The next block finds the
dasharguments in the list of positional arguments. If
$argshas a list of options. If
getoptfails, which occurs when it doesn't recognize an option, it prints an error message, and the script exits with a usage message.
- The following block captures all the options intended for rm in a string. Accumulation stops when the special
--, is encountered.
shiftremoves all the processed arguments from the argument list, leaving the list of files and directories to process.
- The block that begins
for fileis where each file or directory is copied for safekeeping in your personal "tomb." Each file's directory is copied verbatim (
-R) to the tomb, and it is suffixed with the current date and time to make sure the copy is unique and does not clobber a previous archived entry that shares the same name.
- Finally, the file or directory is removed using the same command-line options passed to the script.
However, if you happen to need the file or directory you just deleted (by accident?), you can look in your archive for a pristine copy!
Go forth and automate
The more you work with UNIX, the more likely you are to create scripts. A script saves the time and energy required to retype complex and long sequences of commands, preventing mistakes, too. The Web is full of helpful scripts that others have created for many purposes. Soon, you'll be posting your own incantations as well.
- Speaking UNIX: Check out other parts in this series.
- zsh Mailing List Archive: Read this list to learn more Z shell tricks and tips.
- Z shell: Download the latest version of Z shell from the Z shell home page.
- AIX® and UNIX articles: Check out other articles written by Martin Streicher.
- Search the AIX and UNIX library by topic:
- AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
- New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
- AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.
- IBM trial software: Build your next development project with software for download directly from developerWorks.
- developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
- Podcasts: Tune in and catch up with IBM technical experts.