Speaking UNIX: More shell scripting techniques

Once more, this time with feeling

Like other UNIX® operating systems and Linux®, the IBM® AIX® operating system has several powerful tools that arm systems administrators, developers, and users to tackle day-to-day tasks and to simplify their or their customers' business and life. One such tool in UNIX is the ability to write shell scripts to automate tasks, simplifying difficult or long and tedious jobs.

Share:

Adam Cormany, National Data Center Manager, Scientific Games Corporation

Adam Cormany is currently the manager of the National Data Center, but he has also been a UNIX systems engineer, a UNIX administrator, and operations manager for Scientific Games Corporation. Adam has worked extensively with AIX as well as in Solaris and Red Hat Linux administration for more than 10 years. He is an IBM eServer-Certified Specialist in pSeries AIX System Administration. In addition to administration, Adam has extensive knowledge of shell scripting in Bash, CSH, and KSH as well as programming in C, PHP, and Perl. You can reach him at acormany@yahoo.com.



09 September 2008

Also available in Chinese

Like other UNIX operating systems and Linux, the IBM AIX operating system has several powerful tools that arm systems administrators, developers, and users to tackle day-to-day tasks and to simplify their or their customers' business and life. One such tool in UNIX is the ability to write shell scripts to automate tasks, simplifying difficult or long and tedious jobs.

Although some who have worked on UNIX for a couple of years have dabbled in shell scripting, they are still most likely learning the ins and outs of the operating system and haven't yet mastered scripting. This article provides tips to those wanting to learn more about shell scripting and how to begin writing more advanced scripts. It provides basic fundamentals of programming in general, including how to simplify the script, how to keep the script as flexible as possible, how to write a clean script, documenting inside the script, and debugging the script.

Keep it simple

One issue people run into when learning how to shell script well is duplicating work they've already done in a script. Rather than copying their work and changing a couple of hard-coded values, they could simply create a function to handle the work for both areas of the script. Creating centralized functions also standardizes and provides for a uniform script. If a function works in one area of the script, it's a safe bet it will work elsewhere in the script, as well.

For example, the script shown in Listing 1 should be condensed and simplified considerably into a smaller and more clean-looking program.

Listing 1. Example of a script that can be simplified
#!/usr/bin/ksh

if [[ $# -lt 2 ]]
then
  echo "Usage: ${0##*/} <file name #1> <file name #2>
  exit 0
fi

if [[ ! -f "${1}" ]]
then
  echo "Unable to find file '${1}'"
  exit 1
fi

if [[ ! -r "${1}" ]]
then
  echo "Unable to read file '${1}'"
  exit 2
fi

gzip ${1}
ls -l ${1}.gz


if [[ ! -f "${2}" ]]
then
  echo "Unable to find file '${2}'"
  exit 1
fi

if [[ ! -r "${2}" ]]
then
  echo "Unable to read file '${2}'"
  exit 2
fi

gzip ${2}
ls -l ${2}.gz

This script looks awful! (Thankfully, it's only an example.) The script should be condensed as much as possible. To ease the eyes of readers, Listing 2 provides a cleaner version.

Listing 2. Condensed example of the Listing 1 script
#!/usr/bin/ksh

exit_msg() {
  [[ $# -gt 1 ]] && echo "${0##*/} (${1}) - ${2}"
  exit ${1:-0}
}

[[ $# -lt 2 ]] && exit_msg 0 "Usage: ${0##*/} <file name #1> <file name #2>

for _FNAME in $@
do
  [[ ! -f "${_FNAME}" ]] && exit_msg 1 "Unable to find file '${_FNAME}'"
  [[ ! -r "${_FNAME}" ]] && exit_msg 2 "Unable to read file '${_FNAME}'"

  gzip ${_FNAME}
  ls -l ${_FNAME}.gz
done

Notice the differences? By adding a simple function to display a message and exit with the appropriate return code as well as moving everything into a for loop, the script looks cleaner and is easier to understand.


Keep it flexible

Another issue that novices to programming and shell scripting encounter is hard-coding static values into a program or shell script. This limits the flexibility of a script and, in short, is bad programming. To keep administrators or developers from constantly having to modify a script to work a certain way with other values, use variables and supply arguments to the script or function.

For example, Listing 3 is an example of a poorly written and inflexible script.

Listing 3. Example of a script that isn't flexible
#!/bin/bash

if [[ -f /home/cormany/FileA ]]
then
  echo "Found file '/home/cormany/FileA'"
elif [[ -f /home/cormany/DirA/FileA ]]
then
  echo "Found file '/home/cormany/DirA/FileA'"
else
  echo "Unable to find file FileA"
fi

This script works but is limited to searching for a single file in only two locations.

To expand on the idea, the script in Listing 4 provides the same feel but allows users to search for any file in any location.

Listing 4. Making a script more flexible
#!/bin/bash

exit_msg() {
  [[ $# -gt 1 ]] && echo "${0##*/} (${1}) - ${2}"
  exit ${1:-0}
}

[[ $# -lt 2 ]] && exit_msg 1 "Usage: ${0##*/} <file name> <location>"

_FNAME="${1}"
_DNAME="${2}"

[[ ! -d "${_DNAME}" ]] && exit_msg 2 "Unable to read or find directory '${_DNAME}'"

if [[ -f "${_DNAME}/${_FNAME}" ]]
then
  exit_msg 0 "Found file '${_DNAME}/${_FNAME}'"
else
  exit_msg 3 "Unable to find file '${_DNAME}/${_FNAME}'"
fi

This example is more flexible, as it allows users to enter any file they want to search for in any directory.


Give them options

When writing a shell script, some users may say, "It would be great to have this!" or "I'd love to be able to do that!" while others may not necessarily agree and may not want to perform the same action. People love options, so why not give them some? The built-in shell command getopt does just that job.

Listing 5 provides a basic example of how getopt works in AIX.

Listing 5. getopt example
#!/usr/bin/ksh

_ARGS=`getopt -o x --long xxxxx -n ${0##*/} -- "$@"`
while [[ $# -gt 0 ]]
do
  case "${1}" in
    -x|--xxxxx)  echo "Arg x hit!"; shift;;
            --)  shift; break;;
             *)  echo "Invalid Option: ${1}"; break;;
  esac
done

When executing the script containing getopt, named opttest, using a valid argument of -x-or --xxxxx, getopt recognizes the switch and executes the code within the case switch:

# ./hm -x
Arg x hit!

And again with an invalid switch or option:

# ./hm -a
Invalid Option: -a

Document, document, document

We all fall prey to this issue at one point in our careers. You've been asked to look at a script that was written 10 years ago by someone who no longer works for the company. No problem, you say? Typically, it won't be an issue, but if the script is complex, executes commands you’re not accustom to, is written in a different style than you're use to, or just doesn't work, it's extremely helpful to get some hints as to what the person was thinking when the script was originally created. Other times, you may be the person who developed a script that you thought was to be used once and never again. Or maybe you wrote a huge script that you've worked on for weeks and know inside and out, but if someone else looked at it, that person would be completely confused. These are just a few examples of why documentation is sometimes as important to the developer as the script is to the user.

Take the function shown in Listing 6 from a snippet of code.

Listing 6. Example of a script with no comments
confirm_and_exit() {
  [[ ${_DEBUG_LEVEL} -ge 3 ]] && set -x
  while [[ -z ${_EXIT_ANS} ]]
  do
    cup_echo "Are you sure you want to exit? [Y/N]  
        \c" ${_PROMPT_ERR_ROW} ${_PROMPT_ERR_COL}
    ${_TPUT_CMD} cnorm
    read ${_NO_EOL_FLAG:+${_READ_FLAG:-'-n'}} ${_NO_EOL_FLAG} _EXIT_ANS
    ${_TPUT_CMD} civis
  done

  case ${_EXIT_ANS} in
    [Nn])  unset _EXIT_ANS; return 0;;
    [Yy])  exit_msg 0 1 "Exiting Script";;
       *)  invalid_selection ${_EXIT_ANS}; unset _EXIT_ANS;;
  esac
  return 0
}

If you've worked with shell scripting for a while, you may be able to read this. However, someone just learning scripting will look at this and not know what this function is doing. Taking a few extra minutes to add comments in the script can make a world of difference. Listing 7 shows the same function with comments.

Listing 7. Example of a script with comments
#########################################
# function confirm_and_exit
#########################################
confirm_and_exit() {
  # if the debug level is set to 3 or higher, send every evaluated line to stdout
  [[ ${_DEBUG_LEVEL} -ge 3 ]] && set –x

  # Continue to prompt the user until they provide a valid answer
  while [[ -z ${_EXIT_ANS} ]]
  do
    # prompt user if they want to exit the script
    # cup_echo function calls tput cup <x> <y>
    # syntax:
    # cup_echo <string to display> <row on stdout to display> 
        <column on stdout to display>
    cup_echo "Are you sure you want to exit? [Y/N]  
        \c" ${_PROMPT_ERR_ROW} ${_PROMPT_ERR_COL}

    # change cursor to normal via tput
    ${_TPUT_CMD} cnorm

    # read value entered by user
    # if _NO_EOL_FLAG is supplied, use value of _READ_FLAG or “-n”
    # if _NO_EOL_FLAG is supplied, use value as characters aloud on read
    # assign value entered by user to variable _EXIT_ANS
    read ${_NO_EOL_FLAG:+${_READ_FLAG:-'-n'}} ${_NO_EOL_FLAG} _EXIT_ANS

    # change cursor to invisible via tput
    ${_TPUT_CMD} civis
  done

  # if user entered “n”, return to previous block of code with return code 0
  # if user entered “y”, exit the script
  # if user entered anything else, execute function invalid_selection
  case ${_EXIT_ANS} in
    [Nn])  unset _EXIT_ANS; return 0;;
    [Yy])  exit_msg 0 1 "Exiting Script";;
       *)  invalid_selection ${_EXIT_ANS}; unset _EXIT_ANS;;
  esac

  # exit function with return code 0
  return 0
}

This may seem tedious and a bit of overkill for such a small function, but the value of the comments to a novice shell scripter or someone just looking at the function can be invaluable.

Another extremely helpful use for comments in shell scripts is to explain what variables may be as well as explaining return codes.

The example in Listing 8 is from the beginning of a shell script.

Listing 8. Example of variables not documented
#!/usr/bin/bash
trap 'exit_msg 1 0 "Signal Caught. Exiting..."' HUP INT QUIT KILL ABRT
trap 'window_size_changed' WINCH

_MSG_SLEEP_TIME=3
_RETNUM_SIZE=6
_DEBUG_LEVEL=0
_TMPDIR="/tmp"
_SP_LOG="${0##*/}.log"
_SP_REQUESTS="${HOME}/sp_requests"
_MENU_ITEMS=15
LESS="-P LINE\: %l"
export _SP_REQUESTS _TMPDIR _SP_LOG _DB_BACKUP_DIR 
    export _DEBUG_LEVEL _NEW_RMSYNC _RMTOTS_OFFSET_COL

Again, it's quite difficult to understand what the traps are suppose to do or what the values are for in each variable. Unless you follow through a script completely, these variables mean nothing. In addition, there's no mention of any return codes used in the script. This can make troubleshooting a problem in a shell script much more difficult than it really needs to be. Adding a few comments to the lines in Listing 8 and adding a section on which return codes are used and their descriptions cuts down on confusion immensely. Take a look at Listing 9 below.

Listing 9. Example of variables documented
#!/usr/bin/bash
#########################################################################
# traps
#########################################################################
# trap when a user is attempting to leave the script
trap 'exit_msg 1 0 "Signal Caught. Exiting..."' HUP INT QUIT KILL ABRT
trap 'window_size_changed' WINCH                # trap when a user has resized the window
#########################################################################

#########################################################################
# defined/exported variables
#########################################################################
_MSG_SLEEP_TIME=3               # seconds to sleep for all messages
                                      # (if not defined, default will is 1 second)
_CUSTNUM_SIZE=6                   # length of a customer number in this location
                                      # (if not defined, default is 6)
_DEBUG_LEVEL=0                      # log debug messages. log level is accumulative
                                      # (i.e. 1 = 1, 2 = 1 & 2, 3 = 1, 2, & 3)
                                      # (if not defined, default is 0)
                                      # Log levels:
                                      #  0 = No messages
                                      #  1 = brief messages (start script, errors, etc)
                                      #  2 = environment setup (set / env)
                                      #  3 = set -x (A LOT of spam)
_TMPDIR="/tmp"          # directory to put work/tmp files
                                      # (if not defined, default is /tmp)
_SP_LOG="${0##*/}.log"   # log of script events
_SP_REQUESTS="${HOME}/sp_requests"
			# file to customer record requests,
                                       # also read at startup
_MENU_ITEMS=15        # default number of items to display per page
                                       # (it not defined, default is 10)
LESS="-P LINE\: %l"        # format 'less' prompt. MAN less if more info


# export the variables defined above
export _MSG_SLEEP_TIME _CUSTNUM_SIZE _DEBUG_LEVEL _TMPDIR 
    _SP_LOG _SP_REQUESTS _MENU_ITEMS
#########################################################################

Doesn't that look better? Everything is organized and detailed, and someone reading the script for the first time will have a better chance in understanding what the program is doing.


Debugging

You've finished writing the script, and now it's time to run the program for the first time. But when you execute the script, some unexpected errors are displayed. Now what? No one is perfect, and writing scripts from scratch to the point of not getting errors takes a lot of time and experience—most of the time, even that won't save someone from easily missing a character or transposing a couple of characters. Don't worry: The shells in AIX and other flavors of UNIX and Linux have it covered and can help with debugging.

For example, the shell script in Listing 10, named make_errors (and rightfully so), has been written and is ready to be executed.

Listing 10. Example of a script with errors
#!/bin/bash

_X=1
while [[ ${_X} -le 10 ]]
do
  [[ ${_X} -lt 5 ]] && echo "X is less than 5!

  _Y=`expr ${_X) + 1`

  if [[ ${_Y} -eq 6 ]]
    echo "Y is now equal to ${_Y}"
  fi

  _X=${_Y}
done

However, when executing the script initially, the following error is displayed:

# ./make_errors
./make_errors: line 11: unexpected EOF while looking for matching `"'
./make_errors: line 16: syntax error: unexpected end of file

One awesome debugging tool that you're already using and didn't know is Vim. Vim is a powerful text editor but also lends a helpful hand in debugging. If your .exrc or .vimrc file is set up to display colors for certain error conditions, Vim will do most of the work for you, as Figure 1, below, shows.

Figure 1. Debugging with Vim
Debugging with Vim

The first error (line 11: unexpected EOF while looking for matching `"') says something is going on with line 11, but after looking at the line, nothing looks wrong with it. Take a look at line 9. A double quotation mark (") is missing at the end of the string being echo-ed. This is a good example why you must look at the script as a whole when debugging. The line number displayed may not always be where the actual error originated. Line 11 is reporting an error because line 9 started to encapsulate a string with double quotation marks, but the string was not fully encapsulated until line 11. To correct the error, add double quotation marks to the end of line 9.

Something else is blatantly showing up as an error, as well. In line 11, after the value of variable _X is a closed parenthesis ()) highlighted in red. This is Vim doing your job for you and telling you that there's something wrong. It looks like the value of variable _X started with an opening curly bracket ({) but didn't end correctly with the closing curly bracket (}). Simply changing ) to } should do the trick.

Nice job: Two errors fixed so far. Run the script again and see what happens:

./make_errors: line 12: syntax error near unexpected token `fi'
./make_errors: line 12: `  fi'

Yet another error. This error says that there's a problem in line 12, but the line only has a fi completing the if statement. What's wrong with that? Keep in mind what happened with the previous error. Not all errors originate at the line at which the shell reports the problem. The shell is simply reporting where an error occurred, but this could mean that the cause of the error started before the shell reported a failure. It's a safe bet that in a script this small, the error may be in the actual if statement. Thinking back to basic shell scripting logic, an if statement consists of if, then, and fi. Looking at the conditional statement, it looks like there's a missing then. Simply add the then into the script. When you're finished, the script should look like Listing 11.

Listing 11. Corrected script from Listing 10
#!/bin/bash

_X=1
while [[ ${_X} -le 10 ]]
do
  [[ ${_X} -lt 5 ]] && echo "X is less than 5!"

  _Y=`expr ${_X} + 1`

  if [[ ${_Y} -eq 6 ]]
  then
    echo "Y is now equal to ${_Y}"
  fi

  _X=${_Y}
done

Run the script one more time:

# ./make_errors
X is less than 5!
X is less than 5!
X is less than 5!
X is less than 5!
Y is now equal to 6

Congratulations! The script now works as expected.

The set -x option

Sometimes, performing the basic troubleshooting steps of a shell script isn't as easy as it was in the above example. If all else fails, you're banging your head against the wall, and have no idea why the script is failing, the final step is to pull out the big guns! Ksh, Bash, and other modern shells include the switch -x in their set commands. Using the set –x option, every command evaluated is expanded and displayed to stdout. To make the evaluated code stand out, set –x uses the value of the PS4 variable and prepends it to every line of code displayed. Keep in mind this can be a lot of text to absorb, so be patient when looking through it.

Tone down the loop count on the previous example, add set -x to the beginning of the script as well as a comment, and execute it, as shown in Listing 12.

Listing 12. Example of set -x
#!/bin/bash

set -x

# loop through and display some test statements
_X=1
while [[ ${_X} -le 4 ]]
do
  [[ ${_X} -lt 2 ]] && echo "X is less than 2!

  _Y=`expr ${_X} + 1`

  if [[ ${_Y} -eq 3 ]]
  then
    echo "Y is now equal to ${_Y}"
  fi

  _X=${_Y}
done

Before you execute the script, change PS4 to something that will stand out:

# export PS4="DEBUG => "

Next, spam yourself with possibly very valuable information, as shown in Listing 13.

Listing 13. Example output from set -x
# ./make_errors
DEBUG => _X=1
DEBUG => [[ 1 -le 4 ]]
DEBUG => [[ 1 -lt 2 ]]
DEBUG => echo 'X is less than 2!'
X is less than 2!
DDEBUG => expr 1 + 1
DEBUG => _Y=2
DEBUG => [[ 2 -eq 3 ]]
DEBUG => _X=2
DEBUG => [[ 2 -le 4 ]]
DEBUG => [[ 2 -lt 2 ]]
DDEBUG => expr 2 + 1
DEBUG => _Y=3
DEBUG => [[ 3 -eq 3 ]]
DEBUG => echo 'Y is now equal to 3'
Y is now equal to 3
DEBUG => _X=3
DEBUG => [[ 3 -le 4 ]]
DEBUG => [[ 3 -lt 2 ]]
DDEBUG => expr 3 + 1
DEBUG => _Y=4
DEBUG => [[ 4 -eq 3 ]]
DEBUG => _X=4
DEBUG => [[ 4 -le 4 ]]
DEBUG => [[ 4 -lt 2 ]]
DDEBUG => expr 4 + 1
DEBUG => _Y=5
DEBUG => [[ 5 -eq 3 ]]
DEBUG => _X=5
DEBUG => [[ 5 -le 4 ]]

As you can see, there's a lot of information here: Every single command has been evaluated and executed. Also notice that the comment place in the shell script was not displayed in the debug information. This is because the string was a comment and therefore not executed after evaluation. Thankfully, nothing was wrong with this script after you made the original corrections!

One thing to keep in mind when using set -x is that if the script you're evaluating has internal functions, set -x will carry over to its child function if placed in the root body of the code. However, if set -x is placed only in the internal function, only code and child functions called within the internal function will carry in the debug option; the root body of the shell script will not, as it knows nothing of its child function calling the routine.


Conclusion

We're always improving on our method of programming, regardless of whether it's shell script, C, the Java™ language, or another language someone is working on. Stick to the basic rules of simplifying the work, keeping it clean and flexible, and documenting your code, and soon you'll be writing top-notch shell scripts with a little help from the debugging method you've learned. Good luck!

Resources

Learn

Get products and technologies

  • IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=336640
ArticleTitle=Speaking UNIX: More shell scripting techniques
publish-date=09092008