Linux tip: Bash parameters and parameter expansions

Document options Document options requiring JavaScript are not displayed Discuss Rate this page Help us improve this content

Level: Intermediate

Ian Shields (ishields@us.ibm.com), Senior Programmer, IBM

16 May 2007

Do you sometimes wonder how to use parameters with your scripts, and how to pass them to internal functions or other scripts? Do you need to do simple validity tests on parameters or options, or perform simple extraction and replacement operations on the parameter strings? This tip helps you with parameter use and the various parameter expansions available in the bash shell.

The bash shell is available on many Linux® and UNIX® systems today, and is a common default shell on Linux. In this tip you will learn how to handle parameters and options in your bash scripts and how to use the shell's parameter expansions to check or modify parameters. This article focuses on bash, and the examples were all run on Linux systems with bash as the shell. However, the same expansions are available in many other shells, such as ksh, ash, or dash, and you may use them with these shells on other UNIX systems or even environments such as Cygwin. This tip builds on the tools covered in the earlier tip Linux tip: Bash test and comparison functions. Some of the material in this article is excerpted from the developerWorks tutorial LPI exam 102 prep: Shells, scripting, programming, and compiling, which covers many basic scripting techniques.

Passed parameters

Part of the beauty of functions and shell scripts is the ability to make a single function or script behave differently by passing parameters to it. In this section you will learn how to identify and use the parameters that are passed.

Inside a function or script, you can refer to the parameters using the bash special variables in Table 1. You prefix these with a $ symbol to reference them as with other shell variables.

Note: If you have more than 9 parameters, you cannot use $10 to refer to the tenth one. You must first either process or save the first parameter ($1), then use the shift command to drop parameter 1 and move all remaining parameters down 1, so that $10 becomes $9 and so on. The value of $# will be updated to reflect the remaining number of parameters. In practice, you will most often want to iterate over the parameters to a function or shell script, or a list created by command substitution using a for statement, so this constraint is seldom a problem.

Now you can define a simple function to do nothing more than tell you how many parameters it has and display them as shown in Listing 1.

                
[ian@pinguino ~]$ testfunc () { echo "$# parameters"; echo "$@"; }
[ian@pinguino ~]$ testfunc
0 parameters

[ian@pinguino ~]$ testfunc a b c
3 parameters
a b c
[ian@pinguino ~]$ testfunc a "b c"
2 parameters
a b c

Shell scripts handle parameters in the same way as functions. Indeed, you will frequently find yourself assembling scripts out of many small functions. Listing 2 shows a shell script, testfunc.sh, for the same simple task and the result of running it with one of the above inputs. Remember to mark your script executable using chmod +x.

                
[ian@pinguino ~]$ cat testfunc.sh
#!/bin/bash
echo "$# parameters"
echo "$@";
[ian@pinguino ~]$ ./testfunc.sh a "b c"
2 parameters
a b c

In Table 1 you discovered that the shell may refer to the list of passed parameters as $* or $@ and that whether you quote these expressions or not affects how they are interpreted. Whether you use $*, "$*", $@, or "$@", you won't see much difference in the output of the above function, but rest assured that when things become more complex, the distinctions will matter very much, as you wish to analyze parameters or perhaps pass some to other functions or scripts. Listing 3 shows a function that prints out the number of parameters and then prints the parameters according to these four alternatives. Listing 4 shows the function in action. The default IFS variable uses a space as its first character, so Listing 4 adds a vertical bar as the first character of the IFS variable to show more clearly where this character is being used in the expansion of "$*".

                
[ian@pinguino ~]$ type testfunc2
testfunc2 is a function
testfunc2 ()
{
    echo "$# parameters";
    echo Using '$*';
    for p in $*;
    do
        echo "[$p]";
    done;
    echo Using '"$*"';
    for p in "$*";
    do
        echo "[$p]";
    done;
    echo Using '$@';
    for p in $@;
    do
        echo "[$p]";
    done;
    echo Using '"$@"';
    for p in "$@";
    do
        echo "[$p]";
    done
}

                
[ian@pinguino ~]$ IFS="|${IFS}" testfunc2 abc "a bc" "1 2
> 3"
3 parameters
Using $*
[abc]
[a]
[bc]
[1]
[2]
[3]
Using "$*"
[abc|a bc|1 2
3]
Using $@
[abc]
[a]
[bc]
[1]
[2]
[3]
Using "$@"
[abc]
[a bc]
[1 2
3]

Study the differences carefully, particularly for the quoted forms and the parameters that include white space such as blanks or newline characters. From the single pair of [] characters, note that the expansion of "$*" is indeed a single word.

Back to top

Options and getopts

Traditional UNIX and Linux commands consider some of the passed arguments to be options. Historically, these were single character switches distinguished from other parameters by having a leading hyphen or minus sign. For convenience several options can be combined as in the command ls -lrt, which provides a long (option -l) directory listing in reverse order (option -r) sorted by modification time (option -t).

You can use the same technique with shell scripts and the getopts builtin command eases your task. To see how this works, consider the example script, testopt.sh, shown in Listing 5.

                
#!/bin/bash
echo "OPTIND starts at $OPTIND"
while getopts ":pq:" optname
  do
    case "$optname" in
      "p")
        echo "Option $optname is specified"
        ;;
      "q")
        echo "Option $optname has value $OPTARG"
        ;;
      "?")
        echo "Unknown option $OPTARG"
        ;;
      ":")
        echo "No argument value for option $OPTARG"
        ;;
      *)
      # Should not occur
        echo "Unknown error while processing options"
        ;;
    esac
    echo "OPTIND is now $OPTIND"
  done

The getopts command uses a couple of predetermined variables. The OPTIND variable is initially set to 1. Thereafter it contains the index of the next parameter to be processed. The getopts command returns true if an option is found, so a common option processing paradigm uses a while loop with a case statement as in this example. The first argument to getopts is a list of option letters to be recognized, in this case, p and r. A colon (:) after an option letter indicates that the option requires a value; for example, a -f option might be used to indicate a file name, as in the tar command. The leading colon in this example tells getopts to be silent and suppress the normal error messages, as this script will provide its own error handling.

The second parameter, optname in this example, is the name of a variable which will receive the name of the option found. If an option is expected to have a value, the value, if present, will be placed in the variable OPTARG. In silent mode, either of the following two error conditions may occur.

1. If an unrecognized option is found, then optname will contain a ? and OPTARG will contain the unknown option.
2. If an options that requires a value is found but the value is not, then optname will contain a : and OPTARG will contain the name of the option whose argument is missing.

If not in silent mode, these errors cause a diagnostic error message and OPTARG is unset. The script may use the values of ? or : in optname to detect and possibly handle the error.

Listing 6 shows two examples of running this simple script.

                
[ian@pinguino ~]$ ./testopt.sh -p -q
OPTIND starts at 1
Option p is specified
OPTIND is now 2
No argument value for option q
OPTIND is now 3
[ian@pinguino ~]$ ./testopt.sh -p -q -r -s tuv
OPTIND starts at 1
Option p is specified
OPTIND is now 2
Option q has value -r
OPTIND is now 4
Unknown option s
OPTIND is now 5

If you need to do so, you may pass a set of arguments to getopts for evaluation. You will need to reset OPTIND to 1 yourself if you call getopts for a new set of arguments in a script where you have already used it with other arguments. See the man or info pages for bash for additional details.

Back to top

Parameter expansions

So now you have looked at how parameters are passed into a function or script and how to identify options. Now to start processing the options and parameters. It would be nice to know what arguments are left after processing the options. Perhaps then you will need to validate parameter values, or assign defaults to missing parameters. This section will introduce some of the parameter expansions that are available in bash. Of course, you also have the full power of the Linux or UNIX commands such as sed or awk for more complex jobs, but you should also know how to use shell expansions.

Let's start off by building a script from the option analysis and parameter analysis functions that you saw above. Our testargs.sh script is shown in Listing 7.

                
#!/bin/bash

showopts () {
  while getopts ":pq:" optname
    do
      case "$optname" in
        "p")
          echo "Option $optname is specified"
          ;;
        "q")
          echo "Option $optname has value $OPTARG"
          ;;
        "?")
          echo "Unknown option $OPTARG"
          ;;
        ":")
          echo "No argument value for option $OPTARG"
          ;;
        *)
        # Should not occur
          echo "Unknown error while processing options"
          ;;
      esac
    done
  return $OPTIND
}

showargs () {
  for p in "$@"
    do
      echo "[$p]"
    done
}

optinfo=$(showopts "$@")
argstart=$?
arginfo=$(showargs "${@:$argstart}")
echo "Arguments are:"
echo "$arginfo"
echo "Options are:"
echo "$optinfo"

Try running this script a few times to see how it works, then we'll examine it in more detail. Listing 8 shows some sample output.

                
[ian@pinguino ~]$ ./testargs.sh -p -q qoptval abc "def ghi"
Arguments are:
[abc]
[def ghi]
Options are:
Option p is specified
Option q has value qoptval
[ian@pinguino ~]$ ./testargs.sh -q qoptval -p -r abc "def ghi"
Arguments are:
[abc]
[def ghi]
Options are:
Option q has value qoptval
Option p is specified
Unknown option r
[ian@pinguino ~]$ ./testargs.sh "def ghi"
Arguments are:
[def ghi]
Options are:

Notice how the arguments are separated from the options. The showopts function analyzes the options as before, but uses the return statement to return the value of the OPTIND variable to the calling statement. The calling process assigns this value to a variable, argstart. This value is then used to select a subset of the original parameters, consisting of those parameters that were not processed as options. This is done using the parameter expansion
${@:$argstart}
Remember to use quotes around this expression to keep parameters with embedded spaces together as you saw back in Listing 2.

If you're new to scripts and functions, note the following:

1. The return statement returns an exit value from the showopts function which is accessed from the caller using $?. You might also use return values from functions with command such as test or while to control branching or looping.
2. Bash functions can include the optional word "function", for example:
   function showopts ()
   This is not part of the POSIX standard and is not supported on shells such as dash, so if you use it, don't make your shebang line
   #!/bin/sh
   as this will give you the system's default shell which might not work as you wish.
3. Function output, for example the output produced by the echo statements in the two functions here, is not printed, but made accessible to the caller. If it is not assigned to a variable, or it is not otherwise used as part of the calling statement, then the shell will attempt to execute it rather than display it.

Subsets and substrings

The general form of this expansion is ${PARAMETER:OFFSET:LENGTH}, where the LENGTH argument is optional. So, if you only want to select a specific subset of the script arguments, you can use the full version to say how many arguments to select. For example, ${@:4:3} refers to the three arguments starting at argument 4, namely arguments 4, 5 and 6. You can use this expansion to select individual parameters beyond those immediately accessible using $1 through $9. ${@:15:1} is a way of accessing parameter 15 directly.

You can use this expansion with individual parameters as well as the whole parameter set represented by $* or $@. In this case, the parameter is treated as a string and the number refer to offset and length. For example, if the variable x has the value "some value" then
${x:3:5}
would have the value "e val", as shown in Listing 9.

                
[ian@pinguino ~]$ x="some value"
[ian@pinguino ~]$ echo "${x:3:5}"
e val

Lengths

You have already seen that $# refers to the number of parameters and that the ${PARAMETER:OFFSET:LENGTH} expansion applies to both individual parameters as well as $* and $@, so it should come as no surprise that an analogous construct, ${#PARAMETER} is available to determine the length of an individual parameter. The simple testlength function shown in Listing 10 illustrates this. Try it out yourself.

                
[ian@pinguino ~]$ testlength () { for p in "$@"; do echo ${#p};done }
[ian@pinguino ~]$ testlength 1 abc "def ghi"
1
3
7

Pattern matching

Parameter expansion also includes some pattern matching capabilities with the same wild card capabilities as used in filename expansion or globbing. Note that this is not regular expression matching as used by grep.

Listing 11 shows some basic usage of the pattern matching expansions.

                
[ian@pinguino ~]$ x="a1 b1 c2 d2"
[ian@pinguino ~]$ echo ${x#*1}
b1 c2 d2
[ian@pinguino ~]$ echo ${x##*1}
c2 d2
[ian@pinguino ~]$ echo ${x%1*}
a1 b
[ian@pinguino ~]$ echo ${x%%1*}
a
[ian@pinguino ~]$ echo ${x/1/3}
a3 b1 c2 d2
[ian@pinguino ~]$ echo ${x//1/3}
a3 b3 c2 d2
[ian@pinguino ~]$ echo ${x//?1/z3}
z3 z3 c2 d2

Back to top

Putting it together

Before we cover a few remaining points, let's look at a real world example of parameter handling. I build the developerWorks author package (see Resources on a Linux system using a bash script. We store the various files I need in subdirectories of a library we'll call developerworks/library. The most recent release was version 5.7, so the schema files are found in developerworks/library/schema/5.7, the XSL files are found in developerworks/library/xsl/5.7, and the sample templates are in developerworks/library/schema/5.7/templates. Clearly, a single parameter providing the version, 5.7 in this case, would be sufficient for the script to build paths to all these files. So the script takes a -v parameter which must have a value. Validation of this parameter is performed later by constructing the path and then checking that it exists using [ -d "$pathname" ]

This works fine for production builds, but during development, the files are stored in different directories:

• developerworks/library/schema/5.8/archive/test-5.8/merge-0430
• developerworks/library/xsl/5.8/archive/test-5.8/merge-0430 and
• developerworks/library/schema/5.8/archive/test-5.8/merge-0430/templates-0430

where the version is now 5.8, and the 0430 represents month and day of the latest test version.

To handle this, I added a parameter, -p, which contains the supplemental piece of path information—archive/test-5.8/merge-0430. Now I, or someone else might forget the leading or trailing slash and some Windows user might use back slashes instead of regular slashes, so I decided to just handle that in the script. Also, you'll notice that the path to the template directory contains the date twice, so I needed to somehow pick off the date, -0430 in this run.

Listing 12 shows the code I used to handle the two parameters and cleanse the partial path according to these requirements. The value of the -v option is stored in the ssversion variable while the cleansed version of the -p variable is stored in pathsuffix and the date, complete with leading hyphen is stored in datesuffix. The comments explain what is happening at each step. You'll recognize several parameter expansions, including length, substring, pattern matching, and pattern replacement in even this short piece of script.

                
while getopts ":v:p:" optval "$@"
  do
    case $optval in
      "v")
        ssversion="$OPTARG"
      ;;
      "p")
        # Convert any backslashes to forward slashes
        pathsuffix="${OPTARG//\\//}"
        # Ensure this is a leading / and no trailing one
        [ ${pathsuffix:0:1} != "/" ] && pathsuffix="/$pathsuffix"
        pathsuffix=${pathsuffix%/}
        # Strip off the last hyphen and what follows
        dateprefix=${pathsuffix%-*}
        # Use the length of what remains to get the hyphen and what follows
        [ "$dateprefix" != "$pathsuffix" ] && datesuffix="${pathsuffix:${#dateprefix}}"
        ;;
      *)
        errormsg="Unknown parameter or option error with option - $OPTARG"
        ;;
    esac
  done

As with most things in Linux, and perhaps programming in general, this is not the only solution for this problem, but it does illustrate a more practical use of the expansions you have learned about.

Back to top

Default values

In the last section you saw how to assign option values to variables such as ssversion or pathsuffix. In this case, an empty version will be detected later and an empty path suffix is what happens for production builds, so that's acceptable. What if you need to assign default values for parameters that aren't specified? The shell expansions shown in Table 3 will help you with this task.

Listing 13 illustrates these expansions and the differences between them.

                
[ian@pinguino ~]$ unset x;y="abc def"; echo "/${x:-'XYZ'}/${y:-'XYZ'}/$x/$y/"
/'XYZ'/abc def//abc def/
[ian@pinguino ~]$ unset x;y="abc def"; echo "/${x:='XYZ'}/${y:='XYZ'}/$x/$y/"
/'XYZ'/abc def/'XYZ'/abc def/
[[ian@pinguino ~]$ ( unset x;y="abc def"; echo "/${x:?'XYZ'}/${y:?'XYZ'}/$x/$y/" )\
>  >so.txt 2>se.txt
[ian@pinguino ~]$ cat so.txt
[ian@pinguino ~]$ cat se.txt
-bash: x: XYZ
[[ian@pinguino ~]$ unset x;y="abc def"; echo "/${x:+'XYZ'}/${y:+'XYZ'}/$x/$y/"
//'XYZ'//abc def/

Back to top

Passing parameters

There are some subtleties to parameter passing that can trip you if you are not careful. You already know about the importance of quoting and how quoting affects use of $* and $@, but consider the following case. Suppose you want a script or function that operates on all files or perhaps directories in the current working directory. For this illustration, consider the ll-1.sh and ll-2.sh scripts shown in Listing 14.

                
#!/bin/bash
# ll-1.sh
for f in "$@"
  do
    ll-2.sh "$f"
  done

#!/bin/bash
ls -l "$@"

The script ll-1.sh simply passes each of its parameters in turn to the script ll-2.sh and ll-2.sh does a long directory listing of the passed parameter. My test directory contains a pair of empty files, "file1" and "file 2". Listing 15 shows the output of the scripts.

                
[ian@pinguino test]$ ll-1.sh *
-rw-rw-r-- 1 ian ian 0 May 16 15:15 file1
-rw-rw-r-- 1 ian ian 0 May 16 15:15 file 2

So far, so good. But if you forget to use the * parameter, the script doesn't do anything. It doesn't automatically work on the contents of the current working directory as, for example, the ls command does. A simple fix might be to add a check for this condition in ll-1.sh and use the output of the ls command to generate the input for ll-2.sh when nothing is provided to ll1-sh. A possible solution is shown in Listing 16.

                
#!/bin/bash
# ll-1.sh - revision 1
for f in "$@"
  do
    ll-2.sh "$f"
  done
[ $# -eq 0 ] && for f in "$(ls)"
  do
    ll-2.sh "$f"
  done

Note that we carefully quoted the output of the ls command to make sure that we handled "file 2" correctly. Listing 17 shows the result of running the new ll-1.sh with the * and without.

                
[ian@pinguino test]$ ll-1.sh *
-rw-rw-r-- 1 ian ian 0 May 16 15:15 file1
-rw-rw-r-- 1 ian ian 0 May 16 15:15 file 2
[ian@pinguino test]$ ll-1.sh
ls: file1
file 2: No such file or directory

Are you surprised? When you are passing parameters, particularly if they are the result of command output, things can be tricky. The clue here is in the error message which shows that the filenames are separated by newline characters. There are many ways of handling this problem, but one simple way is to use the read builtin as shown in Listing 18. Try it for yourself.

                
#!/bin/bash
# ll-1.sh - revision 2
for f in "$@"
  do
    ll-2.sh "$f"
  done
[ $# -eq 0 ] && ls | while read f
  do
    ll-2.sh "$f"
  done

The moral of the story is that attention to detail and testing with odd inputs will make your scripts more reliable. Good luck!

Back to top

Learn more

If you'd like to know more about bash scripting in Linux, read the tutorial "LPI exam 102 prep: Shells, scripting, programming, and compiling," from which part of this article was excerpted. To learn about other commands that you can use to analyze text such as parameter values, see the tutorial "LPI exam 101 prep: GNU and UNIX commands," You'll find other Resources below. Finally, don't forget to rate this page.

Resources

• Review the tutorial "LPI exam 102 prep: Shells, scripting, programming, and compiling" (developerWorks, Jan 2007) for more information on bash shell customization and scripting in Linux. It is part of a larger LPI exam prep tutorial series covering Linux fundamentals and preparing you for system administrator certification.
  
  
• Read these developerWorks articles for other ways to work with bash:
  • Bash by example, Part 1: Fundamental programming in the Bourne again shell (bash)
  • Bash by example, Part 2: More bash programming fundamentals
  • Bash by example, Part 3: Exploring the ebuild system
  • System Administration Toolkit: Get the most out of bash
  • Working in the bash shell
  
  
• "Shell Command Language" defines the shell command language as specified by The Open Group and IEEE.
  
  
• In the developerWorks Linux zone, find more resources for Linux developers, including our readers' favorite Linux articles and tutorials over the last month.
  
  
• Stay current with developerWorks technical events and Webcasts.

• Order the SEK for Linux, a two-DVD set containing the latest IBM trial software for Linux from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
  
  
• With IBM trial software, available for download directly from developerWorks, build your next development project on Linux.
  
  

• Participate in the discussion forum.
  
  
• Get involved in the developerWorks community through our developer blogs, forums, podcasts, and community topics in our new developerWorks spaces.
  
  

About the author

		Ian Shields works on a multitude of Linux projects for the developerWorks Linux zone. He is a Senior Programmer at IBM at the Research Triangle Park, NC. He joined IBM in Canberra, Australia, as a Systems Engineer in 1973, and has since worked on communications systems and pervasive computing in Montreal, Canada, and RTP, NC. He has several patents and has published several papers. His undergraduate degree is in pure mathematics and philosophy from the Australian National University. He has an M.S. and Ph.D. in computer science from North Carolina State University. You can contact Ian at ishields@us.ibm.com.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!

Back to top