Getting more done with less: Using scripts and utilities to run commands across all your IBM AIX servers quickly

System administrators are increasingly required to manage more servers, decreasing the amount of time they are able to dedicate to each server. Time-saving utilities are crucial in ensuring efficiency in managing these servers. Simple, serially run one-line Secure Shell (SSH) loops and parallel-run commands using utilities, such as dsh and parallel SSH, drastically improve efficiency, especially in large-enterprise environments.

Brian K. Smith (ixbrian@gmail.com), System Administrator

Photo of Brian SmithBrian Smith is a UNIX/Linux system administrator and lives in Colorado Springs, CO with his wife and two children. He has worked with IBM AIX for the last seven years. His areas of expertise include AIX, IBM PowerVM, shell scripting, Perl programming, and Linux. You can find more information about his work at Brian Smith's AIX / UNIX / Linux / Open Source blog.



08 July 2013

Also available in Chinese Russian

Introduction

In most companies, system administrators are being asked to do more with less. The number of IBM® AIX® instances that a system administrator must manage has grown drastically as virtualization technologies, such as IBM PowerVM®, have taken hold. PowerVM allows a single piece of hardware to run dozens or even hundreds of IBM AIX logical partitions (LPARs). Years ago, system administrators might have been responsible to manage only a couple of dozen AIX instances, while in today's environments it is common to find system administrators with hundreds of AIX LPARs to manage.

If you are in a large AIX environment with many LPARs, it isn't practical to log in to each one when changes are needed or when you need to collect information from each server. In these large environments, the best option is to carefully use technology that allows you to quickly run a command or script across many servers. This article covers several options for running commands and scripts across a large number of servers.


SSH key overview

All the options discussed in this article rely on SSH keys being set up between a central management server and the rest of the servers.

The first step is to pick a central management server to use. You should pick a server that is as secure as possible and has network access to all other servers in your environment. One option is to create a new LPAR just for this purpose, and lock it down as much as possible. Other potential options are a Network Installation Manager (NIM) server or a system monitoring server.

The next step is to pick a user account to use. Many tasks that you might likely want to automate require root privileges. However, it is not recommended to directly use the root account due to potential security issues. A good alternative is to create a normal user account and grant it root access with a utility, such as sudo.

After you have identified which server you will use as your central management server and have identified a user account, the next step is to generate an SSH key. There are many tutorials available for SSH keys, so this article will not elaborate on the topic. Essentially, you need to use the ssh-keygen command to create a private and public key on the central management server. The private key is only on the central management server, while the public key can be copied to all other servers.

You need to establish a new user account on each server, create a ".ssh/authorized_keys" file in the user's home directory and put the public key in the file. If you plan to run commands with root-level access, you should also set up sudo or something similar to grant the user additional access. If you use sudo, use the "NOPASSWD" option so that sudo does not prompt the user for a password.

To verify that the SSH keys are working, you should attempt to use ssh to run a remote command such as hostname on one of the servers. For example, if the server name is "server1", you should be able to type ssh server1 hostname, and the response should be server1. Using sudo, you should be able to validate this by running the ssh server1 sudo su -c whoami, command which should return root.


Running a single command on multiple servers using a simple for loop

One method for running a command remotely is a simple one-line loop command. The names of the servers on which the command will be run are read out of a file. Each line of the file has a single server name. Listing 1 shows an example of the for loop command line to run a command on multiple servers.

Listing 1. Example of an ssh for loop
for server in `cat serverlist`; do printf "%-20s" $server; 
ssh -q -o "BatchMode yes" $server oslevel -s 2>&1; echo; done | grep -v ^$

Let's break this down in detail:

  • for server in `cat serverlist`; do – Creates a loop such that every server in the serverlist file will be processed by the loop and the server name will be set as the $server variable.
  • printf "%-20s" $server; – Displays the server's name, left justified, with padded spaces. This will cause the server names and output to all line up properly.
  • ssh -q -o "BatchMode yes" $server oslevel -s 2>&1; – Calls the ssh command with a couple of options. The -q option prevents the login banner from being displayed. The -o "BatchMode yes" option causes SSH to run in a batch mode and not prompt the user for anything. The $server will be changed to the current server's name by the for loop. In this example, the oslevel -s is the command that we need to run on the remote server. The 2>&1 option is used to redirect the standard error to the standard output. Without this, the output might be out of order as the standard error is not buffered.
  • echo; – Ensures that whether or not the command produces an output, the cursor is moved to the following line in order for the next server name in the loop to line up in its correct place. It does this by using a carriage return at the end of each line, setting up the correct format if there is no output returned following the command. If the command did produce an output, this causes an extra blank line, however, these lines will be removed in the next step.
  • done | grep -v ^$ – Ends the for loop, then filters the output for any blank lines and removes them.

Figure 1 shows an example of this command line and the output. The file serverlist has a list of server names (one per line) and the command that will be run on each server is oslevel -s.

Figure 1. An example of a for loop command line and its output
example of this command line and the output

Figure 2 shows an example of a for loop command line using sudo and its output.

Figure 2. An example of a for loop command line using sudo for root access and its output
An example of a for loop command line using sudo for root access and its output

In Figure 2, a user account is deleted on multiple servers. Root-level access to run rmuser is granted by sudo. There is no output on aixtest1 and aixtest3, which means that the user was deleted. On aixtest2, aixtest4, and aixtest5 the rmuser reports an error because the user does not exist.

The downside to this for loop method is that each command is run on one server at a time (serially). It generally takes a minimum of ¼ of a second to establish an SSH connection and run a quick command. If there is network latency or the command being run takes longer, it might take multiple seconds per server or longer. If you are dealing with hundreds or thousands of servers, the for loop approach may take too long. However, it is very simple and effective for smaller scale cases.


Running a script on multiple servers using a simple for loop

Many times, the task at hand requires more than just running a single command across all of the servers. In these cases, you might need to run a script on the remote servers.

One way to do this is to copy the script to each server [using Secure Copy Protocol (SCP) or another file protocol], remotely run the script with SSH, then optionally delete the script with another SSH connection. However, this is very inefficient as it requires multiple connections to each server and requires copying the script file to each server.

A much better way to do this is to remotely run either a shell interpretor or a script interpretor (such as Perl) on the remote server and redirect the script you want to run from the local server. Using this method, the script is run on the remote server without ever having to copy the script file to the remote server.

Listing 2 shows the contents of the info.sh script that exists on the central management server. This is a simple script that detects whether the server is AIX or not. If it is AIX, the OS level, CPU mode, entitled CPU capacity, and the amount of memory are displayed.

Listing 2. The info.sh script that exists on the central management server
#!/usr/bin/ksh

if [ "`uname`" = "AIX" ]; then
printf "This server is running AIX level           : "
oslevel -s
echo "CPU and Memory info: "
lparstat -i | egrep "^Mode|^Entitled Capacity  |^Online Memory"
echo "------------------------------"
else
echo "This is not an AIX server"
fi

Listing 3 shows a for loop similar to the first one we covered. However, in this for loop, the remote command we are running is the Korn shell (ksh) interpretor, and we are redirecting the input to ksh from the info.sh file. This causes the info.sh script to be run on each remote server without actually having to copy the script to the servers.

Listing 3. An example of a for loop command to run local “info.sh” script on each server listed in the servelist file
for server in `cat serverlist`; do printf "%-20s\n" $server; 
ssh -q -o "BatchMode yes" $server 'ksh 2>&1' < ./info.sh ; echo; done | grep -v ^$

Listing 3 shows an example of a for loop command to run local “info.sh” script on each server listed in the servelist file without having to copy script to each server.

Figure 3 shows that when this command is run, the info.sh script is run on each server specified in the serverlist file.

Figure 3. Using a simple for loop to run a script on remote servers using input redirection
Using a simple for loop to run a script on remote servers using input redirection

This is an efficient way to run scripts on a large number of servers without ever actually copying the script to each server. However, the script is run serially so it might not be practical if the script takes longer time to complete or you have a large number of servers that the script will be run on.


Comparing serial and parallel running of scrips

The previously covered for loops run commands on the remote servers serially, which means one after another. There are also options to run commands in parallel, meaning the central management server starts commands on multiple servers simultaneously. For example, suppose you need to run a command on 35 servers and each server takes 1 second for the SSH connection to be established and the command to be run. If run serially, it would take 35 seconds for this to complete. If run in parallel running 16 servers simultaneously, it would take approximately 3 seconds (35 servers / 16 run simultaneously = approximately 3 seconds). There is a significant speedup when running commands in parallel on a large number of servers. Figure 4 shows a diagram that compares serial and parallel running of commands.

Figure 4. Diagram showing difference between serial and parallel execution of scripts.
Diagram showing difference between serial and parallel execution of scripts

Running a command on multiple servers using the dsh command

AIX includes a dsh command that is designed to run a command on multiple servers in parallel. By default, dsh can run up to 64 commands in parallel (this is customizable using the -f flag).

The first step in using dsh is to make sure that it is installed on your central management server. This can be checked by running which dsh; . If a path is returned to dsh, the server has it installed. If not, you need to either install the csm.dsh file set for AIX 6.1 or the dsm.dsh and dsm.core file sets for AIX 7.1. The appropriate file set for AIX 6.1 or AIX 7.1 should be on your AIX product installation media.

The next step is to set some environment variables that instruct dsh how to operate.

Listing 4. Environment variables to set for dsh
export DSH_NODE_LIST="/home/user/serverlist"
export DSH_NODE_RSH=/usr/bin/ssh
export DSH_NODE_OPTS="-q -o BatchMode=yes"

Listing 4 shows examples of the variables that must be set before using dsh.

The DSH_NODE_LIST variable should be set to a file name that specifies the servers on which you need to run the commands, one server per line.

The DSH_NODE_RSH variable is set to /usr/bin/ssh to instruct dsh to use SSH as the connection protocol.

The DSH_NODE_OPTS variable is set to -q -o BatchMode=yes to instruct the SSH client to not display the login banner and not prompt the user for anything.

After these variables are set, a command can be run on multiple servers by running dsh , followed by the command line that should be run on each server. Figure 5 shows the oslevel command being run on several servers using dsh.

Figure 5. "oslevel" command being run on several servers using "dsh"
oslevel command being run on several servers using dsh

Another useful functionality of dsh is the dshbak command. This can format the output of dsh in an easier to use format if the command returns a lot of output. To use dshbak, simply run the normal dsh command and pipe the output to dshbak as shown in Figure 6.

Figure 6. Using "dshbak" to format the output of "dsh"
Using

Using parallel SSH to run commands and scripts on multiple servers

Parallel SSH is a utility that is designed to run SSH commands on multiple servers in parallel and works similarly to the AIX dsh utility. However, parallel SSH has several advantages including: being cross platform (same utility works on Linux® and other UNIX® variants), being an open source, and having the capability to run scripts using redirection.

Parallel SSH is available in most Linux distribution repositories for easy installation.

To use parallel SSH on AIX you will first need to install version 2.4 or later of the Python scripting language. I found that the parallel SSH setup.py installation program can be difficult to run on AIX, and it is not necessary. Instead, you can manually copy the parallel SSH binaries easily to any location from which you wantto run them.

To use parallel SSH to run a command on several servers, use a command similar to the one listed in listing 5.

Listing 5. Example command to have pssh to run a command on several servers

# pssh -h serverlist -i oslevel -s

The -h serverlist command specifies the list of servers in the serverlist file(one server name per line). The -i option specifies that the output must be inline on the screen. oslevel -s is the name of the command that should be run on each of the servers.

Figure 7 shows an example output of running the oslevel command on several servers using pssh.

Figure 7. Running the "oslevel" command on several servers using "pssh"
Running the

Using pssh, it is also possible to quickly and easily run a script on multiple servers. This can be done without copying the script to each server using input direction. This is basically the same method covered earlier in the article under the Running a script on multiple servers using a simple for loop section. However, in this case, it is done in parallel and thus it is much faster.

To run the info.sh script (shown in Listing 2) using pssh, use the command line shown in Listing 6.

Listing 6. Using "pssh" to run a local script on several servers

# pssh -h serverlist -i -I < info.sh

The -I switch tells pssh to read the command to be run from the standard input. The < info.sh option redirects the info.sh script to the standard input of pssh, which in effect runs this script on each server without actually having to copy the script to each server.

Figure 8 shows the output of running the info.sh script on several servers using pssh.

Figure 8. Running the info.sh script on several servers using "pssh"
Running the info.sh script on several servers using pssh

Going further

The AIX dsh suite also includes a command named dcp that allows files to be copied to or from multiple servers in parallel. This can make pushing out files to many LPAR's a very quick and easy process.

Parallel SSH also includes similar copy commands, which both run in parallel. The pscp command is used to copy files to servers, while pslurp is used to copy files from servers.

Figure 9 shows the pscp command in use to copy the info.sh file into the /tmp directory on several servers.

Figure 9. Using "pscp" to copy info.sh to the /tmp directory on several servers
Using

Conclusion

As demonstrated, there are several methods to run commands or scripts on multiple servers. System administrators can use both simple one-line loops that serially run the command or script on each server, and also utilities that run commands in parallel (such as dsh and parallel SSH). Using these tools and methods can allow system administrators to be more efficient, getting more done in less time.


Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=935606
ArticleTitle=Getting more done with less: Using scripts and utilities to run commands across all your IBM AIX servers quickly
publish-date=07082013