In most companies, system administrators are being asked to do more with less. The number of IBM® AIX® instances that a system administrator must manage has grown drastically as virtualization technologies, such as IBM PowerVM®, have taken hold. PowerVM allows a single piece of hardware to run dozens or even hundreds of IBM AIX logical partitions (LPARs). Years ago, system administrators might have been responsible to manage only a couple of dozen AIX instances, while in today's environments it is common to find system administrators with hundreds of AIX LPARs to manage.
If you are in a large AIX environment with many LPARs, it isn't practical to log in to each one when changes are needed or when you need to collect information from each server. In these large environments, the best option is to carefully use technology that allows you to quickly run a command or script across many servers. This article covers several options for running commands and scripts across a large number of servers.
SSH key overview
All the options discussed in this article rely on SSH keys being set up between a central management server and the rest of the servers.
The first step is to pick a central management server to use. You should pick a server that is as secure as possible and has network access to all other servers in your environment. One option is to create a new LPAR just for this purpose, and lock it down as much as possible. Other potential options are a Network Installation Manager (NIM) server or a system monitoring server.
The next step is to pick a user account to use. Many tasks that you might likely want to automate require root privileges. However, it is not recommended to directly use the root account due to potential security issues. A good alternative is to create a normal user account and grant it root access with a utility, such as sudo.
After you have identified which server you will use as your central management server and have identified a user account, the next step is to generate an SSH key. There are many tutorials available for SSH keys, so this article will not elaborate on the topic. Essentially, you need to use the
ssh-keygen command to create a private and public key on the central management server. The private key is only on the central management server, while the public key can be copied to all other servers.
You need to establish a new user account on each server, create a ".ssh/authorized_keys" file in the user's home directory and put the public key in the file. If you plan to run commands with root-level access, you should also set up sudo or something similar to grant the user additional access. If you use sudo, use the "NOPASSWD" option so that sudo does not prompt the user for a password.
To verify that the SSH keys are working, you should attempt to use
ssh to run a remote command such as
hostname on one of the servers. For example, if the server name is "server1", you should be able to type
ssh server1 hostname, and the response should be
server1. Using sudo, you should be able to validate this by running the
ssh server1 sudo su -c whoami, command which should return
Running a single command on multiple servers using a simple for loop
One method for running a command remotely is a simple one-line loop command. The names of the servers on which the command will be run are read out of a file. Each line of the file has a single server name. Listing 1 shows an example of the for loop command line to run a command on multiple servers.
Listing 1. Example of an ssh for loop
for server in `cat serverlist`; do printf "%-20s" $server; ssh -q -o "BatchMode yes" $server oslevel -s 2>&1; echo; done | grep -v ^$
Let's break this down in detail:
for server in `cat serverlist`; do– Creates a loop such that every server in the serverlist file will be processed by the loop and the server name will be set as the
printf "%-20s" $server;– Displays the server's name, left justified, with padded spaces. This will cause the server names and output to all line up properly.
ssh -q -o "BatchMode yes" $server oslevel -s 2>&1;– Calls the ssh command with a couple of options. The
-qoption prevents the login banner from being displayed. The
-o "BatchMode yes"option causes SSH to run in a batch mode and not prompt the user for anything. The
$serverwill be changed to the current server's name by the for loop. In this example, the
oslevel -sis the command that we need to run on the remote server. The
2>&1option is used to redirect the standard error to the standard output. Without this, the output might be out of order as the standard error is not buffered.
echo; –Ensures that whether or not the command produces an output, the cursor is moved to the following line in order for the next server name in the loop to line up in its correct place. It does this by using a carriage return at the end of each line, setting up the correct format if there is no output returned following the command. If the command did produce an output, this causes an extra blank line, however, these lines will be removed in the next step.
done | grep -v ^$– Ends the for loop, then filters the output for any blank lines and removes them.
Figure 1 shows an example of this command line and the output. The file serverlist has a list of server names (one per line) and the command that will be run on each server is
Figure 1. An example of a for loop command line and its output
Figure 2 shows an example of a for loop command line using sudo and its output.
Figure 2. An example of a for loop command line using sudo for root access and its output
In Figure 2, a user account is deleted on multiple servers. Root-level access to run rmuser is granted by sudo. There is no output on
aixtest3, which means that the user was deleted. On
rmuser reports an error because the user does not exist.
The downside to this for loop method is that each command is run on one server at a time (serially). It generally takes a minimum of ¼ of a second to establish an SSH connection and run a quick command. If there is network latency or the command being run takes longer, it might take multiple seconds per server or longer. If you are dealing with hundreds or thousands of servers, the for loop approach may take too long. However, it is very simple and effective for smaller scale cases.
Running a script on multiple servers using a simple for loop
Many times, the task at hand requires more than just running a single command across all of the servers. In these cases, you might need to run a script on the remote servers.
One way to do this is to copy the script to each server [using Secure Copy Protocol (SCP) or another file protocol], remotely run the script with SSH, then optionally delete the script with another SSH connection. However, this is very inefficient as it requires multiple connections to each server and requires copying the script file to each server.
A much better way to do this is to remotely run either a shell interpretor or a script interpretor (such as Perl) on the remote server and redirect the script you want to run from the local server. Using this method, the script is run on the remote server without ever having to copy the script file to the remote server.
Listing 2 shows the contents of the info.sh script that exists on the central management server. This is a simple script that detects whether the server is AIX or not. If it is AIX, the OS level, CPU mode, entitled CPU capacity, and the amount of memory are displayed.
Listing 2. The info.sh script that exists on the central management server
#!/usr/bin/ksh if [ "`uname`" = "AIX" ]; then printf "This server is running AIX level : " oslevel -s echo "CPU and Memory info: " lparstat -i | egrep "^Mode|^Entitled Capacity |^Online Memory" echo "------------------------------" else echo "This is not an AIX server" fi
Listing 3 shows a for loop similar to the first one we covered. However, in this for loop, the remote command we are running is the Korn shell (ksh) interpretor, and we are redirecting the input to ksh from the info.sh file. This causes the info.sh script to be run on each remote server without actually having to copy the script to the servers.
Listing 3. An example of a for loop command to run local “info.sh” script on each server listed in the servelist file
for server in `cat serverlist`; do printf "%-20s\n" $server; ssh -q -o "BatchMode yes" $server 'ksh 2>&1' < ./info.sh ; echo; done | grep -v ^$
Listing 3 shows an example of a for loop command to run local “info.sh” script on each server listed in the servelist file without having to copy script to each server.
Figure 3 shows that when this command is run, the info.sh script is run on each server specified in the serverlist file.
Figure 3. Using a simple for loop to run a script on remote servers using input redirection
This is an efficient way to run scripts on a large number of servers without ever actually copying the script to each server. However, the script is run serially so it might not be practical if the script takes longer time to complete or you have a large number of servers that the script will be run on.
Comparing serial and parallel running of scrips
The previously covered for loops run commands on the remote servers serially, which means one after another. There are also options to run commands in parallel, meaning the central management server starts commands on multiple servers simultaneously. For example, suppose you need to run a command on 35 servers and each server takes 1 second for the SSH connection to be established and the command to be run. If run serially, it would take 35 seconds for this to complete. If run in parallel running 16 servers simultaneously, it would take approximately 3 seconds (35 servers / 16 run simultaneously = approximately 3 seconds). There is a significant speedup when running commands in parallel on a large number of servers. Figure 4 shows a diagram that compares serial and parallel running of commands.
Figure 4. Diagram showing difference between serial and parallel execution of scripts.
Running a command on multiple servers using the
AIX includes a
dsh command that is designed to run a command on multiple servers in parallel. By default,
dsh can run up to 64 commands in parallel (this is customizable using the
The first step in using
dsh is to make sure that it is installed on your central management server. This can be checked by running
which dsh; . If a path is returned to
dsh, the server has it installed. If not, you need to either install the csm.dsh file set for AIX 6.1 or the dsm.dsh and dsm.core file sets for AIX 7.1. The appropriate file set for AIX 6.1 or AIX 7.1 should be on your AIX product installation media.
The next step is to set some environment variables that instruct
dsh how to operate.
Listing 4. Environment variables to set for
export DSH_NODE_LIST="/home/user/serverlist" export DSH_NODE_RSH=/usr/bin/ssh export DSH_NODE_OPTS="-q -o BatchMode=yes"
Listing 4 shows examples of the variables that must be set before using
DSH_NODE_LIST variable should be set to a file name that specifies the servers on which you need to run the commands, one server per line.
DSH_NODE_RSH variable is set to
/usr/bin/ssh to instruct
dsh to use SSH as the connection protocol.
DSH_NODE_OPTS variable is set to
-q -o BatchMode=yes to instruct the SSH client to not display the login banner and not prompt the user for anything.
After these variables are set, a command can be run on multiple servers by running
dsh , followed by the command line that should be run on each server. Figure 5 shows the
oslevel command being run on several servers using
Figure 5. "oslevel" command being run on several servers using "dsh"
Another useful functionality of
dsh is the
dshbak command. This can format the output of
dsh in an easier to use format if the command returns a lot of output. To use
dshbak, simply run the normal
dsh command and pipe the output to
dshbak as shown in Figure 6.
Figure 6. Using "dshbak" to format the output of "dsh"
Using parallel SSH to run commands and scripts on multiple servers
Parallel SSH is a utility that is designed to run SSH commands on multiple servers in parallel and works similarly to the AIX
dsh utility. However, parallel SSH has several advantages including: being cross platform (same utility works on Linux® and other UNIX® variants), being an open source, and having the capability to run scripts using redirection.
Parallel SSH is available in most Linux distribution repositories for easy installation.
To use parallel SSH on AIX you will first need to install version 2.4 or later of the Python scripting language. I found that the parallel SSH setup.py installation program can be difficult to run on AIX, and it is not necessary. Instead, you can manually copy the parallel SSH binaries easily to any location from which you wantto run them.
To use parallel SSH to run a command on several servers, use a command similar to the one listed in listing 5.
Listing 5. Example command to have
pssh to run a command on several servers
# pssh -h serverlist -i oslevel -s
-h serverlist command specifies the list of servers in the serverlist file(one server name per line). The -i option specifies that the output must be inline on the screen.
oslevel -s is the name of the command that should be run on each of the servers.
Figure 7 shows an example output of running the
oslevel command on several servers using
Figure 7. Running the "oslevel" command on several servers using "pssh"
pssh, it is also possible to quickly and easily run a script on multiple servers. This can be done without copying the script to each server using input direction. This is basically the same method covered earlier in the article under the Running a script on multiple servers using a simple for loop section. However, in this case, it is done in parallel and thus it is much faster.
To run the info.sh script (shown in Listing 2) using
pssh, use the command line shown in Listing 6.
Listing 6. Using "pssh" to run a local script on several servers
# pssh -h serverlist -i -I < info.sh
-I switch tells
pssh to read the command to be run from the standard input. The
< info.sh option redirects the info.sh script to the standard input of
pssh, which in effect runs this script on each server without actually having to copy the script to each server.
Figure 8 shows the output of running the info.sh script on several servers using
Figure 8. Running the info.sh script on several servers using "pssh"
dsh suite also includes a command named dcp that allows files to be copied to or from multiple servers in parallel. This can make pushing out files to many LPAR's a very quick and easy process.
Parallel SSH also includes similar copy commands, which both run in parallel. The
pscp command is used to copy files to servers, while
pslurp is used to copy files from servers.
Figure 9 shows the
pscp command in use to copy the info.sh file into the /tmp directory on several servers.
Figure 9. Using "pscp" to copy info.sh to the /tmp directory on several servers
As demonstrated, there are several methods to run commands or scripts on multiple servers. System administrators can use both simple one-line loops that serially run the command or script on each server, and also utilities that run commands in parallel (such as
dsh and parallel SSH). Using these tools and methods can allow system administrators to be more efficient, getting more done in less time.