- About this series
- Simplifying remote login
- Running a remote command
- Direct interactive sessions
- Running a remote command across multiple machines
- Using remote execution for performance monitoring
- Running the same operation across multiple machines
- Downloadable resources
- Related topics
System Administration Toolkit
Distributed administration using SSH
This content is part # of # in the series: System Administration Toolkit
This content is part of the series:System Administration Toolkit
Stay tuned for additional content in this series.
About this series
The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.
Simplifying remote login
Secure Shell (SSH) tools provide a secure method for logging in and exchanging information with a remote host. A number of different tools are provided, including the general-purpose SSH tool (which provides a remote terminal connection), SCP (a secure, host-to-host, copy solution), and SFTP, a secure file copy solution that works in a similar fashion to the standard FTP tools.
All of these tools are secure in that the information that is exchanged is encrypted. In addition, the authentication of connections is secured using a public or private key mechanism. One of the main benefits of SSH is that you can bypass the normal login and password exchange by copying your public key to a remote machine.
Although this is useful when using SSH to log in to a remote machine (as it means you don't have to provide a password), it is even more useful when performing remote administration. Having to type in a password can also make automated remote administration (for example, running a command through cron) impossible, because in an automated script, you won't be around to type in the password!
When using SSH to run commands across multiple machines without exchanging your public key, you need to type in your password for each machine.
A quick and simple way of setting this up is to create a public key:
$ ssh-keygen -t rsa
Follow the on-screen instructions, but don't set a password when prompted, as you will then need to enter the password each time you want to use the key. This creates a private and a public key file. Now you just need to append the contents of the public key file in .ssh/id_rsa.pub, and append it to the .ssh/authorized_keys file on the remote host and user you want to use when logging in. You need to append the public key file contents to each machine you want to log in to automatically.
Running a remote command
There are many ways in which you can run a remote command.
You can run a single remote command by adding the command you want to run to SSH after the login or host information. For example, to get the disk information for a remote host, you might use the command and get the output in Listing 1 below.
Listing 1. Running a simple command through SSH
$ ssh firstname.lastname@example.org df Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda3 14544820 3611520 10194464 27% / udev 128044 564 127480 1% /dev /dev/hdc1 1968872 50340 1818516 3% /var/tmp /dev/hdc2 1968904 1482220 386668 80% /usr/portage /dev/hdc3 1968904 35760 1833128 2% /home/build shm 128044 0 128044 0% /dev/shm
Bear in mind that the sequence in Listing 1 requires you to enter a password if you haven't already exchanged your public key with the remote host.
You can also execute a sequence of commands by separating each command with a semicolon and then placing the entire sequence of commands into quotes so that it is identified as a single argument.
An example of executing both a disk check and an uptime check is shown in Listing 2.
Listing 2. Executing a disk and an uptime check
$ ssh email@example.com "df;uptime" Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda3 14544820 3611520 10194464 27% / udev 128044 564 127480 1% /dev /dev/hdc1 1968872 50340 1818516 3% /var/tmp /dev/hdc2 1968904 1488100 380788 80% /usr/portage /dev/hdc3 1968904 35760 1833128 2% /home/build shm 128044 0 128044 0% /dev/shm 14:31:27 up 12 min, 2 users, load average: 0.01, 0.05, 0.06
You can string as many commands as you like into this operation. Filtering, for example, using grep or other tools, is also possible, but you need to make sure to embed the entire remote command expression into the quotes (see Listing 3).
Listing 3. Filtering using grep
$ ssh firstname.lastname@example.org "cat /var/log/messages|grep 'su\['" Dec 17 18:05:37 localhost su: pam_authenticate: Permission denied Dec 17 18:05:37 localhost su: FAILED su for root by mc Dec 17 18:05:37 localhost su: - pts/1 mc:root Dec 17 18:06:31 localhost su: pam_authenticate: Permission denied Dec 17 18:06:31 localhost su: FAILED su for root by mc Dec 17 18:06:31 localhost su: - pts/1 mc:root Dec 17 18:06:40 localhost su: pam_authenticate: Permission denied Dec 17 18:06:40 localhost su: FAILED su for root by mc ...
The first item to note about Listing 3 is that you are logging in directly to the remote machine as root. This is because the file you want to view is only accessible to the superuser. You must ensure that your system is configured to allow remote root logins for this to work.
The second important note about this example is that you've performed the grep operation remotely. In actual fact, you don't need to do this. The standard input and output of the remote host are replicated to the local machine, so the output from the command can be filtered locally, as shown here in Listing 4.
Listing 4. Output filtered locally
$ ssh email@example.com "cat /var/log/messages" | grep 'su\[' Dec 17 18:05:37 localhost su: pam_authenticate: Permission denied Dec 17 18:05:37 localhost su: FAILED su for root by mc Dec 17 18:05:37 localhost su: - pts/1 mc:root Dec 17 18:06:31 localhost su: pam_authenticate: Permission denied Dec 17 18:06:31 localhost su: FAILED su for root by mc Dec 17 18:06:31 localhost su: - pts/1 mc:root Dec 17 18:06:40 localhost su: pam_authenticate: Permission denied Dec 17 18:06:40 localhost su: FAILED su for root by mc Dec 17 18:06:40 localhost su: - pts/1 mc:root
Of course, the effect is essentially the same.
Using the remote pipe method, though, is useful when the information or command
that you want to pipe with is remote. For example, you can use
ls in combination with
determine the disk usage of different directories with the command shown in
Listing 5. Determining disk usage of different commands
ssh firstname.lastname@example.org "ls -d /usr/local/* |xargs du -sh " Password: 4.0K /usr/local/bin 4.0K /usr/local/games 4.0K /usr/local/lib 0 /usr/local/man 4.0K /usr/local/sbin 12K /usr/local/share 4.0K /usr/local/src
Before moving on to redistributing these techniques to multiple machines, there's a quick trick for running remote interactive sessions directly without having to log in first.
Direct interactive sessions
As shown previously, you can directly run a number of different commands and chains of commands. One of the benefits of the SSH solution is that although the command itself is executed remotely, the input and output of the command are sourced from the calling machine. You can use this as a method for exchanging information between the two machines relating to the commands that you want to execute.
The commands that you execute can cover almost anything from a range of different commands. However, because you are running commands directly from the command line, there are limits to what you can execute directly with this method. For example, trying to edit a remote file with an editor using this method and techniques shown above usually fail (see Listing 6).
Listing 6. Edit a remote file fails
$ ssh email@example.com "emacs /etc/amavisd.conf" emacs: standard input is not a tty
You can resolve this by forcing SSH to allocate a pseudo-tty device so that you can interact directly with the remote application.
Running a remote command across multiple machines
So far, you have concentrated on running a single command or command string on a single remote machine. Although the interactive session trick is useful when performing remote administration directly with SSH, it is likely that you will want to automate the process, which means that the interactive element is unlikely to be of very much use.
To run the same command remotely across a number of machines, you need to build a simple wrapper around the SSH command and the remote command that you want to run so that the process is repeated on each remote machine.
You can do this with a very simple
for loop, as
demonstrated in Listing 7 below.
for loop to run the command remotely
for remote in firstname.lastname@example.org mc@redhat; do echo $remote; ssh $remote 'df -h'; done email@example.com Filesystem Size Used Avail Use% Mounted on /dev/hda3 14G 4.1G 9.2G 31% / udev 126M 564K 125M 1% /dev /dev/hdc1 1.9G 56M 1.8G 4% /var/tmp /dev/hdc2 1.9G 1.3G 558M 70% /usr/portage /dev/hdc3 1.9G 35M 1.8G 2% /home/build shm 126M 0 126M 0% /dev/shm mc@redhat Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 7.1G 5.5G 1.3G 82% / /dev/hda1 99M 13M 82M 14% /boot none 125M 0 125M 0% /dev/shm
You can easily turn this into a simple script, as shown here in Listing 8.
Listing 8. Reducing the
for loop to simple command
#!/bin/bash # Script to run a command across multiple machines # Global options TIMEOUT=10 ERRLOG=/tmp/remote-err-$$.log OUTLOG=/tmp/remote-out-$$.log # Extract the command line MACHINES=$1;shift COMMAND=$1;shift for machine in $MACHINES do echo $machine ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG 2 >>$ERRLOG done cat $OUTLOG cat $ERRLOG >&2 rm -f $OUTLOG $ERRLOG
The MACHINES and COMMAND are "as-is" as you extract them from the command line. When using the script, you must put the user or host combinations and the command into double quotes to ensure they are identified as a single argument.
The only other addition is the
TIMEOUT option. This
ConnectTimout option to SSH to ensure that
when running a command you don't needlessly wait to connect to a host that might
not be available. The default is set at the head of the script and should ensure
you don't wait too long.
When running the commands, you send the output to a couple of log files, one for standard output and the other for standard error. Then you output these individually to the appropriate location. This highlights one of the benefits of SSH—the remote machine redirects to the same location (standard output, standard error), so you can redirect locally while retaining the meaning of the output.
For example, you can repeat the df check using this script:
$ runremote.sh "gentoo redhat" "df -h"
Because you redirected the standard output and error, you can even generate a log of the whole process:
$ runremote.sh "gentoo redhat" "df -h" 2>/tmp/error.log
Using remote execution for performance monitoring
When using runremote.sh, you might want to play with the exact value used for the timeout value, and you might even want to change the value, depending on what you are doing. For example, if you were using this script to get a snapshot of the current status by using uptime across a bunch of machines, you wouldn't want to wait too long for the connection and command to take place, otherwise the snapshot would be imprecise.
Also, the script, as it stands, runs the command sequentially. Not only does this take a long time if you have a large number of machines, but the time delay between the first machine and the last executing the chosen command might be so significant that correlation across machines might be impossible.
A slightly adjusted script, runremote2.sh, is shown in Listing 9. This executes the remote command almost simultaneously (by running it in the background), and then also pipes the output to individual log files.
Listing 9. Script that executes the remote command almost simultaneously
#!/bin/bash # Script to run a command across multiple machines # Global options TIMEOUT=10 ERRLOG=/tmp/remote-err-$$.log OUTLOG=/tmp/remote-out-$$.log # Extract the command line MACHINES=$1;shift COMMAND=$1;shift for machine in $MACHINES do echo $machine >>$OUTLOG.$machine ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG.$machine 2>>$ERRLOG.$machine & done # Wait for children to finish wait cat $OUTLOG.* cat $ERRLOG.* >&2 rm -f $OUTLOG.* $ERRLOG.*
In this script, you also echo the machine name out to the command log (unique for
each machine supplied). To ensure that the script doesn't exit before all the
remote commands have executed, you need to add a
command to wait for the children of the script to finish.
Now you can use the script to check multiple machines simultaneously (see Listing 10).
Listing 10. Using the script to check multiple machines simultaneously
$ runremote2.sh "narcissus gentoo.vm droopy@nostromo mcbrown@nautilus" 'uptime' droopy@nostromo 19:15 up 9 days, 23:42, 1 user, load averages: 0.01 0.03 0.00 gentoo.vm 18:10:23 up 1 day, 10:02, 2 users, load average: 1.72, 1.84, 1.79 mcbrown@nautilus 19:15 up 10:08, 4 users, load averages: 0.40 0.37 0.29 narcissus 19:15 up 8 days, 7:04, 4 users, load averages: 0.53 0.54 0.57
This kind of monitoring can be useful when you want to get a whole network picture—for example, to check a problem with a group or cluster of machines when running Web or database services and want to identify potential spikes or issues simultaneously across that group of machines.
Be aware, however, that there will still be delays, especially if a machine is particularly busy—the time for the connection to be made and the command to be executed could leave some significant time delays across different machines.
Running the same operation across multiple machines
Creating users across a number of machines can be a pain. There are obviously plenty of solutions for trying to resolve the difficulty from the use of single-sign on utilities, such as the Network Information Service (NIS) or LDAP-based solutions, but you don't always have to synchronize the users in this way.
You could use SSH to do this for you by running the
adduser command across multiple machines. But under
Solaris, the name of the command is
command-line options are largely the same, so you could use
run-remote.sh twice (see Listing 11).
Listing 11. Running run-remote.sh twice
$ runremote.sh "gentoo redhat" "adduser -u 1000 -G sales,marketing mcbrown" $ runremote.sh "solaris solaris-x86" "useradd -u 1000 -G sales,marketing mcbrown"
You've now created the same user across a number of machines with the same groups and the same user ID, but this is hardly practical.
A much better way would be to use the tips demonstrated in the "System Administration Toolkit: Standardizing your UNIX command-line tools" article (see Related topics) to use the same command across multiple machines:
$ runremote.sh "gentoo solaris" "adduser.sh -u 1000 -G sales,marketing mcbrown"
In this article, you've examined a simple, but powerful, method to run commands on a remote machine. Although the basics of the process are straightforward, you can also create additional functionality to complete some robust, automated remote administration tasks (for example, the ability to redirect and pipe remote local input together). By implementing some simple shell script tricks, you can even use the system to remotely administer a number of machines simultaneously, simplifying many of the repetitive tasks and performance monitoring.
- System Administration Toolkit: Check out other parts in this series.
- "System Administration Toolkit: Standardizing your UNIX command-line tools" (Martin Brown, developerWorks, May 2006): Read this article to learn how to use the same command across multiple machines.
- "Scheduling recurring tasks in Java" (Tom White, developerWorks, November 2003): Learn how to build a simple, general scheduling framework for task execution conforming to an arbitrarily complex schedule.
- Read Wikipedia pages on crontab.
- "Scheduling recurring tasks in Java" (Tom White, developerWorks, November 2003): Read this article to learn how to build a simple, general scheduling framework for task execution conforming to an arbitrarily complex schedule.
- For an article series that teaches you how to
program in bash, see:
- "Bash by example, Part 1: Fundamental programming in the Bourne again shell (bash)" (Daniel Robbins, developerWorks, March 2000)
- "Bash by example, Part 2: More bash programming fundamentals" (Daniel Robbins, developerWorks, April 2000)
- "Bash by example, Part 3: Exploring the ebuild system" (Daniel Robbins, developerWorks, May 2000)
- IBM Redbooks: Different systems use different tools, and Solaris to Linux Migration: A Guide for System Administrators helps you identify some key tools.
- Check out other articles and tutorials written by Martin Brown:
- AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
- AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.
- Search the AIX and UNIX library by topic:
- IBM trial software: Build your next development project with software for download directly from developerWorks.