System Administration Toolkit: Distributed administration using SSH

Use Secure Shell (SSH) to run commands on remote UNIX® systems and, with some simple scripts, put together a system that enables you to manage many systems simultaneously from one machine without having to log in directly to the machines themselves. Also examine the basics of a distributed management system and some scripts and solutions using the technique.

Share:

Martin Brown (mc@mcslp.com), Professional Writer, Freelance

Martin Brown has been a professional writer for more than seven years. He is the author of numerous books and articles across a range of topics. His expertise spans myriad development languages and platforms—Perl, Python, Java™, JavaScript, Basic, Pascal, Modula-2, C, C++, Rebol, Gawk, Shellscript, Windows®, Solaris, Linux, BeOS, Mac OS X and more—as well as Web programming, systems management, and integration. He is a Subject Matter Expert (SME) for Microsoft® and regular contributor to ServerWatch.com, LinuxToday.com, and IBM developerWorks. He is also a regular blogger at Computerworld, The Apple Blog, and other sites. You can contact him through his Web site.



14 August 2007

Also available in Chinese Russian

About this series

The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.

Simplifying remote login

Secure Shell (SSH) tools provide a secure method for logging in and exchanging information with a remote host. A number of different tools are provided, including the general-purpose SSH tool (which provides a remote terminal connection), SCP (a secure, host-to-host, copy solution), and SFTP, a secure file copy solution that works in a similar fashion to the standard FTP tools.

All of these tools are secure in that the information that is exchanged is encrypted. In addition, the authentication of connections is secured using a public or private key mechanism. One of the main benefits of SSH is that you can bypass the normal login and password exchange by copying your public key to a remote machine.

Although this is useful when using SSH to log in to a remote machine (as it means you don't have to provide a password), it is even more useful when performing remote administration. Having to type in a password can also make automated remote administration (for example, running a command through cron) impossible, because in an automated script, you won't be around to type in the password!

When using SSH to run commands across multiple machines without exchanging your public key, you need to type in your password for each machine.

A quick and simple way of setting this up is to create a public key:

$ ssh-keygen -t rsa

Follow the on-screen instructions, but don't set a password when prompted, as you will then need to enter the password each time you want to use the key. This creates a private and a public key file. Now you just need to append the contents of the public key file in .ssh/id_rsa.pub, and append it to the .ssh/authorized_keys file on the remote host and user you want to use when logging in. You need to append the public key file contents to each machine you want to log in to automatically.

Running a remote command

There are many ways in which you can run a remote command.

You can run a single remote command by adding the command you want to run to SSH after the login or host information. For example, to get the disk information for a remote host, you might use the command and get the output in Listing 1 below.

Listing 1. Running a simple command through SSH
$ ssh mc@gentoo.vm df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda3             14544820   3611520  10194464  27% /
udev                    128044       564    127480   1% /dev
/dev/hdc1              1968872     50340   1818516   3% /var/tmp
/dev/hdc2              1968904   1482220    386668  80% /usr/portage
/dev/hdc3              1968904     35760   1833128   2% /home/build
shm                     128044         0    128044   0% /dev/shm

Bear in mind that the sequence in Listing 1 requires you to enter a password if you haven't already exchanged your public key with the remote host.

You can also execute a sequence of commands by separating each command with a semicolon and then placing the entire sequence of commands into quotes so that it is identified as a single argument.

An example of executing both a disk check and an uptime check is shown in Listing 2.

Listing 2. Executing a disk and an uptime check
$ ssh mc@gentoo.vm "df;uptime"
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda3             14544820   3611520  10194464  27% /
udev                    128044       564    127480   1% /dev
/dev/hdc1              1968872     50340   1818516   3% /var/tmp
/dev/hdc2              1968904   1488100    380788  80% /usr/portage
/dev/hdc3              1968904     35760   1833128   2% /home/build
shm                     128044         0    128044   0% /dev/shm
 14:31:27 up 12 min,  2 users,  load average: 0.01, 0.05, 0.06

You can string as many commands as you like into this operation. Filtering, for example, using grep or other tools, is also possible, but you need to make sure to embed the entire remote command expression into the quotes (see Listing 3).

Listing 3. Filtering using grep
$ ssh root@gentoo.vm "cat /var/log/messages|grep 'su\['"
Dec 17 18:05:37 localhost su[19218]: pam_authenticate: Permission denied
Dec 17 18:05:37 localhost su[19218]: FAILED su for root by mc
Dec 17 18:05:37 localhost su[19218]: - pts/1 mc:root
Dec 17 18:06:31 localhost su[19221]: pam_authenticate: Permission denied
Dec 17 18:06:31 localhost su[19221]: FAILED su for root by mc
Dec 17 18:06:31 localhost su[19221]: - pts/1 mc:root
Dec 17 18:06:40 localhost su[19222]: pam_authenticate: Permission denied
Dec 17 18:06:40 localhost su[19222]: FAILED su for root by mc
...

The first item to note about Listing 3 is that you are logging in directly to the remote machine as root. This is because the file you want to view is only accessible to the superuser. You must ensure that your system is configured to allow remote root logins for this to work.

The second important note about this example is that you've performed the grep operation remotely. In actual fact, you don't need to do this. The standard input and output of the remote host are replicated to the local machine, so the output from the command can be filtered locally, as shown here in Listing 4.

Listing 4. Output filtered locally
$ ssh root@gentoo.vm "cat /var/log/messages" | grep 'su\[' 
Dec 17 18:05:37 localhost su[19218]: pam_authenticate: Permission denied
Dec 17 18:05:37 localhost su[19218]: FAILED su for root by mc
Dec 17 18:05:37 localhost su[19218]: - pts/1 mc:root
Dec 17 18:06:31 localhost su[19221]: pam_authenticate: Permission denied
Dec 17 18:06:31 localhost su[19221]: FAILED su for root by mc
Dec 17 18:06:31 localhost su[19221]: - pts/1 mc:root
Dec 17 18:06:40 localhost su[19222]: pam_authenticate: Permission denied
Dec 17 18:06:40 localhost su[19222]: FAILED su for root by mc
Dec 17 18:06:40 localhost su[19222]: - pts/1 mc:root

Of course, the effect is essentially the same.

Using the remote pipe method, though, is useful when the information or command that you want to pipe with is remote. For example, you can use ls in combination with du to determine the disk usage of different directories with the command shown in Listing 5.

Listing 5. Determining disk usage of different commands
ssh root@gentoo.vm "ls -d /usr/local/* |xargs du -sh "
Password: 
4.0K    /usr/local/bin
4.0K    /usr/local/games
4.0K    /usr/local/lib
0       /usr/local/man
4.0K    /usr/local/sbin
12K     /usr/local/share
4.0K    /usr/local/src

Before moving on to redistributing these techniques to multiple machines, there's a quick trick for running remote interactive sessions directly without having to log in first.

Direct interactive sessions

As shown previously, you can directly run a number of different commands and chains of commands. One of the benefits of the SSH solution is that although the command itself is executed remotely, the input and output of the command are sourced from the calling machine. You can use this as a method for exchanging information between the two machines relating to the commands that you want to execute.

The commands that you execute can cover almost anything from a range of different commands. However, because you are running commands directly from the command line, there are limits to what you can execute directly with this method. For example, trying to edit a remote file with an editor using this method and techniques shown above usually fail (see Listing 6).

Listing 6. Edit a remote file fails
$ ssh root@gentoo.vm "emacs /etc/amavisd.conf"
emacs: standard input is not a tty

You can resolve this by forcing SSH to allocate a pseudo-tty device so that you can interact directly with the remote application.

Running a remote command across multiple machines

So far, you have concentrated on running a single command or command string on a single remote machine. Although the interactive session trick is useful when performing remote administration directly with SSH, it is likely that you will want to automate the process, which means that the interactive element is unlikely to be of very much use.

To run the same command remotely across a number of machines, you need to build a simple wrapper around the SSH command and the remote command that you want to run so that the process is repeated on each remote machine.

You can do this with a very simple for loop, as demonstrated in Listing 7 below.

Listing 7. for loop to run the command remotely
for remote in mc@gentoo.vm mc@redhat; do echo $remote; ssh $remote 'df -h'; done
mc@gentoo.vm
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3              14G  4.1G  9.2G  31% /
udev                  126M  564K  125M   1% /dev
/dev/hdc1             1.9G   56M  1.8G   4% /var/tmp
/dev/hdc2             1.9G  1.3G  558M  70% /usr/portage
/dev/hdc3             1.9G   35M  1.8G   2% /home/build
shm                   126M     0  126M   0% /dev/shm
mc@redhat
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      7.1G  5.5G  1.3G  82% /
/dev/hda1              99M   13M   82M  14% /boot
none                  125M     0  125M   0% /dev/shm

You can easily turn this into a simple script, as shown here in Listing 8.

Listing 8. Reducing the for loop to simple command
#!/bin/bash

# Script to run a command across multiple machines

# Global options

TIMEOUT=10
ERRLOG=/tmp/remote-err-$$.log
OUTLOG=/tmp/remote-out-$$.log

# Extract the command line

MACHINES=$1;shift
COMMAND=$1;shift

for machine in $MACHINES
do
    echo $machine
    ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG 2 >>$ERRLOG

done

cat $OUTLOG
cat $ERRLOG >&2
rm -f $OUTLOG $ERRLOG

The MACHINES and COMMAND are "as-is" as you extract them from the command line. When using the script, you must put the user or host combinations and the command into double quotes to ensure they are identified as a single argument.

The only other addition is the TIMEOUT option. This sets the ConnectTimout option to SSH to ensure that when running a command you don't needlessly wait to connect to a host that might not be available. The default is set at the head of the script and should ensure you don't wait too long.

When running the commands, you send the output to a couple of log files, one for standard output and the other for standard error. Then you output these individually to the appropriate location. This highlights one of the benefits of SSH—the remote machine redirects to the same location (standard output, standard error), so you can redirect locally while retaining the meaning of the output.

For example, you can repeat the df check using this script:

$ runremote.sh "gentoo redhat" "df -h"

Because you redirected the standard output and error, you can even generate a log of the whole process:

$ runremote.sh "gentoo redhat" "df -h" 2>/tmp/error.log

Using remote execution for performance monitoring

When using runremote.sh, you might want to play with the exact value used for the timeout value, and you might even want to change the value, depending on what you are doing. For example, if you were using this script to get a snapshot of the current status by using uptime across a bunch of machines, you wouldn't want to wait too long for the connection and command to take place, otherwise the snapshot would be imprecise.

Also, the script, as it stands, runs the command sequentially. Not only does this take a long time if you have a large number of machines, but the time delay between the first machine and the last executing the chosen command might be so significant that correlation across machines might be impossible.

A slightly adjusted script, runremote2.sh, is shown in Listing 9. This executes the remote command almost simultaneously (by running it in the background), and then also pipes the output to individual log files.

Listing 9. Script that executes the remote command almost simultaneously
#!/bin/bash

# Script to run a command across multiple machines

# Global options

TIMEOUT=10
ERRLOG=/tmp/remote-err-$$.log
OUTLOG=/tmp/remote-out-$$.log

# Extract the command line

MACHINES=$1;shift
COMMAND=$1;shift

for machine in $MACHINES
do
    echo $machine >>$OUTLOG.$machine
    ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG.$machine
	2>>$ERRLOG.$machine &
done

# Wait for children to finish

wait

cat $OUTLOG.*
cat $ERRLOG.* >&2
rm -f $OUTLOG.* $ERRLOG.*

In this script, you also echo the machine name out to the command log (unique for each machine supplied). To ensure that the script doesn't exit before all the remote commands have executed, you need to add a wait command to wait for the children of the script to finish.

Now you can use the script to check multiple machines simultaneously (see Listing 10).

Listing 10. Using the script to check multiple machines simultaneously
$ runremote2.sh "narcissus gentoo.vm droopy@nostromo mcbrown@nautilus" 'uptime'
droopy@nostromo
19:15  up 9 days, 23:42, 1 user, load averages: 0.01 0.03 0.00
gentoo.vm
 18:10:23 up 1 day, 10:02,  2 users,  load average: 1.72, 1.84, 1.79
mcbrown@nautilus
19:15  up 10:08, 4 users, load averages: 0.40 0.37 0.29
narcissus
19:15  up 8 days,  7:04, 4 users, load averages: 0.53 0.54 0.57

This kind of monitoring can be useful when you want to get a whole network picture—for example, to check a problem with a group or cluster of machines when running Web or database services and want to identify potential spikes or issues simultaneously across that group of machines.

Be aware, however, that there will still be delays, especially if a machine is particularly busy—the time for the connection to be made and the command to be executed could leave some significant time delays across different machines.

Running the same operation across multiple machines

Creating users across a number of machines can be a pain. There are obviously plenty of solutions for trying to resolve the difficulty from the use of single-sign on utilities, such as the Network Information Service (NIS) or LDAP-based solutions, but you don't always have to synchronize the users in this way.

You could use SSH to do this for you by running the adduser command across multiple machines. But under Solaris, the name of the command is useradd. The command-line options are largely the same, so you could use run-remote.sh twice (see Listing 11).

Listing 11. Running run-remote.sh twice
$ runremote.sh "gentoo redhat" "adduser -u 1000 -G sales,marketing mcbrown"
$ runremote.sh "solaris solaris-x86" "useradd -u 1000 -G sales,marketing mcbrown"

You've now created the same user across a number of machines with the same groups and the same user ID, but this is hardly practical.

A much better way would be to use the tips demonstrated in the "System Administration Toolkit: Standardizing your UNIX command-line tools" article (see Resources) to use the same command across multiple machines:

$ runremote.sh "gentoo solaris" "adduser.sh -u 1000 -G sales,marketing mcbrown"

Summary

In this article, you've examined a simple, but powerful, method to run commands on a remote machine. Although the basics of the process are straightforward, you can also create additional functionality to complete some robust, automated remote administration tasks (for example, the ability to redirect and pipe remote local input together). By implementing some simple shell script tricks, you can even use the system to remotely administer a number of machines simultaneously, simplifying many of the repetitive tasks and performance monitoring.

Resources

Learn

Get products and technologies

  • IBM trial software: Build your next development project with software for download directly from developerWorks.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=248185
ArticleTitle=System Administration Toolkit: Distributed administration using SSH
publish-date=08142007