About this series

The typical UNIX® administrator has a key range of utilities, tricks, and systems he or she uses regularly to aid in the process of administration. There are key utilities, command-line chains, and scripts that are used to simplify different processes. Some of these tools come with the operating system, but a majority of the tricks come through years of experience and a desire to ease the system administrator's life. The focus of this series is on getting the most from the available tools across a range of different UNIX environments, including methods of simplifying administration in a heterogeneous environment.

Simplifying remote login

Secure Shell (SSH) tools provide a secure method for logging in and exchanging information with a remote host. A number of different tools are provided, including the general-purpose SSH tool (which provides a remote terminal connection), SCP (a secure, host-to-host, copy solution), and SFTP, a secure file copy solution that works in a similar fashion to the standard FTP tools.

All of these tools are secure in that the information that is exchanged is encrypted. In addition, the authentication of connections is secured using a public or private key mechanism. One of the main benefits of SSH is that you can bypass the normal login and password exchange by copying your public key to a remote machine.

Although this is useful when using SSH to log in to a remote machine (as it means you don't have to provide a password), it is even more useful when performing remote administration. Having to type in a password can also make automated remote administration (for example, running a command through cron) impossible, because in an automated script, you won't be around to type in the password!

When using SSH to run commands across multiple machines without exchanging your public key, you need to type in your password for each machine.

A quick and simple way of setting this up is to create a public key:

$ ssh-keygen -t rsa

Follow the on-screen instructions, but don't set a password when prompted, as you will then need to enter the password each time you want to use the key. This creates a private and a public key file. Now you just need to append the contents of the public key file in .ssh/id_rsa.pub, and append it to the .ssh/authorized_keys file on the remote host and user you want to use when logging in. You need to append the public key file contents to each machine you want to log in to automatically.

Running a remote command

There are many ways in which you can run a remote command.

You can run a single remote command by adding the command you want to run to SSH after the login or host information. For example, to get the disk information for a remote host, you might use the command and get the output in Listing 1 below.

                
$ ssh mc@gentoo.vm df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda3             14544820   3611520  10194464  27% /
udev                    128044       564    127480   1% /dev
/dev/hdc1              1968872     50340   1818516   3% /var/tmp
/dev/hdc2              1968904   1482220    386668  80% /usr/portage
/dev/hdc3              1968904     35760   1833128   2% /home/build
shm                     128044         0    128044   0% /dev/shm

Bear in mind that the sequence in Listing 1 requires you to enter a password if you haven't already exchanged your public key with the remote host.

You can also execute a sequence of commands by separating each command with a semicolon and then placing the entire sequence of commands into quotes so that it is identified as a single argument.

An example of executing both a disk check and an uptime check is shown in Listing 2.

                
$ ssh mc@gentoo.vm "df;uptime"
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda3             14544820   3611520  10194464  27% /
udev                    128044       564    127480   1% /dev
/dev/hdc1              1968872     50340   1818516   3% /var/tmp
/dev/hdc2              1968904   1488100    380788  80% /usr/portage
/dev/hdc3              1968904     35760   1833128   2% /home/build
shm                     128044         0    128044   0% /dev/shm
 14:31:27 up 12 min,  2 users,  load average: 0.01, 0.05, 0.06
 

You can string as many commands as you like into this operation. Filtering, for example, using grep or other tools, is also possible, but you need to make sure to embed the entire remote command expression into the quotes (see Listing 3).

                
$ ssh root@gentoo.vm "cat /var/log/messages|grep 'su\['"
Dec 17 18:05:37 localhost su[19218]: pam_authenticate: Permission denied
Dec 17 18:05:37 localhost su[19218]: FAILED su for root by mc
Dec 17 18:05:37 localhost su[19218]: - pts/1 mc:root
Dec 17 18:06:31 localhost su[19221]: pam_authenticate: Permission denied
Dec 17 18:06:31 localhost su[19221]: FAILED su for root by mc
Dec 17 18:06:31 localhost su[19221]: - pts/1 mc:root
Dec 17 18:06:40 localhost su[19222]: pam_authenticate: Permission denied
Dec 17 18:06:40 localhost su[19222]: FAILED su for root by mc
...

The first item to note about Listing 3 is that you are logging in directly to the remote machine as root. This is because the file you want to view is only accessible to the superuser. You must ensure that your system is configured to allow remote root logins for this to work.

The second important note about this example is that you've performed the grep operation remotely. In actual fact, you don't need to do this. The standard input and output of the remote host are replicated to the local machine, so the output from the command can be filtered locally, as shown here in Listing 4.

                
$ ssh root@gentoo.vm "cat /var/log/messages" | grep 'su\[' 
Dec 17 18:05:37 localhost su[19218]: pam_authenticate: Permission denied
Dec 17 18:05:37 localhost su[19218]: FAILED su for root by mc
Dec 17 18:05:37 localhost su[19218]: - pts/1 mc:root
Dec 17 18:06:31 localhost su[19221]: pam_authenticate: Permission denied
Dec 17 18:06:31 localhost su[19221]: FAILED su for root by mc
Dec 17 18:06:31 localhost su[19221]: - pts/1 mc:root
Dec 17 18:06:40 localhost su[19222]: pam_authenticate: Permission denied
Dec 17 18:06:40 localhost su[19222]: FAILED su for root by mc
Dec 17 18:06:40 localhost su[19222]: - pts/1 mc:root

Of course, the effect is essentially the same.

Using the remote pipe method, though, is useful when the information or command that you want to pipe with is remote. For example, you can use ls in combination with du to determine the disk usage of different directories with the command shown in Listing 5.

                
ssh root@gentoo.vm "ls -d /usr/local/* |xargs du -sh "
Password: 
4.0K    /usr/local/bin
4.0K    /usr/local/games
4.0K    /usr/local/lib
0       /usr/local/man
4.0K    /usr/local/sbin
12K     /usr/local/share
4.0K    /usr/local/src

Before moving on to redistributing these techniques to multiple machines, there's a quick trick for running remote interactive sessions directly without having to log in first.

Direct interactive sessions

As shown previously, you can directly run a number of different commands and chains of commands. One of the benefits of the SSH solution is that although the command itself is executed remotely, the input and output of the command are sourced from the calling machine. You can use this as a method for exchanging information between the two machines relating to the commands that you want to execute.

The commands that you execute can cover almost anything from a range of different commands. However, because you are running commands directly from the command line, there are limits to what you can execute directly with this method. For example, trying to edit a remote file with an editor using this method and techniques shown above usually fail (see Listing 6).

                
$ ssh root@gentoo.vm "emacs /etc/amavisd.conf"
emacs: standard input is not a tty

You can resolve this by forcing SSH to allocate a pseudo-tty device so that you can interact directly with the remote application.

Running a remote command across multiple machines

So far, you have concentrated on running a single command or command string on a single remote machine. Although the interactive session trick is useful when performing remote administration directly with SSH, it is likely that you will want to automate the process, which means that the interactive element is unlikely to be of very much use.

To run the same command remotely across a number of machines, you need to build a simple wrapper around the SSH command and the remote command that you want to run so that the process is repeated on each remote machine.

You can do this with a very simple for loop, as demonstrated in Listing 7 below.

                
for remote in mc@gentoo.vm mc@redhat; do echo $remote; ssh $remote 'df -h'; done
mc@gentoo.vm
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3              14G  4.1G  9.2G  31% /
udev                  126M  564K  125M   1% /dev
/dev/hdc1             1.9G   56M  1.8G   4% /var/tmp
/dev/hdc2             1.9G  1.3G  558M  70% /usr/portage
/dev/hdc3             1.9G   35M  1.8G   2% /home/build
shm                   126M     0  126M   0% /dev/shm
mc@redhat
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      7.1G  5.5G  1.3G  82% /
/dev/hda1              99M   13M   82M  14% /boot
none                  125M     0  125M   0% /dev/shm

You can easily turn this into a simple script, as shown here in Listing 8.

                
#!/bin/bash

# Script to run a command across multiple machines

# Global options

TIMEOUT=10
ERRLOG=/tmp/remote-err-$$.log
OUTLOG=/tmp/remote-out-$$.log

# Extract the command line

MACHINES=$1;shift
COMMAND=$1;shift

for machine in $MACHINES
do
    echo $machine
    ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG 2 >>$ERRLOG

done

cat $OUTLOG
cat $ERRLOG >&2
rm -f $OUTLOG $ERRLOG

The MACHINES and COMMAND are "as-is" as you extract them from the command line. When using the script, you must put the user or host combinations and the command into double quotes to ensure they are identified as a single argument.

The only other addition is the TIMEOUT option. This sets the ConnectTimout option to SSH to ensure that when running a command you don't needlessly wait to connect to a host that might not be available. The default is set at the head of the script and should ensure you don't wait too long.

When running the commands, you send the output to a couple of log files, one for standard output and the other for standard error. Then you output these individually to the appropriate location. This highlights one of the benefits of SSH—the remote machine redirects to the same location (standard output, standard error), so you can redirect locally while retaining the meaning of the output.

For example, you can repeat the df check using this script:

$ runremote.sh "gentoo redhat" "df -h"

Because you redirected the standard output and error, you can even generate a log of the whole process:

$ runremote.sh "gentoo redhat" "df -h" 2>/tmp/error.log

Using remote execution for performance monitoring

When using runremote.sh, you might want to play with the exact value used for the timeout value, and you might even want to change the value, depending on what you are doing. For example, if you were using this script to get a snapshot of the current status by using uptime across a bunch of machines, you wouldn't want to wait too long for the connection and command to take place, otherwise the snapshot would be imprecise.

Also, the script, as it stands, runs the command sequentially. Not only does this take a long time if you have a large number of machines, but the time delay between the first machine and the last executing the chosen command might be so significant that correlation across machines might be impossible.

A slightly adjusted script, runremote2.sh, is shown in Listing 9. This executes the remote command almost simultaneously (by running it in the background), and then also pipes the output to individual log files.

                
#!/bin/bash

# Script to run a command across multiple machines

# Global options

TIMEOUT=10
ERRLOG=/tmp/remote-err-$$.log
OUTLOG=/tmp/remote-out-$$.log

# Extract the command line

MACHINES=$1;shift
COMMAND=$1;shift

for machine in $MACHINES
do
    echo $machine >>$OUTLOG.$machine
    ssh -oConnectTimeout=$TIMEOUT $machine $COMMAND >>$OUTLOG.$machine
	2>>$ERRLOG.$machine &
done

# Wait for children to finish

wait

cat $OUTLOG.*
cat $ERRLOG.* >&2
rm -f $OUTLOG.* $ERRLOG.*

In this script, you also echo the machine name out to the command log (unique for each machine supplied). To ensure that the script doesn't exit before all the remote commands have executed, you need to add a wait command to wait for the children of the script to finish.

Now you can use the script to check multiple machines simultaneously (see Listing 10).

                
$ runremote2.sh "narcissus gentoo.vm droopy@nostromo mcbrown@nautilus" 'uptime'
droopy@nostromo
19:15  up 9 days, 23:42, 1 user, load averages: 0.01 0.03 0.00
gentoo.vm
 18:10:23 up 1 day, 10:02,  2 users,  load average: 1.72, 1.84, 1.79
mcbrown@nautilus
19:15  up 10:08, 4 users, load averages: 0.40 0.37 0.29
narcissus
19:15  up 8 days,  7:04, 4 users, load averages: 0.53 0.54 0.57

This kind of monitoring can be useful when you want to get a whole network picture—for example, to check a problem with a group or cluster of machines when running Web or database services and want to identify potential spikes or issues simultaneously across that group of machines.

Be aware, however, that there will still be delays, especially if a machine is particularly busy—the time for the connection to be made and the command to be executed could leave some significant time delays across different machines.

Running the same operation across multiple machines

Creating users across a number of machines can be a pain. There are obviously plenty of solutions for trying to resolve the difficulty from the use of single-sign on utilities, such as the Network Information Service (NIS) or LDAP-based solutions, but you don't always have to synchronize the users in this way.

You could use SSH to do this for you by running the adduser command across multiple machines. But under Solaris, the name of the command is useradd. The command-line options are largely the same, so you could use run-remote.sh twice (see Listing 11).

                
$ runremote.sh "gentoo redhat" "adduser -u 1000 -G sales,marketing mcbrown"
$ runremote.sh "solaris solaris-x86" "useradd -u 1000 -G sales,marketing mcbrown"

You've now created the same user across a number of machines with the same groups and the same user ID, but this is hardly practical.

A much better way would be to use the tips demonstrated in the "System Administration Toolkit: Standardizing your UNIX command-line tools" article (see Resources) to use the same command across multiple machines:

$ runremote.sh "gentoo solaris" "adduser.sh -u 1000 -G sales,marketing mcbrown"

Summary

In this article, you've examined a simple, but powerful, method to run commands on a remote machine. Although the basics of the process are straightforward, you can also create additional functionality to complete some robust, automated remote administration tasks (for example, the ability to redirect and pipe remote local input together). By implementing some simple shell script tricks, you can even use the system to remotely administer a number of machines simultaneously, simplifying many of the repetitive tasks and performance monitoring.

Resources

Learn

• System Administration Toolkit: Check out other parts in this series.
  
  
• "System Administration Toolkit: Standardizing your UNIX command-line tools" (Martin Brown, developerWorks, May 2006): Read this article to learn how to use the same command across multiple machines.
  
  
• "Scheduling recurring tasks in Java" (Tom White, developerWorks, November 2003): Learn how to build a simple, general scheduling framework for task execution conforming to an arbitrarily complex schedule.
  
  
• Read Wikipedia pages on crontab.
  
  
• "The road to better programming: Chapter 11. Crontab management with cfperl" (Teodor Zlatanov, developerWorks, June 2003): In Part 11 of an article series on developing a cfegine interpreter written in Perl, crontab entries can be added or deleted easily.
  
  
• "Scheduling recurring tasks in Java" (Tom White, developerWorks, November 2003): Read this article to learn how to build a simple, general scheduling framework for task execution conforming to an arbitrarily complex schedule.
  
  
• For an article series that teaches you how to program in bash, see:
  • "Bash by example, Part 1: Fundamental programming in the Bourne again shell (bash)" (Daniel Robbins, developerWorks, March 2000)
  • "Bash by example, Part 2: More bash programming fundamentals" (Daniel Robbins, developerWorks, April 2000)
  • "Bash by example, Part 3: Exploring the ebuild system" (Daniel Robbins, developerWorks, May 2000)
  
  
• "Making UNIX and Linux work together" (Martin Brown, developerWorks, April 2006): This article is a guide to getting traditional UNIX distributions and Linux® working together.
  
  
• IBM Redbooks: Different systems use different tools, and Solaris to Linux Migration: A Guide for System Administrators helps you identify some key tools.
  
  
• "Exploring the Linux memory model" (Vikram Shukla, developerWorks, January 2006): This article helps you understand how Linux uses memory, swap space, and exchanges pages and processes between the two.
  
  
• Popular content: See what AIX® and UNIX content your peers find interesting.
  
  
• Check out other articles and tutorials written by Martin Brown:
  • Across developerWorks and IBM
  
  
• AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX?: Visit the "New to AIX and UNIX" page to learn more about AIX and UNIX.
  
  
• AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.
  
  
• Search the AIX and UNIX library by topic:
  • System administration
  • Application development
  • Performance
  • Porting
  • Security
  • Tips
  • Tools and utilities
  • Java™ technology
  • Linux
  • Open source
  
  
• Safari bookstore: Visit this e-reference library to find specific technical resources.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  
• Podcasts: Tune in and catch up with IBM technical experts.
  
  

Get products and technologies

• IBM trial software: Build your next development project with software for download directly from developerWorks.
  
  

Discuss

• Participate in the developerWorks blogs and get involved in the developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX —technical forum
  • AIX 6 Open Beta
  • AIX for Developers Forum
  • Cluster Systems Management
  • IBM Support Assistant
  • Performance Tools—technical
  • Virtualization—technical
  • More AIX and UNIX forums
  
  

About the author

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• About this series
• Simplifying remote login
• Running a remote command
• Direct interactive sessions
• Running a remote command across multiple machines
• Using remote execution for performance monitoring
• Running the same operation across multiple machines
• Summary
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers