Monitoring logs and command output

Keeping an eye on files, processes, and systems

Monitoring system logs or the status of a command that produces file or directory output are common tasks for systems administrators. Two popular open source tools simplify these activities for modern systems administrators: the multitail and watch commands. Both are terminal-oriented commands, which means that they are easily ported to most UNIX® or UNIX-like systems because they do not depend on any specific graphical desktop environment.

Share:

William von Hagen, Systems Administrator, Writer, WordSmiths

William von Hagen has been a UNIX systems administrator for more than 20 years and a Linux advocate since 1993. Bill is the author or co-author of books on subjects such as Ubuntu Linux, Xen Virtualization, the GNU Compiler Collection (GCC), SuSE Linux, Mac OS X, Linux file systems, and SGML. He has also written numerous articles for Linux and Mac OS X publications and Web sites. You can reach Bill at wvh@vonhagen.org.



25 August 2009

Also available in Chinese

Log files provide summary information that is critical to seeing how a system or a specific server is behaving. On UNIX and UNIX-like systems, log files are simply text files that are continually appended, either by the server processes that they are associated with or by system-logging daemons. All UNIX and UNIX-like systems use a general-purpose system-logging daemon, usually referred to as syslogd or simply syslog, which handles log output from many different applications and servers. Linux® systems also run klogd, a specialized daemon that specifically monitors kernel messages.

Classic log monitoring with the tail command

The classic command used to monitor new output in system or server logs (referred to simply as logs in the rest of this article) is the tail command, which shows the last few lines in a specified file (the last 10 lines by default). When executed as tail filename, the tail command exits after displaying the last lines from the specified file, which isn't all that helpful when you want to monitor updates to a log file. To actively monitor a log file, the tail command is most often invoked with the -f option, which displays the last lines of the file and then continues to display subsequent lines as they are added to the specified file.

It's often necessary (or at least useful) to monitor multiple log files simultaneously. Specific server processes that you may want to monitor usually write to their own log files, while syslog sends different types of messages to different log files (depending upon how it is configured in /etc/syslog.conf). As an example, on server systems running Ubuntu, it is often useful to monitor both /var/log/auth.log (where authentication requests and changes are logged) and /var/log/kern.log (where kernel messages are logged). Similarly, on server systems running the Apache Web server, it is useful to monitor both the standard Apache log files: /var/log/apache2/access.log and /var/log/apache2/error.log.

Many systems administrators, including this author in days gone by, simply start multiple xterms (or other graphical console/shell applications), each one of which runs the tail -f command to monitor a specific log file. That provides the necessary information but sacrifices a tremendous amount of screen real estate regardless of the size of the font that you use. As discussed in the rest of this article, the free and open source community provides a few advanced alternatives to this somewhat primitive file monitoring solution.


Monitoring multiple log files simultaneously

MultiTail is a utility that provides a superset of the basic functionality of the tail command, enabling you to monitor multiple files at the same time within the same terminal application. MultiTail (executed as multitail) is a GNU Public License (GPL) Version 2 application that is available in the repositories of most UNIX-like operating systems and can easily be built from source code. The only significant requirement for building multitail is the ncurses library, which enables pseudo-graphical operations to be done within terminal windows.

The simplest way of executing MultiTail is to specify the names of the files that you want to monitor on the command line. For example, the multitail access.log error.log command enables you to monitor both the primary log files for an Apache Web server, as shown in Figure 1.

Figure 1. Viewing multiple Apache logs in MultiTail
Viewing multiple Apache logs in MultiTail

MultiTail has a huge number of options, all of which are documented in its traditional UNIX man page. When running MultiTail in a terminal session, you can request help, actively modify the display, search for specific text, and so on by using a variety of keyboard commands. You can access MultiTail's built-in help at any time by clicking Control-h, which displays an overlaid help window, as shown in Figure 2.

Figure 2. Viewing help in MultiTail
Viewing help in MultiTail

You can scroll through this window using the cursor keys and close it by typing q or Control-g. (MultiTail's help is also bound to the F1 key, but graphical desktops such as GNOME usually intercept that key sequence and display their own online help instead.)


Popular options for the multitail command

Some of the more commonly used MultiTail command-line options are described in Table 1.

Table 1. Popular multitail command options
OptionDescription
--closeidle NEnables you to specify a period of inactivity after which a MultiTail window will close. This can be useful if you are monitoring a large number of files at the same time but only care about those that are actively being updated. Though this option can be convenient, you should use it with care because after a window closes it will not reopen if new output is written to it.
-i fileEnables you to specify the names of the files that you want to monitor. Ordinarily, you simply specify the names of the files that you want to monitor by listing their names on the command line. Preceding a file name with the -i option enables you to monitor a file whose name begins with a dash (-). Without using the -i option, any file whose name begins with a dash is assumed to be a command-line option. (This is much like the use of the -- option in commands such as rm.)
--mark-interval NEnables you to specify a period of time (N seconds) of inactivity that will cause a mark line to be displayed in any MultiTail output window. For example, if you are monitoring the /var/log/auth.log file and no authentication-related messages have been logged to that file in N seconds, MultiTail displays a highlighted line such as ---auth.log DATE TIME--- in the MultiTail output window for that log file. This is especially handy when you are monitoring multiple log files because it makes it easy to quickly identify the log files to which no new messages have been logged.
--no-repeatCauses MultiTail to suppress display of repeated messages in the same window, displaying a summary message ("Last message repeated N times") instead.

Figure 3 shows the output from the following command on a sample machine after 60 seconds:

    multitail --mark-interval 10 --no-repeat auth.log \
              daemon.log kern.log lpr.log mail.err mysql.err \
              wpa_supplicant.log messages
Figure 3. More advanced log file display in MultiTail
More advanced log file display in MultiTail

As you can see from Figure 3, MultiTail divides the available height of your terminal window between all the files that you want to monitor. In most cases, the last one or two lines of log file output is sufficient to identify any truly important messages that are being displayed in one of your log files.

The options used in the previous example are only the tip of the MultiTail iceberg. As you can see from the figures, MultiTail supports various types of highlighting, including pattern-based highlighting that makes it easy to spot messages associated with a specific device or that contain a specific keyword.

The multitail command is not limited to simply monitoring existing files. It also provides options that let you monitor the output of a command (-l command) that produces continuous output. For quick-running commands, the -r N option enables you to repeat a given command after N seconds.


Monitoring command output

Sometimes it's more important to know that a given log file is being written, and when, than it is to know what's being written to it. Apache's access log is a good example—heavily used Web servers can generate thousands of log messages in a few minutes, each of which provides detailed information that isn't really useful. In cases such as these, what most systems administrators actually care about is that the log is constantly growing—in other words, that Apache is running and being accessed.

You can certainly do this visually by repeatedly running a command such as ls -l and comparing file sizes. As mentioned at the end of the previous section, you can also use the multitail command to monitor command output and repeat a specified command multiple times. Unfortunately, this requires a few command-line options:

    multitail -Rc 2 -l "ls -l access.log error.log"

The watch command, which is installed by default on most Linux systems, provides a simpler solution. The watch command automatically runs a specified command at specified intervals, with a default interval of two seconds, and does exactly the right thing when combined with a command such as ls -l. Figure 4 shows the output from the watch ls -l access.log error.log command.

Figure 4. Monitoring log file status using watch
Monitoring log file status using watch

Unlike the multitail command, the watch command runs until interrupted by a keyboard command such as Control-C.

The watch command allows you to watch the output of a program change over time, which can be especially convenient when looking for changes to file sizes that reflect debugging output, download or file creation state, memory or disk usage, and so on. For example, running a command such as watch cat /proc/meminfo provides a simple way of viewing memory use on your system in real-time. When monitoring a command in which multiple lines differ each time that it is executed, you can specify the -d option to highlight the differences between successive runs of the command. You can also use the --differences=cumulative option to accumulate highlighted differences between all runs of the specified command.


Different watch commands on different systems

The watch command is part of the procps package, a GPL v2 package that also provides commands such as ps, top, vmstat, and so on. Be careful if you are using a UNIX or UNIX-like system other than Linux, because other operating systems provide commands called watch that are completely different. For example, on Berkeley Software Distribution (BSD)-inspired systems, watch is usually called cmdwatch or gnuwatch, because these systems have a different watch command that monitors the processes running on a specified physical or virtual terminal. Similarly, IBM® AIX® operating systems provide a watch command that monitors the processes that are created by a specified application.

If your system doesn't provide the standard watch command and you don't have the time or inclination to build the procps package for your system, you can easily simulate it with a simple shell script. The shell script shown in Listing 1, simplewatch.sh, provides much of the same functionality as the watch command.

Listing 1. A shell equivalent for the watch command
    #!/bin/sh

    while [ 1 ] ; do
      clear
      echo "Command: $*"
      date
      echo ""
      ( $* )
      sleep 2
    done

Figure 5 shows the output from running the simplewatch.sh ls -lt \*.log command in the /var/log directory on a sample OpenSolaris system. Quoting any wildcards that you pass to this script isn't necessary but helps simplify the output that it displays.

Figure 5. Output from running the simplewatch.sh ls -lt \*.log command
The simplewatch,sh Script on OpenSolaris

Conclusion

Monitoring log files, file creation and size, and differences between subsequent runs of a specific command are common systems administration tasks. The tail, multitail, and watch commands discussed in this article help automate these potentially boring tasks, making it easy to keep an eye on various aspects of your systems while also getting other work done.


Download

DescriptionNameSize
Shell-script equivalent of the watch commandsimplewatch.sh1KB

Resources

Learn

Get products and technologies

  • Download the latest source code for the package that includes the GNU tail program from the GNU Coreutils home page, or view the complete documentation online.
  • Download the latest source code for the package that includes the GNU watch program from the procps home page.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=422932
ArticleTitle=Monitoring logs and command output
publish-date=08252009