Monitoring logs and command output
Keeping an eye on files, processes, and systems
Log files provide summary information that is critical to seeing
how a system or a specific server is behaving. On UNIX and
UNIX-like systems, log files are simply text files that are
continually appended, either by the server processes that
they are associated with or by system-logging daemons. All UNIX
and UNIX-like systems use a general-purpose system-logging daemon,
usually referred to as
syslogd or simply
syslog, which handles log
output from many different applications and servers. Linux® systems
klogd, a specialized daemon that specifically monitors
Classic log monitoring with the
The classic command used to monitor new output in system or server
logs (referred to simply as logs in the rest of this
article) is the
tail command, which shows the last few lines in a
specified file (the last 10 lines by default). When executed as
tail filename, the
tail command exits
after displaying the last lines from the specified file, which
isn't all that helpful when you want to monitor updates to a log file. To
actively monitor a log file, the
tail command is most often
invoked with the
-f option, which displays the last lines
of the file and then continues to display subsequent lines as they
are added to the specified file.
It's often necessary (or at least useful) to monitor multiple log
files simultaneously. Specific server processes that you may want
to monitor usually write to their own log files, while
sends different types of messages to different log files
(depending upon how it is configured in /etc/syslog.conf). As an example, on server
systems running Ubuntu, it is often useful to monitor both /var/log/auth.log (where authentication
requests and changes are logged) and /var/log/kern.log (where kernel messages are
logged). Similarly, on server systems running the Apache Web
server, it is useful to monitor both the standard Apache log
files: /var/log/apache2/access.log and /var/log/apache2/error.log.
Many systems administrators, including this author in days gone by, simply
start multiple xterms (or other graphical console/shell
applications), each one of which runs the
command to monitor a specific log file. That provides the
necessary information but sacrifices a tremendous amount of screen
real estate regardless of the size of the font that you use. As
discussed in the rest of this article, the free and open source
community provides a few advanced alternatives to this somewhat
primitive file monitoring solution.
Monitoring multiple log files simultaneously
MultiTail is a utility that provides a superset of the basic
functionality of the
tail command, enabling you to monitor
multiple files at the same time within the same terminal
application. MultiTail (executed as
multitail) is a GNU Public
License (GPL) Version 2 application that is available in the
repositories of most UNIX-like operating systems and can easily
be built from source code. The only significant requirement for
multitail is the ncurses library, which enables
pseudo-graphical operations to be done within terminal windows.
The simplest way of executing MultiTail is to specify the names of
the files that you want to monitor on the command line. For
multitail access.log error.log command enables
you to monitor both the primary log files for an Apache Web
server, as shown in Figure 1.
Figure 1. Viewing multiple Apache logs in MultiTail
MultiTail has a huge number of options, all of which are documented in its traditional UNIX man page. When running MultiTail in a terminal session, you can request help, actively modify the display, search for specific text, and so on by using a variety of keyboard commands. You can access MultiTail's built-in help at any time by clicking Control-h, which displays an overlaid help window, as shown in Figure 2.
Figure 2. Viewing help in MultiTail
You can scroll through this window using the cursor keys and close it by typing q or Control-g. (MultiTail's help is also bound to the F1 key, but graphical desktops such as GNOME usually intercept that key sequence and display their own online help instead.)
Popular options for the
Some of the more commonly used MultiTail command-line options are described in Table 1.
Table 1. Popular
multitail command options
|Enables you to specify a period of inactivity after which a MultiTail window will close. This can be useful if you are monitoring a large number of files at the same time but only care about those that are actively being updated. Though this option can be convenient, you should use it with care because after a window closes it will not reopen if new output is written to it.|
|Enables you to specify the names of the files that you want to monitor. Ordinarily, you simply specify the names of the files that you want to monitor by listing their names on the command line. Preceding a file name with the |
|Enables you to specify a period of time (N seconds) of inactivity that will cause a mark line to be displayed in any MultiTail output window. For example, if you are monitoring the /var/log/auth.log file and no authentication-related messages have been logged to that file in N seconds, MultiTail displays a highlighted line such as |
|Causes MultiTail to suppress display of repeated messages in the same window, displaying a summary message ("Last message repeated N times") instead.|
Figure 3 shows the output from the following command on a sample machine after 60 seconds:
multitail --mark-interval 10 --no-repeat auth.log \ daemon.log kern.log lpr.log mail.err mysql.err \ wpa_supplicant.log messages
Figure 3. More advanced log file display in MultiTail
As you can see from Figure 3, MultiTail divides the available height of your terminal window between all the files that you want to monitor. In most cases, the last one or two lines of log file output is sufficient to identify any truly important messages that are being displayed in one of your log files.
The options used in the previous example are only the tip of the MultiTail iceberg. As you can see from the figures, MultiTail supports various types of highlighting, including pattern-based highlighting that makes it easy to spot messages associated with a specific device or that contain a specific keyword.
multitail command is not limited to simply monitoring existing
files. It also provides options that let you monitor the output of
a command (
-l command) that produces
continuous output. For quick-running commands, the
-r N option enables you to repeat a given
command after N seconds.
Monitoring command output
Sometimes it's more important to know that a given log file is being written, and when, than it is to know what's being written to it. Apache's access log is a good example—heavily used Web servers can generate thousands of log messages in a few minutes, each of which provides detailed information that isn't really useful. In cases such as these, what most systems administrators actually care about is that the log is constantly growing—in other words, that Apache is running and being accessed.
You can certainly do this visually by repeatedly running a
command such as
ls -l and
comparing file sizes. As mentioned at the end of the previous
section, you can also use the
multitail command to monitor command
output and repeat a specified command multiple
times. Unfortunately, this requires a few command-line options:
multitail -Rc 2 -l "ls -l access.log error.log"
watch command, which is installed by default on most Linux
systems, provides a simpler solution. The
watch command automatically runs a
specified command at specified intervals, with a default interval
of two seconds, and does exactly the right thing when combined
with a command such as
ls -l. Figure 4 shows the output from the
watch ls -l access.log error.log command.
Figure 4. Monitoring log file status using
multitail command, the
watch command runs until
interrupted by a keyboard command such as Control-C.
watch command allows you to watch the output of a program
change over time, which can be especially convenient when looking
for changes to file sizes that reflect debugging output, download
or file creation state, memory or disk usage, and so on. For
example, running a command such as
/proc/meminfo provides a simple way of viewing memory use
on your system in real-time. When monitoring a command in which
multiple lines differ each time that it is executed, you can specify the
-d option to highlight the differences between successive
runs of the command. You can also use the
--differences=cumulative option to accumulate highlighted
differences between all runs of the specified command.
watch commands on different systems
watch command is part of the procps package, a GPL v2 package
that also provides commands such as
vmstat, and so on. Be
careful if you are using a UNIX or UNIX-like system other than
Linux, because other operating systems provide commands called
watch that are completely different. For example, on Berkeley Software Distribution (BSD)-inspired
watch is usually called
gnuwatch, because these systems have a different
command that monitors the processes running on a specified physical or virtual terminal. Similarly, IBM® AIX® operating systems provide a
command that monitors the processes that are created by a
If your system doesn't provide the standard
watch command and you
don't have the time or inclination to build the procps package for
your system, you can easily simulate it with a simple shell
script. The shell script shown in Listing 1, simplewatch.sh, provides much
of the same functionality as the
Listing 1. A shell equivalent for the
#!/bin/sh while [ 1 ] ; do clear echo "Command: $*" date echo "" ( $* ) sleep 2 done
Figure 5 shows the output from running the
simplewatch.sh ls -lt \*.log command in
the /var/log directory on a sample
OpenSolaris system. Quoting any wildcards that you pass to this
script isn't necessary but helps simplify the output that it
Figure 5. Output from running the
simplewatch.sh ls -lt \*.log command
Monitoring log files, file creation and size, and
differences between subsequent runs of a specific command are
common systems administration tasks. The
commands discussed in this article help automate these potentially
boring tasks, making it easy to keep an eye on various aspects of
your systems while also getting other work done.
- See the MultiTail home page for additional examples, complete documentation, and so on.
- Browse the technology bookstore for books on this and other technical topics.
Download the latest source code for the package that includes the
tailprogram from the GNU Coreutils home page, or view the complete documentation online.
Download the latest source code for the package that includes the
watchprogram from the procps home page.