![Close [x]](http://dw1.s81c.com/i/c.gif){kind=small}
Log files provide summary information that is critical to seeing
how a system or a specific server is behaving. On UNIX and
UNIX-like systems, log files are simply text files that are
continually appended, either by the server processes that
they are associated with or by system-logging daemons. All UNIX
and UNIX-like systems use a general-purpose system-logging daemon,
usually referred to as syslogd or simply syslog, which handles log
output from many different applications and servers. Linux® systems
also run klogd, a specialized daemon that specifically monitors
kernel messages.
Classic log monitoring with the tail command
The classic command used to monitor new output in system or server
logs (referred to simply as logs in the rest of this
article) is the tail command, which shows the last few lines in a
specified file (the last 10 lines by default). When executed as
tail filename, the tail command exits
after displaying the last lines from the specified file, which
isn't all that helpful when you want to monitor updates to a log file. To
actively monitor a log file, the tail command is most often
invoked with the -f option, which displays the last lines
of the file and then continues to display subsequent lines as they
are added to the specified file.
It's often necessary (or at least useful) to monitor multiple log
files simultaneously. Specific server processes that you may want
to monitor usually write to their own log files, while syslog
sends different types of messages to different log files
(depending upon how it is configured in /etc/syslog.conf). As an example, on server
systems running Ubuntu, it is often useful to monitor both /var/log/auth.log (where authentication
requests and changes are logged) and /var/log/kern.log (where kernel messages are
logged). Similarly, on server systems running the Apache Web
server, it is useful to monitor both the standard Apache log
files: /var/log/apache2/access.log and /var/log/apache2/error.log.
Many systems administrators, including this author in days gone by, simply
start multiple xterms (or other graphical console/shell
applications), each one of which runs the tail -f
command to monitor a specific log file. That provides the
necessary information but sacrifices a tremendous amount of screen
real estate regardless of the size of the font that you use. As
discussed in the rest of this article, the free and open source
community provides a few advanced alternatives to this somewhat
primitive file monitoring solution.
Monitoring multiple log files simultaneously
MultiTail is a utility that provides a superset of the basic
functionality of the tail command, enabling you to monitor
multiple files at the same time within the same terminal
application. MultiTail (executed as multitail) is a GNU Public
License (GPL) Version 2 application that is available in the
repositories of most UNIX-like operating systems and can easily
be built from source code. The only significant requirement for
building multitail is the ncurses library, which enables
pseudo-graphical operations to be done within terminal windows.
The simplest way of executing MultiTail is to specify the names of
the files that you want to monitor on the command line. For
example, the multitail access.log error.log command enables
you to monitor both the primary log files for an Apache Web
server, as shown in Figure 1.
Figure 1. Viewing multiple Apache logs in MultiTail
MultiTail has a huge number of options, all of which are documented in its traditional UNIX man page. When running MultiTail in a terminal session, you can request help, actively modify the display, search for specific text, and so on by using a variety of keyboard commands. You can access MultiTail's built-in help at any time by clicking Control-h, which displays an overlaid help window, as shown in Figure 2.
Figure 2. Viewing help in MultiTail
You can scroll through this window using the cursor keys and close it by typing q or Control-g. (MultiTail's help is also bound to the F1 key, but graphical desktops such as GNOME usually intercept that key sequence and display their own online help instead.)
Popular options for the multitail command
Some of the more commonly used MultiTail command-line options are described in Table 1.
Table 1. Popular
multitail command options| Option | Description |
|---|---|
--closeidle N | Enables you to specify a period of inactivity after which a MultiTail window will close. This can be useful if you are monitoring a large number of files at the same time but only care about those that are actively being updated. Though this option can be convenient, you should use it with care because after a window closes it will not reopen if new output is written to it. |
-i file | Enables you to specify the names of the files that you want to monitor. Ordinarily, you simply specify the names of the files that you want to monitor by listing their names on the command line. Preceding a file name with the -i option enables you to monitor a file whose name begins with a dash (-). Without using the -i option, any file whose name begins with a dash is assumed to be a command-line option. (This is much like the use of the -- option in commands such as rm.) |
--mark-interval N | Enables you to specify a period of time (N seconds) of inactivity that will cause a mark line to be displayed in any MultiTail output window. For example, if you are monitoring the /var/log/auth.log file and no authentication-related messages have been logged to that file in N seconds, MultiTail displays a highlighted line such as ---auth.log DATE TIME--- in the MultiTail output window for that log file. This is especially handy when you are monitoring multiple log files because it makes it easy to quickly identify the log files to which no new messages have been logged. |
--no-repeat | Causes MultiTail to suppress display of repeated messages in the same window, displaying a summary message ("Last message repeated N times") instead. |
Figure 3 shows the output from the following command on a sample machine after 60 seconds:
multitail --mark-interval 10 --no-repeat auth.log \
daemon.log kern.log lpr.log mail.err mysql.err \
wpa_supplicant.log messages
|
Figure 3. More advanced log file display in MultiTail
As you can see from Figure 3, MultiTail divides the available height of your terminal window between all the files that you want to monitor. In most cases, the last one or two lines of log file output is sufficient to identify any truly important messages that are being displayed in one of your log files.
The options used in the previous example are only the tip of the MultiTail iceberg. As you can see from the figures, MultiTail supports various types of highlighting, including pattern-based highlighting that makes it easy to spot messages associated with a specific device or that contain a specific keyword.
The multitail command is not limited to simply monitoring existing
files. It also provides options that let you monitor the output of
a command (-l command) that produces
continuous output. For quick-running commands, the -r N option enables you to repeat a given
command after N seconds.
Sometimes it's more important to know that a given log file is being written, and when, than it is to know what's being written to it. Apache's access log is a good example—heavily used Web servers can generate thousands of log messages in a few minutes, each of which provides detailed information that isn't really useful. In cases such as these, what most systems administrators actually care about is that the log is constantly growing—in other words, that Apache is running and being accessed.
You can certainly do this visually by repeatedly running a
command such as ls -l and
comparing file sizes. As mentioned at the end of the previous
section, you can also use the multitail command to monitor command
output and repeat a specified command multiple
times. Unfortunately, this requires a few command-line options:
multitail -Rc 2 -l "ls -l access.log error.log"
|
The watch command, which is installed by default on most Linux
systems, provides a simpler solution. The watch command automatically runs a
specified command at specified intervals, with a default interval
of two seconds, and does exactly the right thing when combined
with a command such as ls -l. Figure 4 shows the output from the watch ls -l access.log error.log command.
Figure 4. Monitoring log file status using
watch
Unlike the multitail command, the watch command runs until
interrupted by a keyboard command such as Control-C.
The watch command allows you to watch the output of a program
change over time, which can be especially convenient when looking
for changes to file sizes that reflect debugging output, download
or file creation state, memory or disk usage, and so on. For
example, running a command such as watch cat
/proc/meminfo provides a simple way of viewing memory use
on your system in real-time. When monitoring a command in which
multiple lines differ each time that it is executed, you can specify the
-d option to highlight the differences between successive
runs of the command. You can also use the
--differences=cumulative option to accumulate highlighted
differences between all runs of the specified command.
Different watch commands on different systems
The watch command is part of the procps package, a GPL v2 package
that also provides commands such as ps, top, vmstat, and so on. Be
careful if you are using a UNIX or UNIX-like system other than
Linux, because other operating systems provide commands called
watch that are completely different. For example, on Berkeley Software Distribution (BSD)-inspired
systems, watch is usually called cmdwatch or
gnuwatch, because these systems have a different watch
command that monitors the processes running on a specified physical or virtual terminal. Similarly, IBM® AIX® operating systems provide a watch
command that monitors the processes that are created by a
specified application.
If your system doesn't provide the standard watch command and you
don't have the time or inclination to build the procps package for
your system, you can easily simulate it with a simple shell
script. The shell script shown in Listing 1, simplewatch.sh, provides much
of the same functionality as the watch command.
Listing 1. A shell equivalent for the
watch command
#!/bin/sh
while [ 1 ] ; do
clear
echo "Command: $*"
date
echo ""
( $* )
sleep 2
done
|
Figure 5 shows the output from running the
simplewatch.sh ls -lt \*.log command in
the /var/log directory on a sample
OpenSolaris system. Quoting any wildcards that you pass to this
script isn't necessary but helps simplify the output that it
displays.
Figure 5. Output from running the
simplewatch.sh ls -lt \*.log command
Monitoring log files, file creation and size, and
differences between subsequent runs of a specific command are
common systems administration tasks. The tail, multitail, and watch
commands discussed in this article help automate these potentially
boring tasks, making it easy to keep an eye on various aspects of
your systems while also getting other work done.
| Description | Name | Size | Download method |
|---|---|---|---|
| Shell-script equivalent of the watch command | simplewatch.sh | 1KB | HTTP |
Information about download methods
Learn
-
See the MultiTail home page for additional examples, complete documentation, and
so on.
-
The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
-
New to AIX and UNIX? Visit the New to AIX and UNIX page to learn more.
-
Browse the technology bookstore for books on this and other technical topics.
Get products and technologies
-
Download the latest source code for the package that includes the
GNU
tailprogram from the GNU Coreutils home page, or view the complete documentation online. -
Download the latest source code for the package that includes the
GNU
watchprogram from the procps home page.
Discuss
-
Check out developerWorks
blogs and get involved in the developerWorks
community.
-
Participate in the AIX and UNIX forums:
- AIX Forum
- AIX Forum for developers
- Cluster Systems management
- IBM Support Assistant Forum
- Performance Tools Forum
- Virtualization Forum
- More AIX and UNIX Forums
William von Hagen has been a UNIX systems administrator for more than 20 years and a Linux advocate since 1993. Bill is the author or co-author of books on subjects such as Ubuntu Linux, Xen Virtualization, the GNU Compiler Collection (GCC), SuSE Linux, Mac OS X, Linux file systems, and SGML. He has also written numerous articles for Linux and Mac OS X publications and Web sites. You can reach Bill at wvh@vonhagen.org.
Table of contents
Local resources
Tags
Use the slider bar to see more or fewer tags.
For articles in technology zones (such as Java technology, Linux, Open source, XML), Popular tags shows the top tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), Popular tags shows the top tags for just that product zone.
For articles in technology zones (such as Java technology, Linux, Open source, XML), My tags shows your tags for all technology zones. For articles in product zones (such as Info Mgmt, Rational, WebSphere), My tags shows your tags for just that product zone.
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
