Problems with the /etc/utmp File in AIX

Answer

The /etc/utmp file is used by the who,w and uptime commands to display when
the system was last booted and who is currently logged in. This document
describes possible solutions for a corrupted utmp file and is applicable
to all AIX versions.


The w command reports incorrect idle time
If the w command shows idle time greater than the uptime of the system, install the latest level of fileset bos.rte.misc_cmds.
See the section "Fixes/problems/updates" below.

Indicators of utmp corruption
Corruption of the utmp file shows up in two ways:
* The uptime and w commands show a time greater than 8000 days since the system was last booted.
* Users are shown as still logged in when in fact they are not.

Both types of corruption have many causes because both AIX commands and third party applications write to the utmp file.

Problem: uptime greater than 8000 days
If record number 0 is overwritten by anyone (normally a third party program), the uptime shows up as greater than 8000 days.

To correct the invalid boot time you must reboot the system. The utmp file is recreated with each boot.

To attempt to discover who or what overwrote the first entry in the file, use the following command to create a readable version of the utmp file and look at record 0:
NOTE: The fwtmp command must first be installed from the bos.acct fileset.

   # /usr/sbin/acct/fwtmp < /etc/utmp  > /tmp/utmp.out

A valid entry looks something like this:

system boot 2 0 0000 0000 1200092500

Instead of the system boot entry, you will probably find an entry like:

jones pts/2 7 19193 0000 0000 1201634700

This output means that the time stamp was corrupted by whatever program jones on pts/2 used to login. A program should never overwrite the first two entries in the utmp file. You would have to talk with user "jones" to see what he did. This is almost always caused by a third party program that is incorrectly writing to the utmp file or a corrupted file system where the data is invalid.

The "who" or "w" commands show users logged in when they are not
When a user logs into the system, the /usr/sbin/getty or /usr/sbin/login programs write an entry in /etc/utmp like:

sandy pts/23 pts/23 7 42300 0000 0000 818973357

The fields output by fwtmp are:

Field #1 = user's name
Field #2 = /etc/inittab entry id
Field #3 = tty used to login on
Field #4 = type of entry
Field #5 = PID (process id)
Field #6 = termination status
Field #7 = exit status
Field #8 = time the entry was made (in seconds since the Unix epoch)
Field #9 = hostname

Note: Not all fields may be present in the utmp file.
The types of entries can be seen by examining the /usr/include/utmp.h file under the ut_type variable. Type 7 is a USER_PROCESS.

When a user logs out, it is the responsibility of the last process running to update the entry in the utmp file. After a logout, the entry should look like:

pts/23 pts/23 8 42300 0000 0000 818973357

The user name is erased and the state is changed from 7 to 8 (DEAD_PROCESS). The who command will only show entries that are in state 7.

How to determine what program caused the corruption
If this is an ongoing problem, auditing can help determine what is writing to the utmp file. Set up auditing and monitor writes to the utmp file. Additionally you can set up a cron job run the who command each minute and send the output to a file. When you notice corruption with the who or w command, check the cron job output file to determine when the corruption occurred. Then look in the audit trail to determine what process was writing to the utmp file at the time the corruption occurred.

This is an example of an audit log output:

event      login  status   time                     command
---------- ------ -------  ----------------------   -------
UTMP_WRITE root   OK       Tue Dec 19 17:00:29 1995 telnetd

The example above shows that the telnetd daemon wrote to the file at 17:00:29.

Known problems
For fixes related to utmp corruption, install the latest level of the following filesets:

bos.rte.misc_cmds
bos.rte.tty
devices.tty.rte
bos.net.tcp.server
bos.net.tcp.client

Fixes for AIX can be downloaded via the Internet from Fix Central
http://www-912.ibm.com/eserver/support/fixes/fcgui.jsp

How to fix the utmp file
Rebooting clears the utmp file and is the recommended method of correcting the results of corruption.