System Accounting -- General Information

Answer

This document contains information on various aspects of system accounting for all levels of AIX Version 5 and 6.

Setting up system accounting

If the accounting software is not installed, it will need to be installed before setting up system accounting. The fileset is bos.acct.

The information generated by system accounting

Accounting generates daily reports in /var/adm/acct/sum. The file names are rprtMMDD, where MM is the month and DD is the date.

The first of each month, a monthly report is created and the daily reports are removed. This is in /var/adm/acct/fiscal and is called fiscrptMM, where MM is the month. The report is for the previous month. For example, fiscrpt02 is the monthly report for January.

The reports contain the following information:

• Lineuse - Amount of time spent on each line (tty, pts), percent of time on line, number of sessions on line, number of logons and number of logoffs
• Daily usage report - Shows the following, per user:
  • Minutes of CPU used, PRIME and NONPRIME
  • Average kilobytes of memory (KCORE), PRIME and NONPRIME
  • Minutes of CONNECT TIME during PRIME and NONPRIME
  • Number of DISK BLOCKS used (from dodisk)
  • Amount of FEES (if computing)
  • Number of PROCESSES
  • Number of SESSIONS
  • Number of DISK SAMPLES
• Daily command summary - Shows the following, per command:
  • Number of times command was run
  • Total KCOREMIN (average kilobytes of memory used times the number of minutes the program ran, the product of the total CPU time, and the mean size in kilobytes)
  • Total CPU minutes
  • Total Real minutes
  • Mean size in kilobytes (memory)
  • Mean CPU time in minutes
  • Hog factor
  • Characters transferred
  • Blocks read
• Monthly command summary - Same as daily command summary
• Last login information

How system accounting is initiated

Follow the steps in the IBM Redbook entitled, "Accounting and Auditing on AIX 5L" section 3.2.1 Starting the accounting system.

Additionally, you may

• Add a call to /usr/sbin/acct/startup in /etc/rc to cause accounting data collection to start when the machine is rebooted.
• Add crontab entries to run accounting reports.

System accounting directories

/usr/sbin/acct

All accounting programs

/usr/lib/acct

Files linked to /usr/sbin/acct

/var/adm

wtmp, pacct and qacct files

/var/adm/acct/fiscal

Monthly reports

/var/adm/acct/nite

Working directory for nighttime accounting processes

/var/adm/acct/sum

Daily reports

The System Management Guide briefly describes each file in these directories.

Space in /var for system accounting

Accounting will cause /var to grow. Running accounting with defaults takes one physical partition (4MB) in /var; this may be increased to at least two physical partitions (8MB). Monitor /var to see if the size will need to be increased. Accounting is not the only reason that /var may be full; the queueing system is also in /var and may take up space if a lot of printing is done.

More detail about space used in /var

Each command that is run adds 40 bytes to the pacct file. So, 25000 commands a day requires 1 MB of free space in /var for the pacct files. This space is freed nightly.

The daily reports could require anywhere from 1-3 MB throughout the month. This space is freed at the end of each month. The monthly reports should require less than 1 MB of free space throughout the year. These numbers will vary with the amount of activity on the system.

Daytime processes

Logins and logouts are logged in /var/adm/wtmp. It is cleared out nightly by runacct. If accounting is not running, this file will grow. This file does not have to exist if accounting is not running, but it is useful. To see an ASCII version of wtmp, /etc/utmp, or /etc/security/failedlogin, use the fwtmp command.

All daily process activity is logged in /var/adm/pacct. Each process completed increases this file by 40 bytes. For heavily used systems, this file can use large amounts of space in /var.

/usr/sbin/acct/ckpacct checks the size of /var/adm/pacct and the amount of free space in /var. It is run from cron and should be run at intervals appropriate for the system.

If /var/adm/pacct is over 1000 blocks, ckpacct will switch the pacct file. This means it will copy pacct to pacct# (# starts with 1 and increases to the next unused number) and clear out pacct again.

If the free space in /var falls below 500 blocks, then ckpacct turns off accounting until space is made available. This will result in loss of accounting data during the period that accounting is turned off. ckpacct will turn accounting on again when more space is available. There is no notification unless the MAILCOM variable is set.

   MAILCOM="mail root adm" 

This can be set in the ckpacct and runacct scripts or in the /etc/environment file. If MAILCOM is set in both places the setting in ckpacct and runacct is used.

Nighttime processes

Accounting is kicked off by cron, usually during the late hours of the day. This occurs if the process is set up according to the set up fax mentioned in the "How system accounting is initiated" section of this document. The scripts that are usually run at night are:

dodisk

Analyzes the amount of disk usage per user

runacct

Creates the daily reports

monacct

Runs once a month to create monthly reports from daily ones

See "About the accounting programs" for more information about these scripts.

System accounting error information

/var/adm/acct/nite/accterr contains the most system accounting error information.

/var/adm/acct/nite/active contains information about the steps that have been completed during the runacct script.

/var/adm/acct/nite/statefile lists the current state of runacct.

It is possible mail will not be received from cron because cron redirects output to the accterr file or to /dev/null; however, if the cron jobs are set up not to do this, there will be mail from cron.

Also, mail will not be received from the runacct script unless the MAILCOM line is uncommented in /usr/sbin/acct/runacct.

About the accounting programs

dodisk

dodisk performs disk usage accounting on all file systems that have account = true in /etc/filesystems. dodisk creates a file for use by runacct called /var/adm/acct/nite/dacct. The dodisk command needs to be started at least 10-30 minutes before runacct to allow it to complete before runacct starts. If the dacct file is not finished before runacct tries to process it, unreliable data will exist in the daily reports.

ckpacct

ckpacct checks /var to ensure it does not run out of space. It also makes sure that /var/adm/pacct does not become too large and unmanageable. It accomplishes this by renaming pacct to pacctxx and starting a new pacct file when pacct grows over 500 disk blocks. The normal interval for running ckpacct is once an hour. It should be run more often on systems that are heavily used. The more commands that are run, the faster the pacct files grow.

runacct

runacct performs daily accounting and generates daily reports in the /var/adm/acct/sum directory. This command is divided into STATEs (procedures). If the process breaks, it can be started again at the correct STATE. Parameters should not be applied when using runacct unless trying to start the process over from a failed attempt. See the following section for more information.

monacct

monacct cleans up daily reports and creates a monthly report in /var/adm/acct/fiscal. See the following section for more information.

Detailed information about runacct

The runacct command can take two arguments; however, they should only be used to start a runacct that previously failed. The documentation states that the command usage is

   runacct [MMDD] [STATE ... ] 

   runacct [MMDD [STATE]] 

If runacct is restarted, use the MMDD for the day that runacct was running (that is, if runacct failed on 0623, run runacct 0623). It will continue at the point of failure. A certain STATE can also be specified at which to start. This is necessary only if a STATE is skipped or redo one that has been done. The valid STATEs are:

   SETUP 
   WTMPFIX 
   CONNECT1 
   CONNECT2 
   PROCESS 
   MERGE 
   FEES 
   DISK 
   QUEUEACCT 
   MERGETACCT 
   CMS 
   USEREXIT 
   CLEANUP 

Any state other than these is invalid and generates errors in the active file.

The following sections list the actions during each state of runacct.

Before the states begin

• Set the statefile for SETUP.
• Set up variables.
• Set up lock files.
• Check /var for sufficient space.
  NOTE: Since free space in /var is checked only at the beginning, running jobs that exhaust the space in /var (such as print jobs) may cause runacct to fail.
• Check for parameters that were passed in:
  • If one parameter was passed in, restart accounting for MMDD at the current STATE that is in statefile.
  • If two parameters were passed in, restart accounting for MMDD at specified STATE.

SETUP - basic set up of files to be used

• Write date and list of files to active file.
• Switch current pacct file.
• Move each pacct file to a file name of Spacct#.MMDD.
• Copy current wtmp file to nite/wtmp.MMDD.
• Append line with current time to end of nite/wtmp.MMDD.
• Clear current wtmp file.
• Write to active file that "file setups complete".
• Set the statefile for WTMPFIX.

WTMPFIX - fix any corruption in the wtmp file

• Clear nite/tmpwtmp and nite/wtmperror.
• Run wtmpfix on nite/wtmp.MMDD.
  • Standard out goes to nite/wtmp.MMDD.
  • Standard error goes to nite/wtmperror.
• Write to active file that "wtmp processing complete."
• Set the statefile for CONNECT1.

CONNECT1 - produce connect time info in ctmp.h format

• Clear the lineuse, reboots, ctmp, and log files in nite directory.
• Run acctcon1 against tmpwtmp (the new wtmp file).
  • Reboot info is written to reboots file (this is the 1st part of the daily report).
  • Lineuse info is written to lineuse file (this is the 2nd part of the daily report).
  • Connect time info is written to ctmp (ctmp.h format).
  • Errors are written to log file (there should not be any).
• Set the statefile for CONNECT2.

CONNECT2 - convert ctmp.h records to tacct records

• Clear ctacct.MMDD file.
• Run acctcon2 with input from ctmp and output to ctacct.MMDD.
• Write to active file that "connect acctg complete".
• Set the statefile for PROCESS.

PROCESS - create process accounting info

• Run acctprc1 against each of the Spacct#.MMDD files.
  • Output to acctprc2, creating corresponding ptacct#.MMDD files.
  • Write to active file for each Spacct#.MMDD file.
• Write to active file that "all process acctg complete for MMDD".
• Set the statefile for MERGE.

MERGE - Merge the ctacct and ptacct files together

• Copy ctacct.MMDD file to daytacct.
• Merge each ptacct#.MMDD file into the daytacct file. (This is done with acctmerge and two temporary files - tmpdayt and daytacct.old.)
• Write to active file that "tacct merge to create daytacct complete."
• Set the statefile for FEES.

FEES - Merge in fee accounting info

• If /var/adm/fee exists then merge fee info into daytacct file.
• Write to active file that "fee processing is complete."
• Set the statefile for DISK.

DISK - Merge in disk accounting info

• If /var/adm/acct/nite/dacct exists (from dodisk) then merge dacct into daytacct file.
• Write to active file that "merged disk records".
• Set up statefile for QUEUEACCT.

QUEUEACCT - merge in queue accounting info

• If /var/adm/qacct exists then merge it into the daytacct file.
• Write to active file that "queueing system records complete".
• Set up statefile for MERGEACCT.

MERGEACCT - create daily tacct files

• Copy nite/daytacct to sum/tacctMMDD file.
• Copy sum/tacct to sum/tacctprev.
• Merge sum/tacctprev and sum/tacctMMDD together into sum/tacct.
• Write to active file that "updated sum/tacct".
• Set up statefile for CMS.

CMS - create command summaries

• Clear sum/daycms.
• Copy sum/cms to sum/cmsprev.
• Run acctcms against Spacct*.MMDD.
  • Output to sum/daycms (a binary file).
• Run acctcms against sum/daycms and sum/cmsprev.
  • Output to sum/cms (a binary file).
• Run acctcms against sum/daycms.
  • Output to nite/daycms (ASCII file).
• Run acctcms against sum/cms.
  • Output to nite/cms (ASCII file).
• Run lastlogin MMDD to update sum/loginlog.
• Write to active file that "command summaries complete".
• Set up statefile for USEREXIT.

USEREXIT - run any extra accounting programs

• If /var/adm/siteacct exists, run it. siteacct should be a script to do additional accounting. It does not exist unless created.
• Set up statefile for CLEANUP.

CLEANUP - Clean up temp files and write daily report

• Clear /var/adm/fee.
• Remove Spacct*.MMDD.
• Run prdaily to create sum/rprtMMDD (daily report).
• Remove nite/lock.
• Remove nite/ptacct#.MMDD and nite/ctacct.MMDD.
• Remove nite/wtmp.MMDD, nite/wtmperrorMMDD, and nite/activeMMDD.
• Move nite/tmpwtmp to nite/owtmp.
• Write to active file that "system accounting completed at TIME."
• Set up statefile for COMPLETE.

Detailed information about monacct

monacct performs these steps:

• Move sum/tacct to fiscal/tacctMM
• Remove sum/tacct*.
• Clear sum/tacct.
• Move sum/cms to fiscal/cmsMM.
• Clear sum/cms.
• Remove sum/rprt*.
• Create fiscal/fiscrptMM from fiscal/tacctMM.
• Append command summary to fiscal/fiscrptMM from fiscal/cmsMM.
• Append lastlogin info to fiscal/fiscrptMM from sum/loginlog.

Additional accounting possibilities

The daily report might be all that is needed; however, the commands a specific user ran can be seen by running acctcom. It generates a file with one line for each command ran and indicates the time the command was run and who ran it. (See product documentation for a complete list of flags for the acctcom command. Only the minimum syntax is used in the examples that follow.)

Since runacct deletes the pacct files, which are needed by acctcom, run acctcom first or save the pacct files before runacct is run.

If acctcom is run before runacct, use the following syntax to run acctcom. Note that the output will be rather large.

   acctcom /var/adm/pacct* > somefile 

To save the pacct files before runacct, the recommended method is to change runacct to save the files before it continues processing:

1. Become the adm user.

2. Run the following commands:

      cd /var/adm 
      mkdir oldpacct     #(directory to save pacct files in) 
   

3. Become the root user.

4. Start an edit session on /usr/sbin/acct/runacct.

5. Find the following line:

      mv ${_i} S${_i}.${_date} 
   

6. Just above the line that you found, add the following:

      cp ${_i} /var/adm/oldpacct/${_i} 
   

If the modified runacct is run before acctcom, use the following syntax to run acctcom:

   acctcom /var/adm/oldpacct/pacct* > somefile 
   rm /var/adm/oldpacct/pacct* 

Recommended fixes

See "AIX Fix Central", http://www-912.ibm.com/eserver/support/fixes/fixcentral/main/pseries/aix for additional fixes. Most of the accounting problems will be dealt within the fileset bos.acct.

Related documentation

• System accounting, which comes from BDS or System V, is documented in the AIX System Management Guide.
• Additional information can also be found in the following:
  UNIX Administration Guide for System V
  (Chapter 7 is on System Accounting)
  by Rebecca Thomas and Rik Sarrow
  Publisher: Prentice and Hall
  ISBN 0-13-942889-5
• "Accounting and Auditing on AIX 5L" IBM Form Number SG24-6396-00