sar Command

Purpose

Collects, reports, or saves system activity information.

Syntax

/usr/sbin/sar [ { -A [ -M ] | [ -a ] [ -b ] [ -c ] [ -d ][ -k ] [ -m ] [ -q ] [ -r ] [ -u ] [ -v ] [ -w ] [ -y ] [ -M ] } ] [ -P processoridentifier, ... | ALL | RST [-O {sortcolumn=col_name[,sortorder={asc|desc}][,topcount=n]}]]] [ [ -@ wparname ] [ -e[YYYYYMMDD]hh [ :mm [ :ss ] ] ] [ -ffile ] [ -iseconds ] [ -ofile ] [ -s[YYYYYMMDD]hh [ :mm [:ss ] ] ][-x] [ Interval [ Number ] ]

sar [-X [-o filename]] [interval[count]]

Description

The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. The accounting system, based on the values in the number and interval parameters, writes information the specified number of times spaced at the specified intervals in seconds. The default sampling interval for the number parameter is 1 second. The collected data can also be saved in the file specified by the -o file flag.

The sar command generates an XML file when the –X option is specified.

The sar command extracts and writes to standard output records previously saved in a file. This file can be either the one specified by the -f flag or, by default, the standard system activity daily data file, the /var/adm/sa/sadd file, where the dd parameter indicates the current day.

Without the -P flag, the sar command reports system-wide (global among all processors) statistics, which are calculated as averages for values expressed as percentages, and as sums otherwise. If the -P flag is given, the sar command reports activity which relates to the specified processor or processors. If -P ALL is given, the sar command reports statistics for each individual processor, followed by system-wide statistics. If -P ALL is used in a workload partition environment and the WPAR is associated with an rset registry, the resource set statistics and the system-wide statistics are displayed; the processors that belong to the resource set are prefixed with an asterisk symbol (*).

You can select information about specific system activities using flags. If you do not specify any flags, you select only system and WPAR unit activity. Specifying the -A flag selects all activities. The sar command prints the number of processors and the number of disks that are currently active before starting to print the statistics.

The default version of the sar command (processor utilization report) might be one of the first facilities the user runs to begin system activity investigation, because it monitors major system resources. If processor utilization is near 100 percent (user + system), the workload sampled is processor-bound. If a considerable percentage of time is spent in I/O wait, it implies that processor execution is blocked waiting for disk I/O. The I/O may be required file accesses or it may be I/O associated with paging due to a lack of sufficient memory.

Note: The time the system spends waiting for remote file access is not accumulated in the I/O wait time. If CPU utilization and I/O wait time for a task are relatively low, and the response time is not satisfactory, consider investigating how much time is being spent waiting for remote I/O. Since no high-level command provides statistics on remote I/O wait, trace data may be useful in observing this. If there is a change in system configuration that affects the output of the sar command, sar prints the average values up to the current iteration and then a warning message about the configuration change. It then continues the output, after printing the updated system configuration information.

Methods Used to Compute CPU Disk I/O Wait Time

The AIX® operating system contains enhancements to the method used to compute the percentage of processor time spent waiting on disk I/O (wio time). The wio time is reported by the commands sar (%wio), vmstat (wa) and iostat (% iowait).

At each clock interrupt on each processor (100 times a second per processor), a determination is made as to which of the four categories (usr/sys/wio/idle) to place the last 10 ms of time. If the processor was busy in usr mode at the time of the clock interrupt, then usr gets the clock tick added into its category. If the processor was busy in kernel mode at the time of the clock interrupt, then the sys category gets the tick. If the processor was not busy, a check is made to see if any I/O to disk is in progress. If any disk I/O is in progress, the wio category is incremented. If no disk I/O is in progress and the processor is not busy, the idle category gets the tick. The inflated view of wio time results from all idle processors being categorized as wio regardless of the number of threads waiting on I/O. For example, systems with just one thread doing I/O could report over 90 percent wio time regardless of the number of processors it has.

The AIX operating system marks an idle processor as wio if an outstanding I/O was started on that processor. This method can report much lower wio times when just a few threads are doing I/O and the system is otherwise idle. For example, a system with four processors and one thread doing I/O will report a maximum of 25 percent wio time. A system with 12 processors and one thread doing I/O will report a maximum of 8 percent wio time. NFS client reads/writes go through the VMM, and the time that biods spend in the VMM waiting for an I/O to complete is now reported as I/O wait time.

If multiple samples and multiple reports are desired, it is convenient to specify an output file for the sar command. Direct the standard output data from the sar command to /dev/null and run the sar command as a background process. The syntax for this is:

sar -A -o data.file interval count > /dev/null &

All data is captured in binary form and saved to a file (data.file). The data can then be selectively displayed with the sar command using the -f option.

The sar command calls a process named sadc to access system data. Two shell scripts (/usr/lib/sa/sa1 and /usr/lib/sa/sa2) are structured to be run by the cron command and provide daily statistics and reports. Sample stanzas are included (but commented out) in the /var/spool/cron/crontabs/adm crontab file to specify when the cron daemon should run the shell scripts. Collection of data in this manner is useful to characterize system usage over a period of time and determine peak usage hours.

You can insert a dummy record into the standard system activity daily data file at the time of system start by un-commenting corresponding lines in the /etc/rc script. The sar command reports time change not positive for any record where processor times are less than the previous record. This occurs if you reboot the system with the dummy record insertion lines in /etc/rc commented out.

Beginning with AIX 5.3, the sar command reports utilization metrics physc and %entc which are related to Micro-Partitioning® and simultaneous multithreading environments. These metrics will only be displayed on Micro-Partitioning and simultaneous multithreading environments. physc indicates the number of physical processors consumed by the partition (in case of system wide utilization) or logical processor (if the -P flag is specified) and %entc indicates the percentage of the allocated entitled capacity (in case of system wide utilization) or granted entitled capacity (if the -P flag is specified). When the partition runs in capped mode, the partition cannot get more capacity than it is allocated. In uncapped mode, the partition can get more capacity than it is actually allocated. This is called granted entitled capacity. If the -P flag is specified and there is unused capacity, sar prints the unused capacity as separate processor with cpu id U.

Beginning with AIX 6.1, the sar command reports the utilization metric %resc, which is related to the workload partition (WPAR) environment. The %resc metric indicates the percentage of processor resource that the WPAR consumes. This field is displayed only if the processor-resource limit is enforced in the WPAR. The sar -P command reports the resource set (RSET) utilization metrics R for the WPAR.

Restriction: The sar command only reports on local activities.

You could also use the System Management Interface Tool (SMIT) smit sar fast path to run this command.

Flags

Item Description
-@ wparname The -@ flag specifies that the command reports the processor use in WPAR from the global environment. The wparname parameter specifies which WPAR processor statistics are to be reported.
Note: The -@ flag is not supported when executed within a workload partition.
Note: Do not use the -@ flag with the -d, -r, -y, -f, or -X flags.
-A Without the -P flag, using the -A flag is equivalent to specifying -abcdkmqruvwy. When used with the -P flag, the -A is equivalent to specifying -acmuw. Without the -M flag, headers are only printed once in multiple lines grouped together before the data for the first interval. When this flag is used with the -M flag, each line of data at each iteration is preceded by the appropriate header.
-a Reports use of file access system routines specifying how many times per second several of the system file access routines have been called. When used with the -P flag, the information is provided for each specified processor; otherwise, it is provided only system-wide. The following values are displayed:
dirblk/s
Number of 512-byte blocks read by the directory search routine to locate a directory entry for a specific file.
iget/s
Calls to any of several i-node lookup routines that support multiple file system types. The iget routines return a pointer to the i-node structure of a file or device.
lookuppn/s
Calls to the directory search routine that finds the address of a v-node given a path name.
-b Reports buffer activity for transfers, accesses, and cache (kernel block buffer cache) hit ratios per second. Access to most files in Version 3 bypasses kernel block buffering and therefore does not generate these statistics. However, if a program opens a block device or a raw character device for I/O, traditional access mechanisms are used making the generated statistics meaningful. The following values are displayed:
bread/s, bwrit/s
Reports the number of block I/O operations. These I/Os are generally performed by the kernel to manage the block buffer cache area, as discussed in the description of the lread/s value.
lread/s, lwrit/s
Reports the number of logical I/O requests. When a logical read or write to a block device is performed, a logical transfer size of less than a full block size may be requested. The system accesses the physical device units of complete blocks and buffers these blocks in the kernel buffers that have been set aside for this purpose (the block I/O cache area). This cache area is managed by the kernel, so that multiple logical reads and writes to the block device can access previously buffered data from the cache and require no real I/O to the device. Application read and write requests to the block device are reported statistically as logical reads and writes. The block I/O performed by the kernel to the block device in management of the cache area is reported as block reads and block writes.
pread/s, pwrit/s
Reports the number of I/O operations on raw devices. Requested I/O to raw character devices is not buffered as it is for block devices. The I/O is performed to the device directly.
%rcache, %wcache
Reports caching effectiveness (cache hit percentage). This percentage is calculated as: [(100)x(lreads - breads)/ (lreads)].
-c Reports system calls. When used with the -P flag, the information is provided for each specified processor; otherwise, it is provided only system-wide. The following values are displayed:
exec/s, fork/s
Reports the total number of fork and exec system calls.
sread/s, swrit/s
Reports the total number of read/write system calls.
rchar/s, wchar/s
Reports the total number of characters transferred by read/write system calls.
scall/s
Reports the total number of system calls.
Tip: The sar command itself can generate a considerable number of reads and writes depending on the interval at which it is run. Run the sar statistics without the workload to understand the sar command's contribution to your total statistics.
-d Reports activity for each block device with the exception of tape drives. The following data is reported:
%busy
Reports the portion of time the device was busy servicing a transfer request.
avque
Reports the average number of requests waiting to be sent to disk.
read/s, write/s, blk/s
Reports the read-write transfers from or to a device in kilobytes/second.
avwait, avserv
Average wait time and service time per request in milliseconds.
Restriction: The -d flag is restricted in workload partitions.
-e[YYYYMMDD] hh[:mm[:ss]] Sets the ending time of the report. The default ending time is 18:00.
  • If you specify the year, month, and date in the YYYYMMDD format, then the -x flag is turned on implicitly.
  • If you do not specify the year, month, and date in the YYYYMMDD format, then the year, month, and date are considered to be that of the first record in the activity data file that matches the specified time
-f file Extracts records from the file (created by -o file flag). The default value of the file parameter is the current daily data file, the /var/adm/sa/sadd file.
Restriction: If you specify the [ interval [ number ] ] parameter, the -f flag is ignored. The -f flag is restricted in workload partitions.
-i seconds Selects data records at seconds as close as possible to the number specified by the Seconds parameter. Otherwise, the sar command reports all seconds found in the data file.
-k Reports kernel process activity. The following values are displayed:
kexit/s
Reports the number of kernel processes terminating per second.
kproc-ov/s
Reports the number of times kernel processes could not be created because of enforcement of process threshold limit.
ksched/s
Reports the number of kernel processes assigned to tasks per second.
-M Enables multiple headers in output when used with at least two combinations of [abckmqruvwy] or with the -A flag. In this mode, each line of data is preceded by the corresponding header at each iteration.
Restriction: This flag is ignored when used without [interval [number]].
-m Reports message (sending and receiving) and semaphore (creating, using, or destroying) activities per second. When used with the -P flag, the information is provided for each specified processor; otherwise, it is provided only system-wide. The following values are displayed:
msg/s
Reports the number of IPC message primitives.
sema/s
Reports the number of IPC semaphore primitives.
-o file Saves the readings in the file in binary form. Each reading is in a separate record and each record contains a tag identifying the time of the reading.
-P processoridentifier, ... | ALL |RST Reports per-processor statistics for the specified processor or processors. Specifying the ALL keyword reports statistics for each individual processor, and globally for all processors. Specifying the RST option reports statistics for the processors present in the rset registry that is associated with the WPAR. Of the flags that specify the statistics to be reported, only the -a, -c, -m, -u, and -w flags are meaningful with the -P flag in the global environment. In the WPAR environment, do not use any flag with the -P flag.
Note: The statistics for each processor that the sar command reports for WPAR are always system-wide.
-q Reports queue statistics. The following values are displayed:
runq-sz
Reports the average number of kernel threads in the run queue.
%runocc
Reports the percentage of the time the run queue is occupied.
swpq-sz
Reports the average number of kernel threads that are waiting in the virtual memory manager queue for resource, input, or output.
%swpocc
Reports the percentage of the time the swap queue is occupied.
Tip: A blank value in any column indicates that the associated queue is empty.
-r Reports paging statistics. The following values are displayed:
cycle/s
Reports the number of page replacement cycles per second.
fault/s
Reports the number of page faults per second. This is not a count of page faults that generate I/O, because some page faults can be resolved without I/O.
slots
Reports the number of free pages on the paging spaces.
odio/s
Reports the number of non paging disk I/Os per second.
Restriction: The -r flag is restricted in workload partitions.
-s[YYYYMMDD] hh[:mm[:ss]] Sets the starting time of the data, causing the sar command to extract records time-tagged at, or following, the time specified. The default starting time is 08:00.
  • If you specify the year, month, and date in the YYYYMMDD format, then the -x flag is turned on implicitly.
  • If you did not specify the year, month, and date in the YYYYMMDD format, then the year, month, and date are considered to be that of the first record in the activity data file that matches the specified time.
-u Reports per processor or system-wide statistics. When used with the -P flag, the information is provided for each specified processor; otherwise, it is provided only system-wide. Because the -uflag information is expressed as percentages, the system-wide information is simply the average of each individual processor's statistics. Also, the I/O wait state is defined system-wide and not per processor. The following values are displayed:
%idle
Reports the percentage of time the processor or processors were idle with no outstanding disk I/O requests.
%sys
Reports the percentage of time the processor or processors spent in execution at the system (or kernel) level.
%usr
Reports the percentage of time the processor or processors spent in execution at the user (or application) level.
%wio
Reports the percentage of time the processor(s) were idle during which the system had outstanding disk/NFS I/O request(s). See detailed description above.
physc
Reports the number of physical processors consumed. This data will be reported if the partition is dedicated and enabled for donation, or is running with shared processors or simultaneous multithreading enabled.
%entc
Reports the percentage of entitled capacity consumed. This will be reported only if the partition is running with shared processors. Because the time base over which this data is computed can vary, the entitled capacity percentage can sometimes exceed 100%. This excess is noticeable only with small sampling intervals.
%resc
Reports the percentage of processor resource consumed. This metric is applicable only for the WPAR environment. It is reported only if the WPAR enforces processor-resource limit.
Tips:
  • The sar command reports system unit activity if no other specific content options are requested. If the -P flag is used and the partition is running with shared processors, and if the partition capacity usage is what is allocated, then a processor row with cpuid U will be reported to show the system-wide unused capacity. If the partition is running with shared processors in uncapped mode, then %entc will report the percentage of granted entitled capacity against each processor row and percentage of allocated entitled capacity in the system-wide processor row. The individual processor utilization statistics is calculated against the actual physical consumption (physc). The system wide statistics is computed against the entitlement and not physical consumption. However, in an uncapped partition, the system wide statistics is still calculated against the physical consumption.
  • Since the time base over which the data is computed varies, the sum of all of the %utilization fields (%user, %sys, %idle, and %wait) can exceed 100 percent.
-v Reports status of the process, kernel-thread, i-node, and file tables. The following values are displayed:
file-sz, inod-sz, proc-sz , thrd-sz
Reports the number of entries in use for each table.
-w Reports system switching activity. When used with the -P flag, the information is provided for each specified processor; otherwise, it is provided only system-wide. The following value is displayed:
pswch/s
Reports the number of context switches per second.
-y Reports tty device activity per second.
canch/s
Reports tty canonical input queue characters. This field is always 0 (zero).
mdmin/s
Reports tty modem interrupts.
outch/s
Reports tty output queue characters.
rawch/s
Reports tty input queue characters.
revin/s
Reports tty receive interrupts.
xmtin/s
Reports tty transmit interrupts.
Restriction: The -y flag is restricted in workload partitions.
-x Displays the date and time for each entry. The -x flag is turned on implicitly whenever the user specifies the data in the YYYYMMDD format for the -s flag or the -e flag.
-OOptions Allows users to specify the command option.

-O options=value...

Following are the supported options:

  • sortcolumn = Name of the metrics in the sar command output
  • sortorder = [asc|desc]
  • topcount = Number of CPUs to be displayed in the sar command sorted output
-X Generates the XML output. The default file name is sar_DDMMYYHHMM.xml unless the user specifies a different file name using with the –o option.
-o Specifies the file name for the XML output.

Security

Access Control: These commands should grant execute (x) access only to members of the adm group.

Attention RBAC users and Trusted AIX users: This command can perform privileged operations. Only privileged users can run privileged operations. For more information about authorizations and privileges, see Privileged Command Database in Security. For a list of privileges and the authorizations associated with this command, see the lssecattr command or the getcmdattr subcommand.

Examples

  1. To report system unit activity, enter the following command:
    sar
  2. To report current tty activity for each 2 seconds for the next 40 seconds, enter the following command:
    sar  -y  -r 2 20
  3. To watch system unit for 10 minutes and sort data, enter the following command:
    sar  -o temp 60 10
  4. To report processor activity for the first two processors, enter the following command:
    sar  -u  -P 0,1
    This produces output similar to the following:
    cpu  %usr  %sys  %wio  %idle
    0      45    45     5      5
    1      27    65     3      5
  5. To report message, semaphore, and processor activity for all processors and system-wide, enter the following command:
    sar  -mu  -P ALL
    On a four-processor system, this produces output similar to the following (the last line indicates system-wide statistics for all processors) :
    cpu  msgs/s  sema/s  %usr  %sys  %wio  %idle
    0      7       2       45    45     5     5
    1      5       0       27    65     3     5
    2      3       0       55    40     1     4
    3      4       1       48    41     4     7
    -     19       3       44    48     3     5
  6. To see physical processor consumed and entitlement consumed for all processors system-wide, run sar command in a shared processor logical partition machine, as follows:
    sar –P ALL
    On a two-logical processor system, this produces output similar to the following (the last line indicates system-wide statistics for all processors, and the line with cpuid U indicates the system-wide Unused capacity):
    cpu    %usr    %sys    %wio   %idle   physc   %entc
     0        0       0       0     100    0.02     3.1
     1        0       0       0     100    0.00     1.0
     U        -       -       0      96    0.48    96.0
     -        0       0       0     100    0.02     4.0
  7. To report system call, kernel process, and paging activities with separate headers for each of the three lines of data at each iteration for every 2 seconds for the next 40 seconds, enter the following command:
    sar -Mckr 2 20
  8. To report all activities with multiple sets of headers for every 2 seconds for the next 40 seconds, enter the following command:
    sar -MA 2 20
  9. To report the processor use statistics in a WPAR from the global environment, enter the following command:
    sar -@ wparname
  10. To report the processor activities for all of the processors present in the rset registry associated with the WPAR from inside a WPAR, enter the following command:
    sar -P RST 1 1
    In a WPAR that is associated with an RSET of two logical processors, the previous command generates a report similar to the following:
    19:34:39 cpu    %usr    %sys    %wio   %idle   physc
    19:34:40  0        0       2       0      98    0.54
              1        0       0       0     100    0.46
              R        0       1       0      99    1.00
  11. To report all of the processor activities from inside a WPAR, enter the following command:
    sar -P ALL 1 1
    In a WPAR that is associated with an RSET of two logical processors, the previous command generates a report similar to the following:
    19:34:39 cpu    %usr    %sys    %wio   %idle   physc
    19:34:40 *0        0       2       0      98    0.54
             *1        0       0       0     100    0.46
              R        0       1       0      99    1.00
              -        0       1       0      99    1.00
  12. To display the sorted output for the column cswch/s with the –w flag, enter the following command:
    sar -w -P ALL  -O sortcolumn=cswch/s 1 1 
  13. To list the top ten CPUs, sorted on the scall/s column, enter the following command:
    sar -c -O sortcolumn=scall/s,sortorder=desc,topcount=10 -P ALL 1

Files

Item Description
/usr/sbin/sar Contains the sar command.
/bin/sar Indicates the symbolic link to the sar command.
/var/adm/sa/sadd Indicates the daily data file, where the dd parameter is a number representing the day of the month.