pai - Record and report PAI counters

Red Hat Enterprise Linux 9.2 LPAR mode

Use the pai command to record counter data for Processor Activity Instrumentation Facility (PAI) counters in sampling mode and to create reports from the recorded data. For more information about PAI counters, see Using the CPU Processor Activity Instrumentation Facility.

pai syntax

The pai command records PAI counters in a ring buffer, for all CPUs or for selected CPUs.

Recording stores data in files paicrypto.<cpu_nr> for cryptographic counters or painnpa.<cpu_nr> for analytic counters. Data is stored by CPU. The <cpu_nr> extension of the file names specifies the CPU number as a three-digit decimal number.

The files are created in the working directory. Existing files are overwritten. Reporting evaluates files that are created by recording.

Recording options and parameters

The −c and −n options run the pai command in recording mode.

-V or --verbose: for recording, displays the read position in the ring buffer, the file name, and the hexadecimal offset for each sample header.
-m or --mapsize <size>: specifies the size of the ring buffer that holds the collected data. The value specifies the number of 4 KB pages to be allocated and must be a power of 2. The default is 131072 for 512 MB. The ring buffer is created with the mmap(2) system call. This option must be specified before -c.
-i or --interval <interval>: specifies the waiting time, in milliseconds, between ring buffer read operations during recording. The default is 1000 milliseconds.
-c or --crypto: records data for cryptographic counters.
-n or --nnpa: records data for analytic counters.
<cpulist>: comma-separated list of CPU numbers and ranges. In a range, a hyphen separates the first CPU number from the last CPU number. If no CPUs are listed, recording extends to all CPUs.

:<data>

specification for including extra data or for filtering the data to be recorded. The specification is a string of alphabetic characters, where each character adds a data point or a filter. The specification is not case-sensitive.

The following characters add data points.

c|C: includes task rename system calls exec and prctl.

f|F: includes task creation and deletion system calls fork and exit.

s|S: includes context switch records created by the kernel scheduler.

Cryptographic counters only: The following characters filter data.

k|K: limits recording of PAI cryptographic counters to counters that changed through kernel calls.

u|U: limits recording of PAI cryptographic counters to counters that changed through user-space calls.

<loops>: the number of read operations during recording. The default is 1.

Recording examples

This example collects analytic counters on CPU 0. The program runs for 10 seconds (10 intervals of 1000 milliseconds).
```
# pai -n0 10
```
The following equivalent command uses the long option:
```
# pai --nnpa=0 10
```
This example collects cryptographic counters on CPUs 1, 5,6, and 7. The counters are limited to kernel calls, and the recording includes task rename system calls. The program runs for one interval of 2 seconds.
```
# pai -i 2000 -c1,5-7:ks
```
This example collects cryptographic counters for all CPUs. The counters are limited to user-space calls.
```
# pai -c:u
```

Reporting options and parameters

The −r option runs the pai command in reporting mode. This is the default.

-V or --verbose: for reporting, displays the file name and a hexadecimal offset for each sample header.
-H or --humantime: changes the timestamp format for sample entries from a hexadecimal number to a human-readable <seconds>.<nanoseconds> format. The timestamp specifies the elapsed time since the Linux® instance was booted.
-S or --summary: prints a summary of counter names and counter numbers with non-zero values. The summary is the last line in the command output. It shows the sum of the counter values of all processed files.
-r or --report <files>: generates a report from one or more files that hold recorded data.; <files> is a blank-separated list of file names. If no files are specified, the command processes all files in the working directory that follow the naming patterns paicrypto.<cpu_nr> and painnpa.<cpu_nr>, where <cpu_nr> specifies a CPU number as a three-digit decimal value.

Reporting example

This example creates a report of all files in the working directory that follow the naming pattern paicrypto.<cpu_nr>.
```
# pai
```
This equivalent command explicitly specifies the long option.
```
# pai --report
```

This command processes a file ~/pai_test/my_test_recording.

# pai ~/pai_test/my_test_recording
0x62a668f2fa 0 event 4096 sample pid 4956/4956 9:0xa7,73:0x8,74:0x18
0x6319c75653 0 event 4096 sample pid 4972/4972 32:0x1
0x6319e2ddee 0 event 4096 sample pid 4972/4972 32:0x1
...

This command processes a file for CPU 0. The recorded data includes extra information as collected with the c, f, and s options.

# pai -r paicrypto.000
0x4b32ceb84380 0 cs-out P next pid 7188/7188 pid 0/0
0x4b32ceb855b6 0 cs-in prev pid 0/0 SWITCH 0->7188 pid 7188/7188
0x4b32ceb88536 0 cs-out - next pid 0/0 pid 7188/7188
0x4b32ceb89158 0 cs-in prev pid 7188/7188 SWITCH 7188->0 pid 0/0
0x4b32ff00715f 0 cs-out P next pid 9111/9111 pid 0/0
0x4b32ff008e82 0 cs-in prev pid 0/0 SWITCH 0->9111 pid 9111/9111
0x4b32ff126254 0 exec sshd pid 9111/9111
0x4b32ff9997d9 0 event 4096 sample pid 9111/9111 9:0x2b,73:0x3
...

Output lines

All output lines begin with a timestamp in jiffies, followed by the CPU number. You can use the --humantime option for values in a human-readable <seconds>.<nanoseconds> format.

The remaining information depends on the type of line.

sample

The line contains sample data.

Unless at least one of the c, f, or s options for extra data are specified for recording, all output lines are sample lines.

The event specification.

event 4096: indicates that the sample data comprises the cryptographic counters. If the data collection was filtered, this specification reads event 4096:u for user-space calls only or event 4096:k for kernel calls only.
event 614: indicates that the sample data comprises the analytic counters.

The sample label is followed by the process identifier, a slash, and the thread identifier.

The line ends with a comma-separated list of counter number and value pairs. In each pair, the counter number and value are separated by a colon. Counter values are in hexadecimal notation and prefixed with 0x. Counters with zero-values are omitted.

Example: This line refers to cryptographic counter data on CPU 0 for a program with pid 9111.

0x4b32ff9997d9 0 event 4096 sample pid 9111/9111 9:0x2b,73:0x3

Interpret the counter data like this:

9:0x2b: The cryptographic counter with decimal number 9 was incremented to 0x2b (decimal 43).
73:0x3: The cryptographic counter with decimal number 73 was incremented to 0x3 (decimal 3).

cs-out and cs-in

The line describes context switching on the CPU. Such lines result from the s option for recording.

Example: This line shows a preemptive context switch out on CPU 0. The pid 0 kernel idle task is ended and a new task with pid 9111 is prepared.

0x4b32ff00715f 0 cs-out P next pid 9111/9111 pid 0/0

Example: This line shows a context switch in on CPU 0. The new task with pid 9111 is started to follow the already ended kernel idle task with pid 0.

0x4b32ff008e82 0 cs-in prev pid 0/0 SWITCH 0->9111 pid 9111/9111

exec and prctl

The line describes task rename system calls. Such lines result from the c option for recording.

Example: This line shows that task 9111 on CPU 0 is changed to run the sshd program code.

0x4b32ff126254 0 exec sshd pid 9111/9111

fork and exit

The line describes task creation and deletion system calls. Such lines result from the f option for recording.

Example: This line shows that task 20401 on CPU 0 is ended.

0xa9dbb549466 0 exit pid 20401 ppid 20290

Generic options

-v or --version: displays version information.
-h or --help: displays a short help text. To view the man page, enter man pai.