Collecting performance metrics

You can monitor virtual server machine code instructions.

Before you begin

  • You need a kernel that is built with the common code options CONFIG_TRACEPOINTS, CONFIG_HAVE_PERF_EVENTS, and CONFIG_PERF_EVENTS.
  • Confirm that the perf tool is installed, for example, by issuing the perf list command.
    # perf list
    ...
    kvm:kvm_s390_sie_enter                             [Tracepoint event]
    ...
    If the command returns a list of supported events, such as the tracepoint event kvm_s390_sie_enter, the tool is installed.

Procedure

Record and display performance metrics with the perf kvm stat command.
  • The record subcommand records performance metrics and stores them in the file perf.data.guest.
    • The perf tool records events until you terminate it by pressing Control and C (Ctrl+C).
    • The report subcommand displays the performance metrics that were previously recorded with the record subcommand.
    • Save a copy of perf.data.guest before you collect new statistics. A new record might overwrite this file.
  • The live subcommand displays the current statistics without saving them.

    The perf tool displays events until you terminate it by pressing Control and C (Ctrl+C).

Example

# ./perf kvm stat record -a
^C[ perf record: Woken up 7 times to write data ]
[ perf record: Captured and wrote 13.808 MB perf.data.guest (~603264 samples) ]

# ./perf kvm stat report


Analyze events for all VMs, all VCPUs:

                                 VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time 

                       Host interruption      14999    35.39%     0.39%      0.45us   1734.88us      0.82us ( +-  19.59% )
              DIAG (0x44) time slice end      13036    30.76%     0.57%      1.06us   1776.08us      1.39us ( +-   9.81% )
       DIAG (0x500) KVM virtio functions      13011    30.70%     1.90%      1.15us   2144.75us      4.65us ( +-   5.08% )
                              0xE5 TPROT        512     1.21%     0.01%      0.79us      2.18us      0.83us ( +-   0.42% )
                               0xB2 TSCH        406     0.96%     0.19%      7.35us    109.43us     14.95us ( +-   2.97% )
                              0xB2 SERVC        117     0.28%     0.15%     10.97us    339.00us     40.46us ( +-   9.17% )
                        External request        113     0.27%     0.01%      0.75us      2.58us      1.56us ( +-   1.55% )
                              0xB2 STSCH         57     0.13%     0.02%      7.30us     26.40us      9.47us ( +-   5.99% )
                              Wait state         40     0.09%    96.48%   3334.30us 464600.00us  76655.28us ( +-  32.97% )
                               0xB2 MSCH         14     0.03%     0.00%      7.22us      9.19us      7.74us ( +-   2.13% )
                               0xB2 SSCH         14     0.03%     0.01%      8.67us     35.41us     16.16us ( +-  16.38% )
                               0xB2 CHSC         10     0.02%     0.00%      7.51us     22.90us     11.06us ( +-  15.20% )
                             I/O request          8     0.02%     0.00%      1.37us      1.97us      1.55us ( +-   5.77% )
                               0xB2 STPX          8     0.02%     0.00%      1.04us      7.10us      1.98us ( +-  37.25% )
                               0xB2 STSI          7     0.02%     0.00%      1.65us     62.09us     22.26us ( +-  41.95% )
                              0xB2 STIDP          4     0.01%     0.00%      1.12us      3.62us      2.62us ( +-  21.07% )
                   SIGP set architecture          3     0.01%     0.00%      1.05us      2.68us      1.60us ( +-  33.74% )
                               0xB2 STAP          3     0.01%     0.00%      1.05us      7.61us      3.39us ( +-  62.25% )
                               0xB2 STFL          3     0.01%     0.00%      1.78us      3.88us      2.84us ( +-  21.31% )
    DIAG (0x204) logical-cpu utilization          2     0.00%     0.00%      4.58us     39.48us     22.03us ( +-  79.19% )
              DIAG (0x308) ipl functions          2     0.00%     0.01%     19.34us    329.25us    174.30us ( +-  88.90% )
     DIAG (0x9c) time slice end directed          1     0.00%     0.00%      1.09us      1.09us      1.09us ( +-   0.00% )
                                0xB2 SPX          1     0.00%     0.00%      4.58us      4.58us      4.58us ( +-   0.00% )
                               0xB2 SETR          1     0.00%     0.00%     56.97us     56.97us     56.97us ( +-   0.00% )
                               0xB2 SSKE          1     0.00%     0.25%   7957.94us   7957.94us   7957.94us ( +-   0.00% )
                              0xB2 STCRW          1     0.00%     0.00%     11.24us     11.24us     11.24us ( +-   0.00% )
    DIAG (0x258) page-reference services          1     0.00%     0.00%      4.87us      4.87us      4.87us ( +-   0.00% )
                               0xB9 ESSA          1     0.00%     0.00%      8.72us      8.72us      8.72us ( +-   0.00% )
                              0xEB LCTLG          1     0.00%     0.00%      9.27us      9.27us      9.27us ( +-   0.00% )

Total Samples:42377, Total events handled time:3178166.35us.

What to do next

For more information about the perf subcommand kvm stat, see the man page or issue the full subcommand with the --help option:

1  perf kvm stat
2.1 record
2.1 report
2.1 live
1  --help

With the collected statistics, you can watch the virtual server behavior and time consumption, and then analyze the recorded events to identify possible performance issues.

  • You can find a description of the general instructions in the z/Architecture® Principles of Operation, SA22-7832, for example:
    Mnemonic Instruction Opcode
    TPROT TEST PROTECTION E501
    TSCH TEST SUBCHANNEL B235
  • Signal-processor orders (SIGP) are also described in the z/Architecture Principles of Operation, SA22-7832.
  • DIAGNOSE (DIAG) instructions send virtual server requests to the hypervisor. Read the contents of /sys/kernel/debug/diag_stat to display a list of which diagnose instruction are called how frequently by your Linux® instance:
    # cat /sys/kernel/debug/diag_stat

    The output might include instructions that are called, but not supported by KVM on IBM Z®. Table 1 lists the supported instructions.

    Table 1. Supported Linux DIAGNOSE instructions
    Number Description Linux use
    0x010 Release pages CMM
    0x044 Voluntary time-slice end In the kernel for spinlock and udelay
    0x09c Voluntary time slice yield Spinlock
    0x258 Page-reference services In the kernel, for pfault
    0x288 Virtual server time bomb The watchdog device driver
    0x308 Re-ipl Re-ipl and dump code
    0x318 CP Name and Version Codes Identifies itself as Linux to the hypervisor
    0x500 Virtio functions Operate virtio-ccw devices