Triggering Performance Data Collection when you have an Intermittent Problem.

Troubleshooting

Problem

The system is experiencing intermittent performance issues, causing partitions or applications to slow down or produce performance-related errors. These issues can manifest as slow response times, application timeouts or crashes, various error messages in AIX or application logs, and abnormal disk activity or lack thereof. These problems can occur on both local and remote servers or environment.

Symptom

These may include the following. The issue can also be on remote servers.

Partition operating at a reduced speed
Applications exhibiting delays, timeouts, or crashes
Multiple error messages appearing in AIX or application logs
Unusual disk activity or a lack thereof

Cause

Insufficient system resources or improper configuration
These can also be a manifestation of issues on remote environments.
Anomalous behavior from applications
Hardware malfunctions

Environment

AIX

Diagnosing The Problem

To diagnose these intermittent performance issues, IBM's Perfpmr tool is utilized. Perfpmr collects performance data that can help identify the root cause of the problem. Two additional tools, perfstat_trigger and ping_trigger, allow commands like Perfpmr to be executed based on predefined thresholds. These triggers run the specified command or exit when the thresholds are met. Both perfstat_trigger and ping_trigger are included in the Perfpmr download package.

Install Perfpmr if it hasn't been installed already, especially if you plan to use it with perf_trigger.
Determine appropriate thresholds by collecting vmstat and iostat data over a longer period. Compare this data during normal operation with data collected during performance issues to identify significant changes.
Alternatively, consider using tools like nmon or topas to help determine optimal thresholds.
Use commands like vmstat to gather data to determine thresholds:
```
vmstat -IWt 1 20 > vmstat.IW.after &
```

Resolving The Problem

Run perfstat_trigger or ping_trigger with your chosen thresholds

Examples of perfstat_trigger usage:
```
./perfstat_trigger -p 10.0 -N 5 -x "perfpmr.sh" 1
```
This example triggers a Perfpmr.sh data collection once the physical CPU consumption exceeds 10.0 physc for five consecutive one-second intervals.
Help sections are available within the perfstat_trigger tool and ping_trigger for further guidance.

Related Information

Perfpmr including perfstat_trigger and ping_trigger

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB08","label":"Cognitive Systems"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG10","label":"AIX"},"ARM Category":[{"code":"a8m0z000000cw0pAAA","label":"Performance-\u003ECPU"},{"code":"a8m0z000000cw1SAAQ","label":"Performance-\u003EDisk"},{"code":"a8m0z000000cw0tAAA","label":"Performance-\u003EMemory"},{"code":"a8m0z000000cw0qAAA","label":"Performance-\u003ENetwork"}],"ARM Case Number":"","Platform":[{"code":"PF002","label":"AIX"}],"Version":"5.3.0;6.1.0;7.1.0;7.2.0;7.3.0"}]

Tips