IBM Support

MustGather: High CPU

Troubleshooting


Problem

Use this document to collect PEX trace data for debugging a high CPU issue.

Resolving The Problem

PEX TPROF traces are used to help isolate which programs/procedures are using the most CPU time. The trace is obtained using Native IBM i commands.

Step 1. Verify that the required Performance Explorer (PEX) PTFs are applied: http://public.dhe.ibm.com/services/us/igsc/idoctor/html/ptfs.html
From an IBM i operating system command line, type the following (Note: An IBM Support Representative might change some of the parameters specified, based on a specific problem being reported):

Step 2. CRTLIB TSXXXXXXX (where XXXXXXX = your Case number)

Step 3. QSYS/ADDPEXDFN DFN(highcpu) TYPE(*TRACE) ADDTHDOPT(*ALL) LSTALLJOB(*NO) INTERVAL(1 *FIXED) MAXSTG(4000000) TRCFULL(*STOPTRC) TRCTYPE(*SLTEVT) JOB(*ALL) TASK(*ALL) SLTEVT(*YES) MCHINST(*NONE) BASEVT((*PMCO *NONE *FORMAT2))

 

Warning:

If the ADDPEXDFN command fails with CPFAF10 (definition already exists) then use RMVPEXDFN DFN(HIGHCPU) and re-add the definition in Step 3.  Alternatively, use a unique definition name in the DFN parameter on Step 3.

Step 4. STRPEX SSNID(RUN1) DFN(HIGHCPU)

Step 5. ENDPEX (no parameters) and press Enter one time.
Refresh and watch the event counter. When the event counter approaches 1 million (which typically takes 3-6 minutes), prepare to stop the trace, but do not stop the trace from the ENDPEX screen. Use Step 6 below to end the trace.

Step 6. Submit this command to batch using JOBQ qsys/qsysnomax so it will run immediately:
SBMJOB CMD(ENDPEX SSNID(RUN1) DTAOPT(*MGTCOL) MGTCOL(TSXXXXXXX/*SSNID)) JOB(ENDPEX) JOBQ(QSYS/QSYSNOMAX)

Step 7. Once the Batch job in Step 6 completes, save the data to a savf.
CRTSAVF QGPL/PEXSAVF
SAVLIB LIB(TSXXXXXXX) DEV(*SAVF) SAVF(QGPL/PEXSAVF) DTACPR(*MEDIUM)

Step 8. Send the data to IBM.
Instructions for Sending Data to IBM i Support
http://www-01.ibm.com/support/docview.wss?uid=nas8N1019224
The preferred data upload method is QMGTOOLS: FTP Data to IBM
http://www-01.ibm.com/support/docview.wss?uid=nas8N1010370
Step 9: Send in a system snapshot by using QMGTOOLS
http://www-01.ibm.com/support/docview.wss?uid=nas8N1010353

Step 10. (Optional)

If you would like to analyze the data, consider using IBM's iDoctor PEX Analyzer tool or PRTPEXRPT commands from the operating system command line. To learn more about iDoctor PEX Analyzer, visit https://www-912.ibm.com/i_dir/idoctor.nsf/pa.html.

To print reports from the i5 operating system command line, run the following:
By program: PRTPEXRPT MBR(run1) LIB(TSXXXXXXX) TYPE(*PROFILE)
By procedure: PRTPEXRPT MBR(run1) LIB(TSXXXXXXX) TYPE(*PROFILE) PROFILEOPT(*SAMPLECOUNT *PROCEDURE)   
By statement:
PRTPEXRPT MBR(run1) LIB(TSXXXXXXX) TYPE(*PROFILE) PROFILEOPT(*SAMPLECOUNT *STATEMENT)

[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CI9AAM","label":"Performance->MustGather"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.1;7.2;7.3;7.4","Line of Business":{"code":"LOB08","label":"Cognitive Systems"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SSC5L9","label":"IBM i 7.2"},"ARM Category":[{"code":"a8m0z0000000CI9AAM","label":"Performance->MustGather"}],"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.2","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SSC52E","label":"IBM i 7.1"},"ARM Category":[{"code":"a8m0z0000000CI9AAM","label":"Performance->MustGather"}],"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.1","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SSTS2D","label":"IBM i 7.3"},"ARM Category":[{"code":"a8m0z0000000CI9AAM","label":"Performance->MustGather"}],"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.3","Line of Business":{"code":"","label":""}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SS9QQS","label":"IBM i 7.4"},"ARM Category":[{"code":"a8m0z0000000CI9AAM","label":"Performance->MustGather"}],"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"7.4","Line of Business":{"code":"","label":""}}]

Historical Number

587269612

Document Information

Modified date:
25 June 2020

UID

nas8N1011762