IBM Support

MustGather: Reporting a Severity 1 Performance Issue

Troubleshooting


Problem

This document should be followed when reporting a high severity performance issue that is currently happening on your IBM i system.  The document covers the following:  performance questionnaire, collecting and uploading data to your case and optional steps for advanced data and recovery if you cannot wait for the data to be reviewed.  If you are reporting a severity one issue that happened in the past, follow the How to Report a Performance Issue that Happened in the Past.

Symptom

IBM i performance

Resolving The Problem

WARNING: The data cannot be reviewed without a clear explanation of the problem.
A problem description of "everything is slow", lacks important details; use Step 1 to provide the required details. 
 
Step 1:  Required Performance Questionnaire
  1. What needs to be urgently fixed? For example, our payroll jobs, named ABCCO, are running one hour longer and causing service agreements to be exceeded.  The problem started after we applied PTFs on Sept. 15.
  2. What business function is impacted by slow performance on the system? For example, when payroll jobs do not complete by midnight, a financial impact occurs as the business is fined if payroll does not complete on pay day.
  3. To what extent has the performance of the business function changed and how is the slowness observed or measured? For example, a process used to complete in few minutes and now takes five hours to complete, causing night batch jobs (named ABCCO*) to run during daytime hours, leading to complaints from QPADEV* interactive users.
  4. What are the IBM i jobs (users or subsystems) responsible for processing the slow function? If you are not sure, what other information can you give us to identify the jobs in question?* For example, DRDA application is slow so focus on the QRWTSRVR jobs with user ID XYZ.
  5. What is the timeline of this issue? For example, we first noticed the problem on Sept 15th at 23:00, but the problem does not happen every night.  On Sept 16th, the job completed normally. We need the problem resolved by Sept. 30th (before the next pay day).
*If the issue appears to be system wide, pick an example of one important job or workload that is measurably slower and provide us with the details as a focus point.
 
Step 2:  Collect performance data using these CL commands and start uploading to the case as soon as possible
 
a. CRTLIB LIB(TSXXXXXXX)
b. CRTSAVF FILE(TSXXXXXXX/TSXXXXXXX)
c. CHKPFRCOL <enter> and then place the cursor on the message text and press F1. Note the library and the first three digits of the collection object in the message.  Use the library name and the first three digits in step e.
d. ENDPFRCOL FRCCOLEND(*YES)
e. SAVOBJ OBJ(Q###*) LIB(QMPGDATA or QPFRDATA)
DEV(*SAVF) SAVF(TSXXXXXXX/TSXXXXXXX)
TGTRLS(*CURRENT) DTACPR(*MEDIUM)
f. STRPFRCOL
g. Upload the TSXXXXXXX/TSXXXXXXX savf to the case and notify your support representative that the data has been uploaded
 
The remaining steps should be done in parallel
If possible, mutli-thread your data collection efforts by asking a colleague to follow these additional steps, which can all be done in parallel.
 
Step 3:  Collect SERVICEDOCS
 
 
Step 4:  If SQL is involved, collect the following
 
b.  Collect three Active Dumps, spaced a few minutes apart: 
CALL QQQOOOCACH <enter> A:D:nnnnnn/User/JobName <enter> 
   nnnnnn/User/JobName is the name of the SQL job that is performing poorly. 
   A message will be returned with the location of the dump in the IFS, retain the name of the directory path. 
 
Step 5:  Start a 15 minute job watcher collection
 
b.   CRTLIB IBMJW
c.   SBMJOB CMD(STRJW DFN(Q5secSQL) COL(RUN1) LIB(IBMJW) ENDCOL((*NBRSEC 900) (*DASDMB 4096))) JOB(JOBWATCHER) JOBQ(QSYS/QSYSNOMAX)
d.   When the JOBWATCHER job ends, use CRTSAVF FILE(TSXXXXXXX/JWDATA)
e.   SAVLIB LIB(IBMJW) DEV(*SAVF) SAVF(TSXXXXXXX/JWDATA) TGTRLS(*CURRENT) DTACPR(*MEDIUM)
 
Step 6:  While waiting for the data to upload, collect a SYSSNAP
  1. QMGTOOLS/CMPVER then F6=Download and update
  2. If the QMGTOOLS library does not exist or the release is incorrect,  install the tool using these instructions.
  3. CALL QSYS/QCMD
  4. QMGTOOLS/SYSSNAP OUTPUT(*IFS) COLLECTDFT(Y) LICLOGS(Y) PALS(Y) DAYSPRV(3)
  5. Attach the .zip file to the case
 
If the system is hung and you must IPL, collect a main storage dump
NOTE:  Collecting a MSD will add time to the IPL, however, it will collect valuable data that may be used to determine root-cause. 

[{"Line of Business":{"code":"LOB68","label":"Power HW"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CI9AAM","label":"Performance-\u003EMustGather"}],"Platform":[{"code":"PF012","label":"IBM i"}],"Version":"All Versions","Type":"MASTER"}]

Historical Number

Product Synonym

Document Information

Modified date:
24 November 2025

UID

ibm16446407

Page Feedback