MustGather: Collecting a PEX activation group heap trace for a set of jobs to debug a heap leak

Resolving The Problem

• If the heap is already full, how can the heap be reset?  
This depends on how the code uses the heap.  Resetting the heap may be as simple as restarting a given job.  But,  if multiple jobs are failing at the same time, an IPL or special instructions may be needed to reset the heap.    
• What jobs should be traced?    A generic job group or a specific job needs to be identified.
• What type of heap is being used?  Activation Group (*ACTGRPHEAP), Resident (*RESHEAP), System (*SYSHEAP), Handle (*HDLHEAP), Local (*LCLHEAP), User (*USRHEAP)?  This document is specific to Activation Group Heap.
• Is a filter by heap address required?  This document is specific to not using a filter.
• How long should the trace be run?  Does the heap consumption happen over a period of hours or days?
• How will successful recreation of the problem be identified?  This document assumes that an error message will be signaled or that the ACTIVATIONHEAPS macro will be used to sample and monitor the size of the heap for growth.    

1. Verify that required Performance Explorer (PEX) PTFs are applied:  
   http://public.dhe.ibm.com/services/us/igsc/idoctor/html/ptfs.html
2. Verify that the QMGTOOLS library is installed and current:
   1. ADDLIBLE QMGTOOLS
   2. GO MG
   3. Option 12:  Display Build Date
      - If the build date is more than one month old and the release is correct, update the tool using option 13:  Check IBM for updated QMGTOOLS.
      - If the QMGTOOLS library  does not exist or the release is incorrect,  install the tool using these instructions.
3. Add the PEX definition for the generic job name or fully qualified job name:
   For the JOB specify either a generic job name, JOB((*ALL/*ALL/generic* *ALL)),  or a fully qualified job name, JOB((999999/job_user/job_name *ALL)).
   ADDPEXDFN DFN(HEAP)
             TYPE(*TRACE) 
             JOB((*ALL/*ALL/generic* *ALL)) or JOB((999999/job_user/job_name *ALL))
             TASK(*ALL)
             MAXSTG(4000000)  
             TRCTYPE(*SLTEVT)
             SLTEVT(*YES)
             MCHINST(*NONE)
            STGEVT((*ACTGRPHEAP)) 
4. Start the trace:
   • The trace must be active before the heap becomes full.
   • QMGTOOLS/STRPSC FUNCTION(*START)
                     LIBRARY(IBMPEX)
                     CLRLIB(N)
                     STRPEX(Y)
                     PEXDFN(HEAP)
                     PEXFILTER(*NONE)
                     SIZE(6)
                     PEX2KEEP(6)
                     JOBWATCHER(N)
                     CSI(N)
5. Recreate the problem:
   • Monitor for a period of time consistent with the time it take to experience the problem.
   • Wait for the heap to grow significantly before stopping the trace.
   • Monitor for errors or use the ACTIVATIONHEAPS macro to monitor for growth.​​​​​​
6. Stop the trace:
   QMGTOOLS/STRPSC FUNCTION(*DUMP)
7. Wait for the submitted job to complete.
8. Analyze the data or send the data to IBM
   • This document explains how to start the analysis using IBM iDoctor for IBM i:  Heap Analysis
   • Many heap issues are caused by application code logic and are not operating system defects.
   • Analysis of the trace may require billable services.
9. Send the data to IBM:
   • Save the IBMPEX data library to a save file 
   • Upload the save file to the case using one of these methods:
     Preferred Method:   https://www.ibm.com/support/pages/qmgtools-ftp-data-ibm
     Alternate Methods:  https://https://www.ibm.com/support/pages/mustgather-instructions-sending-data-ibm-i-support