Troubleshooting
Problem
How to investigate high temporary storage in IBM i
Symptom
Temporary storage growth
Environment
IBMi
Diagnosing The Problem
If the IBM i Support MustGather tools (QMGTOOLS) are installed on the system, see document 645455 - QMGTOOLS: Temp Storage Menu
Otherwise, use WRKSYSSTS to see how much storage is in the system ASP, how full it is, how much total auxiliary storage (all ASPs) the system has, and how much temporary storage is currently in use. Individual customer work-loads and data can cause temporary storage to vary widely from system to system. However, temporary storage greater than 7% of the system ASP, or greater than 5% of total auxiliary storage (for systems with a large user ASPs and/or IASPs) may indicate a problem. Note: 5% of total auxiliary storage is a general "rule of thumb", whereas 7% of the system ASP is the point where DB plan cache clean up kicks in.
Example scenario:
- Use of storage in the system ASP grew 20% (70% to 90%) in the last 24 hours.
If temporary storage only accounts for 4%, the problem is not with temporary storage, but permanent storage. For permanent storage issues, see technical document 644433 - Reducing System ASP Disk Space (DASD) / Storage Used
However, if temporary storage is 18%, then the growth is related to temporary storage and investigation should continue with the steps in this document.
Resolving The Problem
The first, and easiest, step to look for the cause of high temporary storage is to run WRKACTJOB SEQ(*TMPSTG) to see which jobs are currently using the most temporary storage. The temporary storage value for each job is shown in megabytes (MB) and sorted in descending order. If the top job in the list accounts for most of the temporary storage, no further searching should be needed. The next action is most likely to check with the job user to find out if it can be ended and ask for details about its purpose and function., Another option may be to see if holding the job will halt the temp storage growth, or maybe even drop it back to normal.
If a single job is not found using the majority of the temporary storage, you may see a large number of similarly named jobs that total up to nearly the amount of temporary storage in use by the system. In this case, is it caused by each job using more temporary storage than normal, or are there many more jobs of this type than normal?
Some temporary storage is not charged to individual jobs. Some is used by system tasks, and some is reserved for re-use. For cases where jobs do not account for most of the temporary storage, the next step is to query the "buckets" used to track temporary storage by the operating system. The following SQL statement will generate a list of temporary storage buckets in descending order by their amount of temporary storage:
select * from qsys2.systmpstg order by Bucket_Current_Size desc
If the largest bucket, or group of buckets is 8, 9, 11 or 14, please see technical document 706875 - MustGather: Temporary Storage growth in *DATABASE buckets 8, 9, 11, and 14
If bucket 2 *PROCESS is the largest bucket, that indicates many jobs have ended without cleaning up their temporary storage. Please see technical document 687835 - MustGather: Temporary Storage growth in *PROCESS bucket 2
For all other buckets, please continue with this document for data collection. The next step is to collect output from a temporary storage Advanced Analysis (AA) command in System Service Tools (SST). The AA command is RMTSTG (Resource Management Temporary SToraGe) and can be used as follows to get the bucket summary report:
If there is a particular bucket that accounts for most of the temporary storage or is significantly larger than all the other buckets, the next step is to collect the verbose bucket details as follows:
IBM software support will need to review all of the following data -
If a single job is not found using the majority of the temporary storage, you may see a large number of similarly named jobs that total up to nearly the amount of temporary storage in use by the system. In this case, is it caused by each job using more temporary storage than normal, or are there many more jobs of this type than normal?
Some temporary storage is not charged to individual jobs. Some is used by system tasks, and some is reserved for re-use. For cases where jobs do not account for most of the temporary storage, the next step is to query the "buckets" used to track temporary storage by the operating system. The following SQL statement will generate a list of temporary storage buckets in descending order by their amount of temporary storage:
select * from qsys2.systmpstg order by Bucket_Current_Size desc
- Note that you need to run the SQL command from an SQL interface like STRSQL.
If the largest bucket, or group of buckets is 8, 9, 11 or 14, please see technical document 706875 - MustGather: Temporary Storage growth in *DATABASE buckets 8, 9, 11, and 14
If bucket 2 *PROCESS is the largest bucket, that indicates many jobs have ended without cleaning up their temporary storage. Please see technical document 687835 - MustGather: Temporary Storage growth in *PROCESS bucket 2
For all other buckets, please continue with this document for data collection. The next step is to collect output from a temporary storage Advanced Analysis (AA) command in System Service Tools (SST). The AA command is RMTSTG (Resource Management Temporary SToraGe) and can be used as follows to get the bucket summary report:
- Run IBM i command STRSST and sign into SST with a profile that has authority to Display/Alter/Dump
- Select Option 1 - Start a service tool, and press the Enter key
- Select Option 4 - Display/Alter/Dump, and press the Enter key
- Option 2 - Dump to printer, and press the Enter key
- Select Option 2 - Licensed Internal Code (LIC) data, and press the Enter key
- Select Option 14 - Advanced analysis, and press the Enter key
- Under Option, type 1, and type RMTSTG on the blank Command line and press the Enter key
- On the Options screen, leave the Options field blank to collect summary and press the Enter key
- On the Dump Title screen, just press the Enter key
If there is a particular bucket that accounts for most of the temporary storage or is significantly larger than all the other buckets, the next step is to collect the verbose bucket details as follows:
- Run IBM i command STRSST and sign into SST with a profile that has authority to Display/Alter/Dump
- Select Option 1 - Start a service tool, and press the Enter key
- Select Option 4 - Display/Alter/Dump, and press the Enter key
- Option 2 - Dump to printer, and press the Enter key
- Select Option 2 - Licensed Internal Code (LIC) data, and press the Enter key
- Select Option 14 - Advanced analysis, and press the Enter key
- Under Option, type 1, and type RMTSTG on the blank Command line and press the Enter key
- On the Options screen, type -V <id> and press the Enter key (where <id> is the hex bucket ID found in the summary report above)
- On the Dump Title screen, change Through page to 99999999 and press the Enter key
IBM software support will need to review all of the following data -
- WRKSYSSTS OUTPUT(*PRINT)
- DSPPTF OUTPUT(*PRINT)
- AA command RMTSTG summary report
- AA command RMTSTG verbose report for largest bucket
- WRKSYSSTS OUTPUT(*PRINT)
- DSPPTF OUTPUT(*PRINT)
- AA command RMTSTG summary report
- DSPLOG OUTPUT(*PRTSECLVL) MSGID(CPC1217 CPF0907 CPF090A CPF1164 CPF3793 CPI099C CPI0999 CPI0953 CPI0954 CPI0955 CPI099D CPI112E CPI11AB CPI147B MCH2803)
** be sure to set the beginning date/time before the temporary storage returned to normal **
Send the data to the IBM i Global Support Center for review
See document 644037 - MustGather: Instructions for Sending Data to IBM i Support
See document 644037 - MustGather: Instructions for Sending Data to IBM i Support
[{"Type":"MASTER","Line of Business":{"code":"LOB68","label":"Power HW"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000001goBAAQ","label":"IBM i Db2-\u003ETemp Storage"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"and future releases;7.2.0"}]
Was this topic helpful?
Document Information
Modified date:
08 May 2025
UID
nas8N1021141