IBM Support

QMGTOOLS : Mutex trap

Troubleshooting


Problem

This trap helps collect information about a job in mutex wait state for a prolong period.

Diagnosing The Problem

A mutex (basically a lock on a resource) is created by a thread within a job. Mutexes are normal as it prevents several threads from a job to interfere with each other. However, there are times when a thread does not release the mutex, causing other threads within the job to wait.

Resolving The Problem

NOTE:  This trap tool is outdated and should not be used.  Job watcher is the recommended data to set up and run to debug a future MTXW issue.  Please ensure the system is current on PTFs.  This document can be used to set up a job watcher collection to collect a future MTXW issue:
However, at this time, job watcher does not contain the mutex name which is very important for debug as IBM does not debug third party mutex issues.  So if job watcher is going to be used, a DSPJOB to *PRINT for some of the jobs that are in MTXW will need to be collected at the time of the issue so we can get the mutex name.  
If there currently are jobs in MTXW, this document has data that can manually be collected at the time of the issue:

https://www.ibm.com/support/pages/node/666765
--------------------------------------------------------------------------------------------------------------------------------------------------
NOTE:  Before setting up the mutex trap, please make sure the following PTF for APAR MA49489 is applied to the system:

V7R4:  MF69518
V7R3:  MF69517
V7R2:  MF69516

The QMGTOOLS mutex trap helps collect information about a thread that might hold a mutex, preventing other threads from continuing. Data collected is as follows:

- Job logs of the job in mutex wait

- Job log of the mutex holder (if found)

- Call stacks of the job in mutex wait

- Call stack of the mutex holder (if found)

- Display all thread and status of the job in mutex wait (DSPJOB *ALL command)

- Display all thread and status of the mutex holder (if found) - DSPJOB *ALL command

- MUTEXINFO SST macro

- SERVICEDOCS SST macro

- WRKACTJOB output

- WRKSYSSTS output

 
1.
If the QMGTOOLS toolkit is not installed, use the following URL for information on how to download and install the QMGTOOLS toolkit on your IBM i server. 

http://www-01.ibm.com/support/docview.wss?uid=nas8N1011297
2. Add QMGTOOLS to your library list (ADDLIBLE QMGTOOLS) and type GO MG.

3. Go to the Work Management menu (in this case, option 15).
image 11501


4. Choose Mutex trap (in this case option 1)




5. Parameter as follows:

Function *START - start the trap
*STOP - stop the trap
*TEST - test to see whether the user credentials entered (for OS and SST sign on) works
Job status What status do we want to monitor, in this case put in MTXW (mutex wait)
Delay between checks Delay between checking status of active jobs
Minimum seconds in state When a job is found in that status, how long can it be in that status before the trap triggers to capture data
Signon user/pass Sign on credentials for OS sign on
SST user/pass SST sign on credentials
Data library Library to store the collected data
Additional options Internal use
Only value now is SMARTCHAIN to collect the SMARTCHAIN AA macro

6. Job names are as follows:

MUTEXTRP- the main job

MTXSTSCHK - the monitor job that monitors for jobs that are in MTXW status

There is also a 5250 session that is opened that runs commands such as AA macros in SST

So in total, there are 3 jobs, 2 batch and 1 interactive session.


The MTXSTSCHK uses the parameter, Delay Between Checks, to check all jobs for the condition specified in the Job Status parameter. When the MTXSTSCHK job detects a job is in the Job Status state, it keeps track of how long that job is in that state. If that state reaches the specified value in the Minimum Seconds In State value, it starts to dump the data. It also uses the 5250 session to run commands to dump data (for example, AA macros in SST).

Also, a message is sent to QSYSOPR that the trap triggered.



 

The data is stored as a file call MTXSTSxxxx where xxxx is a number in the data lib. For example, the data library is set to QTILIB as default. The following screen capture is an example. XXXX is a number that increments every time the trap is triggered.


[{"Type":"MASTER","Line of Business":{"code":"LOB68","label":"Power HW"},"Business Unit":{"code":"BU070","label":"IBM Infrastructure"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000000CHjAAM","label":"Job and Work Management"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"and future releases;7.2.0;7.3.0;7.4.0;7.5.0;7.6.0"}]

Document Information

Modified date:
25 September 2025

UID

nas8N1019977

Page Feedback