ODBA - Diagnosing hung threads and UORs

Use the following steps to configure your system to gather data that IBM® Software Support can use to determine why IMS threads or UORs are hanging on your system.

Procedure

Turn on the following IMS table traces by using the following IMS commands:
```
/TRACE SET ON TABLE RRST OPTION LOG
/TRACE SET ON TABLE DISP
/TRACE SET ON TABLE SCHD
```
These traces are sent to the IMS OLDS, unless IMS external trace data sets are used. For a test environment, data sent to the OLDS files is appropriate. For production systems, use IMS external tracing. For more information about IMS external tracing, see Diagnostic setup recommendations for IMS.
If it is not already active with the following options, turn on the MVS SYSRRS CTRACE:
1. Place the following lines in the CTIRRSxx PARMLIB member:
```
TRACEOPTS
ON
BUFSIZE(1000M)
OPTIONS('EVENTS(URSERVS,LOGGING,CONTEXT,EXITS,STATECHG,RRSAPI,RE
START)')
```
2. Issue the following MVS Command:
```
TRACE CT,ON,COMP=SYSRRS,PARM=CTIRRSxx
```
3. Issue the following MVS Command to display the trace setting:
```
DISPLAY TRACE,COMP=SYSRRS
```
4. The display output for the SYSRRS portion appears as follows:
```
IEE843I 13.17.07 TRACE DISPLAY 479 SYSTEM STATUS INFORMATION
ST=(ON,1000K,09000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,500K)
COMPONENT MODE BUFFER HEAD SUBS
-------------------------------------------------------
SYSRRS ON 1000M
ASIDS *NONE*
JOBNAMES *NONE*
OPTIONS EVENTS(URSERVS,LOGGING,CONTEXT,EXITS,STATECHG,
                    RRSAPI,RESTART)
```
  Recommendation: The MVS CTRACE should always be active on any system using IMS with RRS. The MVS trace command should be added to the MVS COMMNDxx SYS1.PARMLIB member to ensure that it is always activated at IPL time. No performance impacts have been reported with the SYSRRS CTRACE using the above options.
  After this trace is activated on your system, future operational dumps of IMS must include the RRS address space in the list of jobs to be dumped along with the RRS data spaces (DSPNAME=('rrsjname'.*). Doing so ensures that the RRS control blocks and traces can be accessed for diagnostic purposes.

Turn GTF trace on with the TRACE=SYS,DSP,JOBNAMEP options.

When GTF Trace prompts for the JOBNAME, specify the JOBNAME for the stored procedure address space and IMS CTL Region. Set the GTF trace dataset to 1200 cylinders.

The GTF Messages should appear similar to the following:

START GTFxx.GTF
HASP100 GTFxx.GTF ON STCINRDR
HASP373 GTFxx.GTF STARTED
*01 AHL100A SPECIFY TRACE OPTIONS
R 01,TRACE=SYS,DSP,JOBNAMEP
IEE600I REPLY TO 01 IS;TRACE=SYS,DSP,JOBNAMEP
*02 ALH101A SPECIFY TRACE EVENT KEYWORDS - JOBNAME=
R 02,JOBNAME=(spasname,cltname)
IEE600I REPLY TO 02 IS;JOBNAME=(spasname,cltname)
*03 ALH102A CONTINUE TRACE DEFINITION OR REPLY END
R 03,END
END
IEE600I REPLY TO 03 IS;END
AHL103I TRACE OPTIONS SELECTED-SYS, DSP
AHL103I JOBNAME=(spasname,cltname)
*04 AHL125A RESPECIFY TRACE OPTIONS OR REPLY U
R 04,U
U
IEE600I REPLY TO 04 IS;U

Set the following SLIP trap to capture dumps for RRS related events:
```
SLIP SET,C=U0711,
JL=(imsctl,imsdli,rrsjname,????SPW2),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,LPA,ALLNUC),
ID=U711,DSPNAME=('rrsjname'.*),END
```
The variables in the preceding example are defined in the following list.

imsctl

The IMS control region jobname

imsdli

The IMS DLI region jobname

rrsjname

The RRS region jobname
Use IMS Display commands for hung threads:
```
– DIS ACTIVE
– DIS CCTL
– DIS UOR
```
If a hung ODBA thread or UOR occurs, take a dump of the system so that IBM Software Support can see the related control blocks and access the RRS traces. Use a single dump command as indicated below and not separate dumps for each job.
```
DUMP TITLE=(DUMP IMS/SPAS/RRS)
JOBNAME=(cltname,spasname,dlijname,rrsjname),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,LPA,ALLNUC,GRSQ),
DSPNAME=('rrsjname'.*),END
```
The variables in the preceding example are defined in the following list.

ctlname

The IMS CTL Region jobname

tspasname

The stored procedure address space jobname

dlijname

The IMS DLI region jobname

rrsjname

The RRS region jobname

Tune DUMP MAXSPACE and the SYS1.DUMP space allocation to ensure that your system has sufficient space for the dump.
It is important for IBM Software Support to receive all of the requested diagnostic information as a complete unit from a single occurrence with all data centered around the time of the thread or UOR hang. This information typically includes:
- Dump data sets obtained
- IMS OLDS or SLDS
- MVS SYSLOG
- LOGREC data set
- JES JOBLOG for cltname,spasname
After taking the dump, you can free the hung thread using the recommendations in Stopping Db2 for z/OS stored procedure threads.