Collect troubleshooting data: Using trace for CICS problem determination on z/OS

Resolving The Problem

Follow the instructions listed below to start, stop and format trace:

• CICS Auxiliary Trace
• CICS Internal Trace
• CPSM WLM Trace
• GTF Trace
• Language Environment Feature Trace
• Language Environment Traceback
• MTRACE
• MVS System Trace
• z/OS Communications Server (VTAM) Buffer Trace


• Collect troubleshooting data for CICS products - Read first

CICS Auxiliary Trace

Use CICS auxiliary trace to collect large amounts of trace data or to get a trace of CICS activity in which trace entries are not overwritten.

1. Start auxiliary trace:

• If your problem is occurring at CICS startup, set the tracing status at CICS initialization using the following system initialization table (SIT) parameters:
• SYSTR=ON
• AUXTR=ON
• AUXTRSW=NO
• STNTR=1 to turn level 1 tracing on for all CICS components
  STNTRxx=1 where xx is a specific component abbreviation
  
• Otherwise, use transaction CETR to control trace. Enter the following values on the initial CETR screen:
• Auxiliary Trace Status ===> Started
• Auxiliary Switch Status ==> Next or NO
• Master System Trace Flag ==> On

Enter PF4 to display a list of CICS components. In the column entitled Standard, enter 1 or 1-2 to the right of all components depending on the level of tracing required. Press Enter to apply your changes and then PF3 to exit from the components screen.

2. Recreate the failure or problem.

3. Stop trace by entering Stopped in the Auxiliary Trace Status.

4. Format the trace data using the trace utility print program (DFHTUxxx) where xxx is the version of CICS you used to create the trace; version 670 for CICS TS 4.2, 680 for CICS TS 5.1, 690 for CICS TS 5.2, 700 for CICS TS 5.3, and 710 for CICS TS 5.4

5. Interpret the trace entries.

Back to top

CICS Internal Trace

Use CICS internal trace for "first failure data capture". The CICS internal trace table resides in MVS virtual storage above the 16MB line (not in the EDSAs) providing virtual storage constraint relief (VSCR). On busy systems, large trace tables can fill up in several seconds. Therefore, a trace table size of 10240K is recommended with an alternative of 4096K.

1. Start internal trace:

• If your problem is occurring at CICS startup or you would like to leave internal tracing on at all times, set the tracing status at CICS initialization using the following SIT parameters:
• INTTR=ON
• SYSTR=ON
• STNTR=1 to turn level 1 tracing on for all CICS components
• STNTRxx=1 where xx is a specific component abbreviation
• TRTABSZ=10240
  
• Otherwise, use transaction CETR to control trace. Enter the following values on the initial CETR screen:
• Internal Trace Status ===> Started
• Internal Trace Table Size ===> 10240
• Master System Trace Flag ==> On

Enter PF4 to display a list of CICS components. In the column entitled Standard, enter 1 or 1-2 to the right of all components depending on the level of tracing required. Press Enter to apply your changes and then PF3 to exit from the components screen.

2. Recreate the failure or problem.

3. Stop trace by entering Stopped in the Internal Trace Status.

4. Format the internal trace table in one of two ways (where xxx is the version of CICS you used to create the trace; version 670 for CICS TS 4.2, 680 for CICS TS 5.1, 690 for CICS TS 5.2, 700 for CICS TS 5.3, and 710 for CICS TS 5.4:

• From a CICS system dump, using IPCS and the CICS print dump exit, DFHPDxxx. Specify TR=1 to select abbreviated trace and TR=2 to select full trace. Select parts of the CICS internal trace by specifying the TRS parameter. For example, you could enter the following when using IPCS:
1. Enter 0 to display the Defaults panel
2. Enter the dump dataset name:
   Source ==> DSNAME('dump.dataset.name')
3. Enter =6 to display the Command panel
4. Update DFHPDxxx depending on your CICS release
   Enter one of the following commands:
   ==> VERBX DFHPD710 'TR=1'
   ==> VERBX DFHPD710 'TR=2,TRS=<TASKID=nnnnn>'
5. Enter N in response to message BLS18160D May summary dump data be used by dump access?
• From a transaction dump, using the CICS dump utility program, DFHDUxxx. The default action is that the trace will be printed in abbreviated followed by full formats. DFHDUxxx will only format trace entries for the task that the transaction dump was taken for.

5. Interpret the CICS trace entries.

Back to top

CPSM WLM Trace

Use CICSPlex SM (CPSM) Workload Manager (WLM) trace from the routing region to determine why there is a routing failure.

1. Start auxiliary trace in the CMAS and the routing region:

• If your problem is occurring at CICS startup, set the tracing status at CICS initialization using the following system initialization table (SIT) parameters:
• INTTR=ON
• TRTABSZ=10240
• SYSTR=OFF in the CMAS and SYSTR=ON in the routing region.
• USERTR=ON (CPSM writes user trace entries)
• AUXTR=ON
• AUXTRSW=NO
  
• Otherwise, use transaction CETR to control trace. Enter the following values on the initial CETR screen:
• Internal Trace Status ===> Started
• Internal Trace Table Size ===> 4096
• Master System Trace Flag ==> Off in the CMAS and On in the routing region
• Master User Trace Flag ==> On
• Auxiliary Trace Status ===> Started
• Auxiliary Switch Status ==> NO

Press Enter to apply your changes and then PF3 to exit from the components screen.

2. Turn on WLM trace in the CMAS (for V3.2 and above):

1. From the WUI main menu, click CICSPlex SM operations views.
2. From the CICSPlex SM operations views menu, click MASs known to CICSplex.
3. From this view (EYUSTARTMAS.TABULAR), find the active MAS for the routing region to trace.
4. Click on the active CMAS name for this region to display the CMAS detail.
5. Scroll to the bottom and click on the Trace details link to display the CMAS detail (EYUSTARTCMAS.TRACE) view.
6. Scroll down to the 'Workload Management trace flags' and enter 30-31.
7. Click Apply changes.
   
   

3. Turn on WLM trace in the MAS (for V3.2 and above):

1. Go back to the MASs known to CICSplex view (EYUSTARTMAS.TABULAR).
2. From this view (EYUSTARTMAS.TABULAR), find the active MAS for the routing region to trace.
3. Click on the MAS name for this region to display the MAS detail (EYUSTARTMAS.DETAILED) view.
4. Scroll to the bottom and click on the Trace details link to display the MAS detail (EYUSTARTMAS.TRACE) view.
5. Scroll down to the 'Workload Management trace flags' and enter 18,19,23-27.
6. Click Apply changes.

When using CICS TS V3.1, you can not use the WUI to turn on WLM trace. You can use the COD0 TRACE FLAGS to set trace flags 30-31 in the CMAS and 18,19,23-27 in the MAS. See Using the interactive debugging transactions (COD0 and CODB) in the Knowledge Center for more details.

4. Recreate the failure or problem.

5. Take a dump:

/DUMP COMM=(your dump title) /R xx,JOBNAME=(EYUXvrm,cmasname,lmasname ),DSPNAME=('EYUXvrm'.*),CONT /R yy,SDATA=(ALLNUC,LPA,LSQA,PSA,RGN,SQA,TRT,CSA,GRSQ),END where EYUXvrm is the ESSS for your release, for example EYUX320, for CTS 320.

6. Use the instructions above to turn off the WLM flags in the CMAS and MAS by setting them to blank.

7. Stop auxiliary trace by entering Stopped in the Auxiliary Trace Status.

8. Save the unformatted auxiliary trace datasets for review by CICS Support.

9. If you wish to format the trace data from both the CMAS and the MAS routing region, use the CICSPlex SM trace utility program EYU9XZUT program.


Additional Information
Webcast replay: CPSM Workload Manager Level Tracing

Back to top

Language Environment Feature Trace

Use Language Environment (LE) Feature Trace (FT) to see trace entries for important LE events during the life of a CICS transaction. FT adds trace points inside LE at Event Handle and Set anchor points. It also provides register 13 and parameters before the LE call.

1. Start trace using the instructions for starting CICS auxiliary trace, CICS internal trace, or GTF trace depending on the destination you choose. Start FT trace by turning on level 2 tracing for the CICS application domain (AP=1-2):
• If your problem is occurring at CICS startup, set the tracing status at CICS initialization using the following SIT parameters:
  
• SYSTR=ON
• STNTRAP=(1,2) where AP is the application domain component abbreviation
  
• Otherwise, use transaction CETR to control trace. Enter the following values on the initial CETR screen:
  
• Master System Trace Flag ==> On
  
PF4 to display a list of CICS components. In the column entitled Standard, enter 1-2 to the right of the AP component. Press Enter to apply your changes and then PF3 to exit from the components screen


2. Recreate the failure or problem.
   
   
3. Stop FT by entering PF4 from the initial CETR screen. In the column entitled Standard, enter 1 or OFF to the right of the AP component. Press Enter to apply your changes and then PF3 to exit from the components screen.
   
   
4. Format FT using the instructions for formatting CICS auxiliary trace, CICS internal trace, or GTF trace depending on the destination you choose. Ensure that CEL4TFMT is in the STEPLIB prior to formatting the LE trace entries. Otherwise, you will receive message DFHTR5002 stating that the feature formatting program has failed.
   
   
5. Feature trace entries contain a domain identifier of FT. The format of these entries is slightly different from standard trace points. The module identifier contains the short name of the feature and a full module name. Data Areas contain the following:
• Event Handler Data Areas:
• Data Area 1 contains register 13 and Event Handler ID in the first two full words. The rest of Data Area 1 consists of 4-byte integers representing each parameter that is passed in the call. This is usually the value passed, but may be the address of the structure passed.
• Data Area 2 (Trace entry 1018): EICB
• Set Anchor Data Areas:
• Data Area 1 contains register 13 (first word), CAA address of this run unit (2nd word), and CEEEDB_CREATOR_RINI which equates to identify creator of an enclave (3rd word).

Additional Information
LE/CICS Trace (Feature Trace) Presentation

Back to top

Language Environment Traceback

Use LE traceback to see a sequential list of all active routines and the routine names, statement numbers, and offsets where the exception occurred.

1. Use LE option TERMTHDACT to determine level of information that LE produces when a failure occurs and the destination of the information.
   
   For the best performance, specify TERMTHDACT=(MSG) when traceback is not needed. If the traceback is required, specify TERMTHDACT=(TRACE,CICSDDS). The CICSDDS option directs the LE diagnostic output to the CICS dump dataset, rather than to the CESE transient data queue.
   
   Note: Java™ applications in CICS use MVS LE services rather than CICS LE services. As a result, CICSDDS is not an option and LE writes the traceback information to CEEDUMP.
   
   
2. Format LE storage areas and the AP domain for a task by entering IPCS command VERBX DFHPDxxx 'APS=<taskid=nnnnn>' where nnnnn is the task number:
• CICS routines call the LE verb exit and produce the same output as VERBX CEEERRIP ASID(00nn) CAA(xxxxxxxx) DSA(xxxxxxxx) ALL
• Eliminates the need to determine the DSA address (register 13) and the CAA address (register 12)
• Formats the LE Traceback, CAA, PCB, MEML, RCB, EDB, Runtime options, Heap control blocks, Stack storage control blocks (User and Lib), DSA, and Condition Management control blocks
  
3. For an explanation of the fields in traceback, see item 7 (Traceback) under sections of the Language Environment dump in the z/OS 2.3 Knowledge Center.

Back to top

GTF Trace

Use the generalized trace facility (GTF) to simultaneously write trace data to a single dataset from MVS and multiple subsystems like CICS, VTAM, and DB2.

1. Enter the following MVS commands to start GTF trace under MVS:

S GTF.identifier
R yy,TRACE=USRP,SYS,DSP,JOBNAMEP
R yy,USR=(F6C,xxx),JOBNAME=cicsjob,END
R yy,U
where yy is the number of the message

F6C is the user trace ID for CICS trace. SYS and DSP are optional but commonly requested. See GTF trace options for a full list of trace options.

2. Start GTF trace in CICS:

• If your problem is occurring at CICS startup, set the tracing status at CICS initialization using the following SIT parameters:
• SYSTR=ON
• GTFTR=ON
• STNTR=1 to turn level 1 tracing on for all CICS components
• STNTRxx=1 where xx is a specific component abbreviation
  
• Otherwise, use the transaction CETR to control trace.Enter the following values on the initial CETR screen:
• GTF Trace Status ===> Started
• Master System Trace Flag ==> On
  
Enter PF4 to display a list of CICS components. In the column entitled Standard, enter 1 or 1-2 to the right of all components depending on the level of tracing required. Press Enter to apply your changes and then PF3 to exit from the components screen.

3. Recreate the failure or problem.

4. Use CETR to stop trace by entering Stopped in the GTF Trace Status.

5. Enter MVS command Stop identifier to stop GTF trace.

6. Use IPCS to format GTF trace with the GTFTRACE subcommand:

• Interactively using TSO. For example:
1. Enter 0 to display the Defaults panel
2. Enter the trace data set name:
   Source ==> DSNAME('trace.dataset.name')
3. Enter =6 to display the Command panel
4. Enter one of the following commands:
   ==> GTFTRACE USR(ALL)
   ==> GTFTRACE JOBNAME(cicsjob) USR(F6C)
   ==> GTFTRACE USR(F6C) sys dsp
• As a batch program

7. Read GTF trace


Additional Information
z/OS MVS Diagnosis: Tools and Service Aids

Back to top

MTRACE

Use Master Trace (MTRACE) to view all recent system messages, including CICS messages that are written to the MVS console.

• MTRACE is activated automatically at MVS system initialization.
  
  
• When a system dump is taken, the master address space which MTRACE is part of is always dumped. If MTRACE is not included, ensure that the SDUMP option TRT is in force at the time the dump is written. Write MTRACE to a system dump dataset by entering one of the following:
• CICS transaction CEMT PER SNAP
• MVS command DUMP COMM=(dumpname) followed by a reply of R yy,JOBNAME=( cicsjob) where yy is the number of the message
  
  
• Format MTRACE data from a system dump by entering the IPCS command VERBX MTRACE or using the IPCS Trace Processing selection panel.
• See z/OS MVS IPCS Commands for the MTRACE subcommand
• See z/OS MVS IPCS User's Guide for the panel

• Read Master Trace data.

Back to top

MVS System Trace

Use the MVS System Trace to view an ongoing record of hardware and software events occurring during system initialization and operation.

• The system activates system tracing at initialization and the tracing runs continuously, unless your installation has changed the IBM-supplied system tracing.
  
  
• Enter MVS command TRACE STATUS to check the size of the internal trace table. Message IEE839I will display. For example, if ST=(ON,0256K,00512K) displays this indicates that trace is on and is the default size of 256K. You can change the trace table size or add trace branch instructions by using the TRACE command to customize System Trace.

• When a system dump is taken, the master address space which MVS System Trace is part of is always dumped. If MVS System trace is not included, ensure that the SDUMP option TRT is in force at the time the dump is written. Write MVS System Trace to a system dump dataset by entering one of the following:
• CICS transaction CEMT PER SNAP
• MVS command DUMP COMM=(dumpname) followed by a reply of R yy,JOBNAME=( cicsjob) where yy is the number of the message
  
  
• Use the IPCS subcommand SYSTRACE to view the trace in a system dump. For example:
1. Enter 0 to display the Defaults panel
2. Enter the dump dataset name:
   Source ==> DSNAME('dump.dataset.name')
3. Enter =6 to display the Command panel
4. Enter one of the following commands:
   ==> SYSTRACE TIME(LOCAL) to display the local time with each trace entry for the ASID the dump was taken for.
   ==> SYSTRACE ALL to display trace entries for all ASIDs in system.
   ==> SYSTRACE ASID('nnnn') to display trace entries for a specific four digit ASID, nnnn. This is helpful when trace entries for the CICS ASID do not display.

• Read System Trace output.

Back to top

z/OS Communications Server (VTAM) Buffer Trace

Use z/OS Communications Server (VTAM) buffer trace to investigate terminal incorrect output errors in a VTAM environment.

1. Enter the following MVS commands to start GTF trace under MVS :
   S GTF.identifier
   R yy,TRACE=USRP,JOBNAMEP
   R yy,USR=(F6C,FF1,FE1,FEF),JOBNAME=cicsjob,END
   R yy,U
   where yy is the number of the message
   
   User trace ID F6C is for CICS trace, FF1 is for VTAM buffer trace (user or API component), FE1 is for VTAM internal trace, and FEF is for VTAM buffer trace (TSC component). See GTF trace options for a full list of GTF trace options.
   
   
2. Enter the following MVS command to turn on VTAM buffer trace and VTAM internal trace:
   F NET,TRACE,TYPE=BUF,ID=applid,AMT=FULL,E where applid is the CICS applid
   F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG,SSCP)
   
   
3. Recreate the failure or problem.
   
   
4. Enter the following MVS commands to turn off VTAM trace:
   F NET,NOTRACE,TYPE=VTAM,OPT=END
   F NET,NOTRACE,TYPE=BUF,ID=applid
   
   
5. Enter the MVS command Stop identifier to stop GTF trace
   
   
6. Use IPCS with the GTFTRACE option to format the trace. For example, enter the following from IPCS:
1. Enter 0 to display the Defaults panel
2. Enter the trace data set name:
   Source ==> DSNAME('trace.dataset.name')
3. Enter =6 to display the Command panel
4. Enter one of the following commands:
   ==> GTFTRACE USR(ALL) to format all trace
   ==> GTFTRACE JOBNAME(cicsjob) USR(F6C)
   ==> GTFTRACE USR(F6C,FF1,FE1,FEF)
   ==> GTFTRACE USR(TPIO) to format VTAM TSC trace
   
   
7. See VTAM trace record formats for help interpreting the trace.
   

Additional Information
CICS VTAM exit tracing
Using IPCS to format GTF trace

Back to top