IBM Support

Collect troubleshooting data for a wait or loop in CICS TS for z/OS

Troubleshooting


Problem

You are having a wait or loop problem in CICS Transaction Server for z/OS (CICS TS). You would like to know what documentation you must collect (MustGather) so that the CICS Support team can diagnose your problem. If you gather this documentation before contacting support it will expedite the troubleshooting process, and save you time.

Cause

A wait or hang can occur for many different reasons. For example, the region could be at the MXT limit in which case no new tasks can be attached or the region could be waiting for an ECB to be posted. A loop is the repeated execution of some code.

Resolving The Problem

For the most up-to-date documentation, refer to Collecting CICS troubleshooting data (MustGather) for IBM Support in the CICS TS documentation. You will find the General information you should gather for every type of problem followed by the Component-specific information which includes Troubleshooting data for wait or loop problems.

The following is the original MustGather information:







Collecting general information for all problems

If you have already contacted IBM Support, you can continue on to the component-specific MustGather information. Otherwise, go to Collect troubleshooting data for CICS products - Read first.







Collecting data for (component) problems

If you suspect that the problem is due to another IBM product or subsystem like MVS system logger, DB2, VSAM, or RLS see the MustGather document for that particular component. Otherwise, gather the following documentation for the wait or loop before calling IBM support:

Required (MustGather) data:
  • CICS internal trace that is included in the MVS system dump when tracing is active. The trace should be at least 10240K and when possible level 1 tracing should be on for all CICS components.

  • An MVS system dump of the CICS region taken as soon as you notice the hang or wait. Use the following MVS command followed by the reply to capture the dump:

    DUMP COMM=(dumpname)
     R yy,JOBNAME=(
    cicsjob),
     SDATA=(ALLNUC,PSA,SQA,CSA,LPA,TRT,LSQA,RGN)


    where dumpname is the name you want to give the dump, yy is the reply id, and cicsjob is the job name of your CICS region.

  • CPU utilization for the CICS region. Enter S.DA when using SDSF under TSO, to display the active CICS job. Enter PF11 to display the CPU% which is the percent of CPU time used on behalf of this address space during the most recent interval measured.


Optional (MustGather) data:
  • CICS message log containing the last message created by CICS and the time the message was created.

  • If you are able to recreate the problem, consider using CICS auxiliary trace or GTF Trace in combination with the MVS system dump. The dump is unlikely to tell you anything about system activity in the period leading up to the wait or loop. This is because the trace table will probably wrap before you have had a chance to respond.





Troubleshooting hints and tips



Diagnostic tips:
  • Review the logs and dumps generated at the point of failure.
  • Search the CICS support site for known problems using symptoms like the message number and error codes.
  • If you find a fixing PTF, see Ordering CICS products and maintenance for the options that are available to order CICS maintenance.
  • Gather the documentation and work with the CICS support team to resolve your problem.

Learn more about (component):


Exchanging data with IBM Support

  1. See Exchanging information with IBM Technical Support for FTP and email instructions using the IBM Enhanced Customer Data Repository (ECuRep).
  2. Go to the Service Request page to open or update a problem. If you have a SoftwareXcel enterprise edition for zSeries contract, you can also ask a non-defect installation and usage questions.

    Note: Always update your Service Request (problem record) to indicate that data has been sent.

If you need to speak to an IBM technical support representative call your country representative. If you need to speak to an IBM technical support representative in the US, call 1-800-IBM-SERV.

Internal Use Only

before changes to V16 and new template 6/8/11 tmh


before changes to V16 and new template 6/8/11 tmh
This document contains the following sections:




Collecting documentation required by the support team
Submitting documentation to the support team
Troubleshooting hints and tips
Learning more about this problem or component




Back to Read First


Collecting documentation required by the support team

If you suspect that the problem is due to another IBM product or subsystem like MVS system logger, DB2, VSAM, or RLS see the MustGather document for that particular component. Otherwise, gather the following documentation for the wait or loop before calling IBM support:


Required doc:
  • CICS internal trace that is included in the MVS system dump when tracing is active. The trace should be at least 10240K and when possible level 1 tracing should be on for all CICS components.

  • An MVS system dump of the CICS region taken as soon as you notice the hang or wait. Use the following MVS command followed by the reply to capture the dump:

    DUMP COMM=(dumpname)
     R yy,JOBNAME=(
    cicsjob),
     SDATA=(ALLNUC,PSA,SQA,CSA,LPA,TRT,LSQA,RGN)


    where dumpname is the name you want to give the dump, yy is the reply id, and cicsjob is the job name of your CICS region.

  • CPU utilization for the CICS region. Enter S.DA when using SDSF under TSO, to display the active CICS job. Enter PF11 to display the CPU% which is the percent of CPU time used on behalf of this address space during the most recent interval measured.


Optional doc:
  • CICS message log containing the last message created by CICS and the time the message was created.

  • If you are able to recreate the problem, consider using CICS auxiliary trace or GTF Trace in combination with the MVS system dump. The dump is unlikely to tell you anything about system activity in the period leading up to the wait or loop. This is because the trace table will probably wrap before you have had a chance to respond.


Submitting documentation to the support team

See Exchanging information with IBM Technical Support for FTP and email instructions using the IBM Enhanced Customer Data Repository (ECuRep).

Go to ServiceLink or IBMLink to open an Electronic Technical Response (ETR). If you need instructions, see open new PMRs or convert existing PMRs to electronic PMRs using IBMLink ETR.

If you need to speak to an IBM technical support representative call your country representative. If you need to speak to an IBM technical support representative in the US call 1-800-IBM-SERV.

Always update the PMR to indicate that data has been sent.

Troubleshooting hints and tips

Diagnostic tips:
  • Review the logs and dumps generated at the point of failure.
  • Search the CICS support site for known problems using symptoms like the message number and error codes.
  • If you find a fixing PTF, see Ordering CICS products and maintenance for the options that are available to order CICS maintenance.
  • Gather the documentation and work with the CICS Level2 support team to resolve your problem.


Learning more about this problem or component

This section provides you with information that can help you learn more about this topic:



Hotspot before updating for CICS TS 4.1:
Distinguishing between waits, loops, and poor performance


Old hotspot before 3.2 added - Distinguishing between waits, loops, and poor performance

878 MustGather Database 'DCF Technotes', View 'Products', Document 'MustGather: Documentation for an abend 878 or 80A in CICS on z/OS ' (contains MVS system trace stuff which I think we will want to include).

**
In SDSF, CPU-Time is the accumulated CPU time (TCB plus SRB) consumed on behalf of the address space, for the current job step, in seconds. We might want the customer to look at this.

I didn't include the following in the MustGather because they are included automatically but they will be used in diagnostics.
My notes:
Don't forget to look at LINKAGE STACKS
and pay attention to the Primary and Secondary ASIDs under the XSB
SASID.... 000E PASID.... 000E

Look at the MVS log to see when the last CICS message was generated.

Hang/WAIT Help from TSCUST (by John Gunter):
When one task is in a wait



(A) IPCS
(B) Command
(C) VERBX DFHPD410 'KE=1'(KERNAL Summary)
(D) Get the TCA address of the task in a wait.
(E) Go to the TCA +x'140' which is the RSA. (The regs start at x'C'
and are in the order of 14, 15, 0, 1, 2...)
(F) Go to R1 which is the 'parm list'.
(G) The 1st word will be EIBFN, the request type.
(H) The 3rd word will be the 'pointer' to the list of ECBs.
(I) Within the ECB you will often see the name of who issued the
wait. (Like DSN2EXT2).
(J) Go to R14 from the RSA to see who issued the wait.
Back up to see the module or program name.

Help for waits in CICS V2


(1) When all tasks are in a wait and the CSA CICS TIME or STATUS info
shows that the CICS TIME is current and they DO HAVE 'IPCS';
(A) IPCS
(B) Command
(C) VERBX CICSDATA 'KCP' (Formats the task summary & dispatcher chains).
*NOTE* You may run the chains yourself by doing the following;
(D) Go to the CSA +x'B4' (highest active DCA)
(E) Go to the DCA +x'10' (next DCA address)
(F) Go to the DCA +x'14' (UTCA address)
(G) Go to the DCA +x'18' (DCI indicator)
(H) Go to the DCA +x'1C' (ECB address)
(I) Go to the UTCA +x'20' (REG14 address)
***Usually the 3rd, 4th, or 5th task is hung and all the others are
waiting behind it for some reason.
(a) Determine why it is hung.
(b) Go to the REG14 address and back up to get the module name.
(c) Go into that code to determine why the task was put into a wait.

(2) When all tasks are in a wait and the CSA CICS TIME is current and
they DO NOT HAVE 'IPCS';
(A) REG13 points to the CSA.
(B) Go to the CSA +x'B4' (highest active DCA)
(C) Go to the DCA +x'10' (next DCA address)
(D) Go to the DCA +x'14' (UTCA address)
(E) Go to the DCA +x'18' (DCI indicator)
(F) Go to the DCA +x'1C' (ECB address)
(G) Go to the UTCA +x'20' (REG14 address)
***Usually the 3rd, 4th, or 5th task is hung and all the others are
waiting behind it for some reason.
(a) Determine why it is hung.
(b) Go the REG14 address and back up to get the module name.
(c) Go into that code to determine why the task was put into a wait.

(3) When CICS is in a wait but the CSA CICS TIME or STATUS info shows
that the CICS TIME is 'NOT' current and they DO HAVE 'IPCS';
(A) IPCS
(B) Command
(C) SUMM FORMAT or SUMM REGS
(D) F 'TCB: ADDRESS' (Where ADDRESS is the TCB address)
(E) CMP = completion code (Should be zero normally)
(F) Go to the next RB under the TCB (Pass by STCB)
(The address of this RB is usually in RBP at the top of the TCB)
(NOTE - You may also do a F ACTIVE to get the next active RB).
(G) WLIC (Type of wait)
(H) OPSW (Should point to who issued the wait).

(4) When CICS is in a wait but the CSA CICS TIME or STATUS info shows
that the CICS TIME is 'NOT' current and they DO NOT HAVE 'IPCS';
To be continued........

(5) DLI ENQUEUE or WAIT for CICS R212 and DLI R220.
(A) Transaction in an IWAIT
(B) UTCA +x'0' ---> STCA
(C) STCA +x'55' ---> ISB
(D) ISB +x'1' ---> PST
(E) PST +x'2C0' ---> Que Element
(F) Q Ele +x'C' ---> Req Element
(G) R Ele +x'8' ---> Owners Req chain
(H) Run this chain (+x'0' is the forward pointer) until it points
back to a Que Element.
(I) Q Ele +x'0' ---> PST
(J) PST +x'50C' ---> 1st RSA
(K) 1st RSA +x'4' ---> Next RSA
(L) Next RSA +x'4' ---> R12
To be continued........

(6) CICS LU62 hang.
To be continued........

(7) CICS LU62 to another CICS and both LU62 conections are hung.
To be continued........

(8) CICS MRO hang.
To be continued........


Help for waits in CICS V4


(1) When one task is in a wait;
(A) IPCS
(B) Command
(C) VERBX CICS410 'KE=1'(KERNAL Summary)
(D) Get the KE_NUM of the task in a wait.
(E) Go to the KE STACK entry for this KE_NUM
(F) Go to the LINK REG address in the KE STACK entry before DFHDSSR.
(G) It should point within the module that issued the wait.
*NOTE* VERBX CICS410 'DS=1'(DISPATCHER Summary)
This will give the type of wait that your task is in under
the 'ST' column. You can look this up in the PD Guide.


[{"Product":{"code":"SSGMGV","label":"CICS Transaction Server"},"Business Unit":{"code":"BU011","label":"Systems - zSystems"},"Component":"Dispatcher Domain","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"4.1;4.2;5.1;5.2;5.3;5.4","Edition":""}]

Product Alias/Synonym

CICS/TS CICS TS CICS Transaction Server

Document Information

Modified date:
21 June 2018

UID

swg21214981