Just for CICS - SHUTDOWN or NOT? CICS SHUTDOWN Hangs
ByronBaldwin 06000099ST Visits (4050)
As many in the world of CICS are aware there are a number of reasons why a CICS region can stall during shutdown. In this submission I would like to focus on a Shutdown hang that has a number of transactions which are left in DMWTQUEU waits.
There are a number of different ways to determine why the region is in a stalled state. When your CICS is in this state please obtain a console dump of the region. This can be obtained by issuing the following instructions from your MVS console:
dump comm=(CICS HANG)
Note that you will need to reply to this command as indicated:
Please ensure all trace components are set to standard level 1 except AP and EI which should be set to 1-2, and ensure your trace table size is at least 8M.
Once the dump is obtained, compare the CSA Time of Day ( CSATODP offset X'50' in the CSA ) to the dump time ( IP ST SYS ) and if they are different then your region is likely in a hung state. At this point check the task running on the QR TCB and see what type of wait it is currently experiencing. This can be accomplished by issuing command VERBX DFHPDXXX 'DS' ( XXX is the version of CICS you are running ). Once you get into this domain you will see a number of transactions, including the task running on the QR tcb, in the DMWTQUEU type wait state. DMWTQUEU wait state occurs when the quiescing of a CICS domain has not completed. Once you determine which task is running on the CICS QR TCB, make note of the transaction number and locate that task in the CICS KE domain ( VERBX DFHPDXXX 'KE' ). Once you locate the task then make note of the KE_KTCB address (next to last column in the output). At this point, go into BROWSE mode ( Option 1 on the IPCS Primary Option Menu screen or enter =1 from any command line in IPCS ) and in the POINTER address field fill in the KE_TCB address. At offset X'50' from this address will be the MVS TCB address please retain this address. At this point you should go to the COMMAND option of your IPCS Primary Option Menu and enter command SUMM REGS FORMAT JOBSUM. Now do a search for the MVS TCB address by issuing the following command:
F 'TCB: XXXXXXXX' ( where XXXXXXXX is the MVS TCB address you obtained earlier )
From here do a search for the word ACTIVE ( F ACTIVE ). At this point you will be at the beginning of the RBs for this TCB. Now you need to ensure this TCB is in a wait. To verify this you will locate the highest RB with a WLIC of 00020001 which has a 01 it on in the link area. Once located please obtain the OPSW address for that RB in question. At this point go back into BROWSE mode ( =1 ) and place the OPSW address into the POINTER address and locate the module name. This will let you know who is stopping your CICS region from shutting down. The instructions above will basically work for most hangs but some are a bit more involved and require a different set of investigative techniques. For more information on the documentation required for most waits, see technote "MustGather: Collect troubleshooting data for a wait or loop in CICS on z/OS".
There are a few APARs in CICS which were taken to address some shutdown hangs:
CICS TS V4.1
There also has been a recent vendor product CA SYSVIEW that has been associated with causing shutdown to stall. The fix for this product is TSC5128. See technote "Hang during CICS shutdown after upgrading CA SYSVIEW" for details.
As always I would love to hear your feedback concerning this blog. If you have topics you would like to hear about please feel free to advise as well and I will do my best to accommodate the request. As always DO IT JUST FOR CICS!