Resolving deadlocks in a CICS region

You can diagnose deadlocks between tasks wanting an exclusive lock on the same resource, such as a record in a non-RLS file, a recoverable transient data queue, or any resource represented by an EXEC CICS ENQUEUE.

About this task

Enqueue deadlocks between tasks occur when each of two transactions (say, A and B) needs an exclusive lock on a resource that the other holds already. Transaction A waits for transaction B to release the resource. However, if transaction B cannot release the resource because it, in turn, is enqueued on a resource held by transaction A, the two transactions are deadlocked. Further transactions may then queue, enqueued on the resources held by transactions A and B.

Use the following example to help you diagnose deadlocks. The scenario is that a user of task 32 complains that a terminal is locked and is unable to enter data.

Procedure

Use the command CEMT INQUIRE TASK to display the tasks in the system.

For example, a display similar to the following might appear:

INQUIRE TASK
  STATUS:  RESULTS - OVERTYPE TO MODIFY
   Tas(0000025) Tra(CEMT) Fac(T773) Run Ter Pri( 255 )
      Sta(TO) Use(CICSUSER) Uow(AA8E9505458D8C01)
   Tas(0000028) Tra(TDUP) Fac(T774) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E950545CAD227) Hty(ZCIOWAIT) Hva(DFHZARQ1)
   Tas(0000032) Tra(FUPD) Fac(T775) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E950545DAC004) Hty(ENQUEUE ) Hva(FCDSRECD)
   Tas(0000035) Tra(FUPD) Fac(T784) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E950545DBC357) Hty(ENQUEUE ) Hva(FCDSRECD)
   Tas(0000039) Tra(FUPD) Fac(T778) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E97FE9592F403) Hty(ENQUEUE ) Hva(FCDSRECD)
   Tas(0000042) Tra(FUP2) Fac(T783) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E97FE95DC1B9A) Hty(ENQUEUE ) Hva(FCDSRECD)

Task 32 is waiting on an enqueue Hty(ENQUEUE). You can also see that the task is waiting for a lock on a data set record Hva(FCDSRECD). At this stage, you cannot tell which (if any) task has control of this resource.

Use the command CEMT INQUIRE UOWENQ at the same terminal.

This command displays information about the owners of all enqueues held. More importantly, for deadlock diagnosis purposes, it displays information about the tasks waiting for the enqueues.

A screen similar to the following might be displayed:

INQUIRE UOWENQ
  STATUS:  RESULTS
   Uow(AA8E9505458D8C01) Tra(CEMT) Tas(0000025) Act Exe Own
   Uow(AA8E950545CAD227) Tra(TDUP) Tas(0000028) Act Tdq Own
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Own
   Uow(AA8E950545DBC357) Tra(FUPD) Tas(0000035) Act Dat Wai
   Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Wai
   Uow(AA8E9505458D8C01) Tra(TSUP) Tas(0000034) Ret Tsq Own
   Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Own
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Wai
   Uow(AA8E97FE95DC1B9A) Tra(FUPD) Tas(0000042) Act Dat Own

You can see all the enqueue owners and waiters on the same region on this display. Tasks waiting for an enqueue are displayed immediately after the task that owns the enqueue. Owners and waiters on other regions are not displayed.

If you system is busy, you can clarify the display by displaying only those resources that the task you are interested in owns and waits for.
This is called filtering. You add a filter to the end of the command as follows: CEMT INQUIRE UOWENQ TASK(32).
```
  INQUIRE UOWENQ TASK(32)
  STATUS:  RESULTS
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Own
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Wai
```
You can now see that task 32 owns one enqueue but is also waiting for another. This display shows one line of information per item, listing:
- UOW identifier
- Transaction identifier
- Task identifier
- Enqueue state (active, or retained)
- Enqueue type
- Relation (whether owner of the enqueue or waiter).
To see more information, press ENTER alongside the item that interests you.
If you press ENTER alongside the first entry of the output from CEMT INQUIRE UOWENQ TASK(32), a screen similar to the following might be displayed:
```
INQUIRE UOWENQ TASK(32)
RESULT
  Uowenq
  Uow(AA8E950545DAC004)
  Transid(FUPD)
  Taskid(0000032)
  State(Active)
  Type(Dataset)
  Relation(Owner)
  Resource(ACCT.CICS710.ACCTFILE)
  Qualifier(SMITH)
  Netuowid(..GBIBMIYA.IYA2T774.n......)
  Enqfails(00000000)
```
This shows you details of the enqueue that task 32 owns.
Expand the second entry to display the enqueue that task 32 is waiting for:
```
INQUIRE UOWENQ TASK(32)
RESULT
  Uowenq
  Uow(AA8E950545DAC004)
  Transid(FUPD)
  Taskid(0000032)
  State(Active)
  Type(Dataset)
  Relation(Waiter)
  Resource(INDX.CICS710.ACIXFILE)
  Qualifier(SMITH)
  Netuowid(..GBIBMIYA.IYA2T774.n......)
  Enqfails(00000000)
```
Expanding the one-line display is useful because RESOURCE and QUALIFIER fields are then revealed. These identify the physical resource that is related to the enqueue. You can see, from the first entry in this example, that task 32 owns the enqueue on record identifier “SMITH” in the ACCT.CICS710.ACCTFILE data set. You can also see, from the second expanded entry, that task 32 is waiting on an enqueue - for record identifier “SMITH” in the INDX.CICS710.ACIXFILE data set.
Investigate why task 32 is waiting on the enqueue detailed in the second expanded entry.
You need to find out which task owns this enqueue and why it is holding it for such a long time. You can do this by filtering the CEMT INQUIRE UOWENQ command with the RESOURCE and QUALIFIER options.
1. Enter CEMT INQUIRE UOWENQ RESOURCE(INDX.CICS710.ACIXFILE) QUALIFIER(SMITH).
  This shows the task that owns the enqueue that is being waited on.
```
INQUIRE UOWENQ RESOURCE(INDX.CICS710.ACIXFILE) QUALIFIER(SMITH)
STATUS:  RESULTS
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Own
 Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Wai
```
  This shows you that another task, task 39, owns the enqueue that task 32 is waiting on.
2. Find out why task 39 is holding this enqueue, using the CEMT command again as a filter for task 39. Enter CEMT INQUIRE UOWENQ TASK(39).
```
INQUIRE UOWENQ TASK(39)
STATUS:  RESULTS
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Wai
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Own
```
  This shows you that task 39 is also waiting for an enqueue.
3. Expand the entry that indicates the waiting state. You might see a display similar to the following:
```
INQUIRE UOWENQ TASK(39)
RESULT
  Uowenq
  Uow(AA8E97FE9592F403)
  Transid(FUP2)
  Taskid(0000039)
  State(Active)
  Type(Dataset)
  Relation(Waiter)
  Resource(ACCT.CICS710.ACCTFILE)
  Qualifier(SMITH)
  Netuowid(..GBIBMIYA.IYA2T776.p.nk4..)
  Enqfails(00000000)
```
  This shows you that task 39 is waiting for the enqueue on record “SMITH” in the ACCT.CICS710.ACCTFILE data set. This is the enqueue that task 32 owns.
You can now see that the deadlock is between tasks 32 and 39.
To confirm that your diagnosis is correct, filter by the RESOURCE and QUALIFIER of this enqueue.
This also shows that task 35 also waits on the enqueue owned by task 32.
```
INQUIRE UOWENQ RESOURCE(ACCT.CICS710.ACCTFILE) QUALIFIER(SMITH)
STATUS:  RESULTS
 Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Own
 Uow(AA8E950545DBC357) Tra(FUPD) Tas(0000035) Act Dat Wai
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Wai
```
You are now in a position of knowing which transaction(s) to cancel and investigate further.

Results

You can also use the EXEC CICS INQUIRE UOWENQ command or the EXEC CICS INQUIRE ENQ command in your applications. These return all the information that is available under CEMT INQUIRE UOWENQ. If you want to automate deadlock detection and resolution, these commands are of great benefit.

Note that CEMT INQUIRE UOWENQ can be used only for files accessed in non-RLS mode, because files accessed in RLS mode have their locks managed by VSAM, not by CICS. Deadlock and timeout detection for files accessed in RLS mode is also performed by VSAM.