PH61814: IBM MQ Z/OS QUEUE MANAGER ISSUED S026 ABENDS AND TERMINATED ABNORMALLY WITH REASON=00C510AB

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• One queue manager in a queue sharing group terminated abnormally
  with REASON=00C510AB with normal termination of another queue
  manager in the same queue sharing group. Before the abnormal
  termination it issued S026 abends. The root cause looks like a
  deadlock between tasks attempting to access a locked list header
  and structure failure/lossconn processing. As one of the
  deadlocked tasks holds a latch of the ethr chain, no changes can
  hang while waiting for the deadlock to be resolved - this can
  include impactPeer Level Recovery for any structures (not just
  the one involved in the deadlock), responding to structure
  events (resulting in the S026 abends and eventual queue manager
  termination), checkpoint processing and allocation/deallocation
  processing. The MSTR joblog prints as following:
  
  CSQ1:
  ----
  IXL016I CONNECTOR CSQCONNECTOR1 TO STRUCTURE QSG1CSQ_ADMIN
   DISCONNECTING:
   JOB CSQ1MSTR ASID 01CE REQUESTED DISCONNECT REASON=NORMAL.
  CSQE006I +CSQ1 CSQECLOS Structure CSQ_ADMIN connection
   name CSQCONNECTOR1 disconnected
  CSQ5019I +CSQ1 CSQ5DISC Disconnected from DB2 DB21
  CSQ9022I +CSQ1 CSQYASCP 'STOP QMGR' NORMAL COMPLETION
  
  CSQ2:
  ----
  18.24.12:
  CSQE007I +CSQ2 CSQESTE EEPLDISCFAILCONNECTION event
   received for structure MQAPPL1 connection name CSQCONNECTOR1
  CSQE008I +CSQ2 CSQESTE Recovery event from CSQ1
   received for structure MQAPPL1
  CSQE011I +CSQ2 CSQESTE Recovery phase 1 started for
   structure MQAPPL1 connection name CSQCONNECTOR1
  
  18.26.20:
  *IXL041E CONNECTOR NAME: CSQCONNECTOR2, JOBNAME: CSQ2MSTR,
   ASID: 01D9
   HAS NOT RESPONDED TO THE DISCONNECTED/FAILED CONNECTION EVENT
   FOR SUBJECT CONNECTION: CSQCONNECTOR1.
   DISCONNECT/FAILURE PROCESSING FOR STRUCTURE CSQ2APPL1
   CANNOT CONTINUE.
   MONITORING FOR RESPONSE STARTED: 05/29/2024 18:24:12.864177.
   DIAG: 0000 0000 00000000
  *IXL040E CONNECTOR NAME: CSQCONNECTOR2, JOBNAME: CSQ2MSTR,
   ASID: 01D9
   HAS NOT RESPONDED TO THE USER SYNC POINT EVENT.
   USER SYNC POINT PROCESSING 581 FOR STRUCTURE CSQ2APPL1
   CANNOT CONTINUE.
   MONITORING FOR RESPONSE STARTED: 05/29/2024 18:24:12.890347.
   DIAG: 0000 0000 00000000
  *IXL049E HANG RESOLUTION ACTION FOR CONNECTOR
   NAME: CSQCONNECTOR2
   TO STRUCTURE CSQ2APPL1, JOBNAME: CSQ2MSTR, ASID: 01D9:
   SYSTEM WILL TAKE ACTION AT 05/29/2024 18:36:49.926279.
  
  18.36.49:
  IXL050I CONNECTOR NAME: CSQCONNECTOR2 TO STRUCTURE CSQ2APPL1,
   JOBNAME: CSQ2MSTR, ASID: 01D9
   HAS NOT PROVIDED A REQUIRED RESPONSE AFTER 722 SECONDS.
   TERMINATING CONNECTOR TASK TO RELIEVE THE HANG.
  IXL016I CONNECTOR CSQCONNECTOR2 TO STRUCTURE QSG1CSQ_ADMIN
   TERMINATING
   JOB CSQ2MSTR ASID 01D9 REQUESTED DISCONNECT REASON=FAILURE.
  IXL016I CONNECTOR CSQCONNECTOR2 TO STRUCTURE CSQ2APPL1
   TERMINATING
   JOB CSQ2MSTR ASID 01D9 CONNECTOR TASK TERMINATED.
  CSQE006I +CSQ2 CSQECLOS Structure CSQ_ADMIN connection
   name CSQCONNECTOR2 disconnected
  IEA794I SVC DUMP HAS CAPTURED:
   DUMPID=007 REQUESTED BY JOB (CSQ2MSTR)
   DUMP TITLE=CSQ2,ABN=026-08110102,U=SYSOPR
     ,M=CSQVSRRX,PSW=070C0000814F67CC,ASID=01D9
  IEA794I SVC DUMP HAS CAPTURED:
   DUMPID=008 REQUESTED BY JOB (CSQ2MSTR)
   DUMP TITLE=CSQ2, QUEUE MANAGER TERMINATION REQUESTED,
   REASON=00C510AB
  *CSQV086E +CSQ2 QUEUE MANAGER ABNORMAL TERMINATION
   REASON=00C510AB
  

Local fix

• N/A
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 2 Modification 0,                    *
  *                 Release 3 Modification 0 and                 *
  *                 Release 4 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: Abend S026-08110101 and S026-08110102   *
  *                      followed by abnormal queue manager      *
  *                      termination 6C6 with REASON=00C510AB    *
  *                      occurs several minutes after an         *
  *                      application structure failed and a peer *
  *                      queue manager terminated.               *
  ****************************************************************
  While processing a structure failed event, CSQESTFA waits for
  any in progress commit operations to be forced to the log. One
  such operation was suspended because of a list being locked on
  the terminating queue manager. The list lock should have been
  released by Peer Level Recovery (PLR) for the terminating queue
  manager's connection to the structure, however this could not
  take place due to a latch held by CSQESTFA, resulting in a
  deadlock situation. The deadlock was detected and reported by
  XCF messages IXL041E, IXL040E and IXL049E and abend
  S026-08110101. Several minutes later XCF issued IXL050I and
  abend S026-08110102 causing the queue manager to terminate
  abnormally.
  

Problem conclusion

• CSQESTFA has been updated to prevent this deadlock situation.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UO02258 UO02259 UO03122

Modules/Macros

• CSQESTFA
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R200 PSY UO03122

     UP25/06/11 P F506 ¢

• R300 PSY UO02259

     UP25/03/08 P F503

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.