PH61814: IBM MQ Z/OS QUEUE MANAGER ISSUED S026 ABENDS AND TERMINATED ABNORMALLY WITH REASON=00C510AB

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• One queue manager in a queue sharing group terminated abnormally
  with REASON=00C510AB with normal termination of another queue
  manager in the same queue sharing group. Before the abnormal
  termination it issued S026 abends. The root cause looks like a
  deadlock between tasks attempting to access a locked list header
  and structure failure/lossconn processing. As one of the
  deadlocked tasks holds a latch of the ethr chain, no changes can
  be made to this chain, which will cause additional processing to
  hang while waiting for the deadlock to be resolved - this can
  include impactPeer Level Recovery for any structures (not just
  the one involved in the deadlock), responding to structure
  events (resulting in the S026 abends and eventual queue manager
  termination), checkpoint processing and allocation/deallocation
  processing. The MSTR joblog prints as following: CSQ1: 5/29:
  18.24.12 S0121163 IXL016I CONNECTOR CSQCONNECTOR1 TO STRUCTURE
  QSG1CSQ_ADMIN 829 829 DISCONNECTING: 829 JOB CSQ1MSTR ASID 01CE
  REQUESTED DISCONNECT REASON=NORMAL. 18.24.12 S0121163 CSQE006I
  +CSQ1 CSQECLOS Structure CSQ_ADMIN connection 830 830 name
  CSQCONNECTOR1 disconnected 18.24.13 S0121163 CSQ5019I +CSQ1
  CSQ5DISC Disconnected from DB2 DB21 18.24.13 S0121163 CSQ9022I
  +CSQ1 CSQYASCP 'STOP QMGR' NORMAL COMPLETION CSQ2: 5/29:
  18.24.12 S0153715 CSQE007I +CSQ2 CSQESTE EEPLDISCFAILCONNECTION
  event 122 122 received for structure MQAPPL1 connection name
  CSQCONNECTOR1 18.24.12 S0153715 CSQE008I +CSQ2 CSQESTE Recovery
  event from CSQ1 123 123 received for structure MQAPPL1 18.24.12
  S0153715 CSQE011I +CSQ2 CSQESTE Recovery phase 1 started for 124
  124 structure MQAPPL1 connection name CSQCONNECTOR1 18.26.20
  S0153715 *IXL041E CONNECTOR NAME: CSQCONNECTOR2, JOBNAME:
  CSQ2MSTR, ASID: 01D9 580 580 HAS NOT RESPONDED TO THE
  DISCONNECTED/FAILED CONNECTION EVENT FOR 580 SUBJECT CONNECTION:
  CSQCONNECTOR1. 580 DISCONNECT/FAILURE PROCESSING FOR STRUCTURE
  CSQ2APPL1 580 CANNOT CONTINUE. 580 MONITORING FOR RESPONSE
  STARTED: 05/29/2024 18:24:12.864177. 580 DIAG: 0000 0000
  00000000 18.26.20 S0153715 *IXL040E CONNECTOR NAME:
  CSQCONNECTOR2, JOBNAME: CSQ2MSTR, ASID: 01D9 581 581 HAS NOT
  RESPONDED TO THE USER SYNC POINT EVENT. 581 USER SYNC POINT
  PROCESSING 581 FOR STRUCTURE CSQ2APPL1 CANNOT CONTINUE. 581
  MONITORING FOR RESPONSE STARTED: 05/29/2024 18:24:12.890347. 581
  DIAG: 0000 0000 00000000 18.26.20 S0153715 *IXL049E HANG
  RESOLUTION ACTION FOR CONNECTOR NAME: CSQCONNECTOR2 582 582 TO
  STRUCTURE CSQ2APPL1, JOBNAME: CSQ2MSTR, ASID: 01D9: 582 SYSTEM
  WILL TAKE ACTION AT 05/29/2024 18:36:49.926279. 18.36.49
  S0153715 IXL050I CONNECTOR NAME: CSQCONNECTOR2 TO STRUCTURE
  CSQ2APPL1, 659 659 JOBNAME: CSQ2MSTR, ASID: 01D9 659 HAS NOT
  PROVIDED A REQUIRED RESPONSE AFTER 722 SECONDS. 659 TERMINATING
  CONNECTOR TASK TO RELIEVE THE HANG. 18.36.53 S0153715 IXL016I
  CONNECTOR CSQCONNECTOR2 TO STRUCTURE QSG1CSQ_ADMIN TERMINATING
  700 : 700 JOB CSQ2MSTR ASID 01D9 REQUESTED DISCONNECT
  REASON=FAILURE. 18.36.53 S0153715 IXL016I CONNECTOR
  CSQCONNECTOR2 TO STRUCTURE CSQ2APPL1 TERMINATING: 70 701 JOB
  CSQ2MSTR ASID 01D9 CONNECTOR TASK TERMINATED. 18.36.53 S0153715
  CSQE006I +CSQ2 CSQECLOS Structure CSQ_ADMIN connection 713 713
  name CSQCONNECTOR2 disconnected 18.36.53 S0153715 IEA794I SVC
  DUMP HAS CAPTURED: 699 699 DUMPID=007 REQUESTED BY JOB
  (CSQ2MSTR) 699 DUMP TITLE=CSQ2,ABN=026-08110102,U=SYSOPR
  ,M=CSQVSRRX,PSW=070C0000814F67CC,ASID=01D9 18.36.56 S0153715
  IEA794I SVC DUMP HAS CAPTURED: 812 812 DUMPID=008 REQUESTED BY
  JOB (CSQ2MSTR) 812 DUMP TITLE=CSQ2, QUEUE MANAGER TERMINATION
  REQUESTED, REASON=00C510AB 18.36.56 S0153715 *CSQV086E +CSQ2
  QUEUE MANAGER ABNORMAL TERMINATION REASON=00C510AB
  

Local fix

• N/A
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 2 Modification 0,                    *
  *                 Release 3 Modification 0 and                 *
  *                 Release 4 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: Abend S026-08110101 and S026-08110102   *
  *                      followed by abnormal queue manager      *
  *                      termination 6C6 with REASON=00C510AB    *
  *                      occurs several minutes after an         *
  *                      application structure failed and a peer *
  *                      queue manager terminated.               *
  ****************************************************************
  While processing a structure failed event, CSQESTFA waits for
  any in progress commit operations to be forced to the log. One
  such operation was suspended because of a list being locked on
  the terminating queue manager. The list lock should have been
  released by Peer Level Recovery (PLR) for the terminating queue
  manager's connection to the structure, however this could not
  take place due to a latch held by CSQESTFA, resulting in a
  deadlock situation. The deadlock was detected and reported by
  XCF messages IXL041E, IXL040E and IXL049E and abend
  S026-08110101. Several minutes later XCF issued IXL050I and
  abend S026-08110102 causing the queue manager to terminate
  abnormally.
  

Problem conclusion

• CSQESTFA has been updated to prevent this deadlock situation.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UO02258 UO02259 UO03122

Modules/Macros

• CSQESTFA
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R200 PSY UO03122

     UP25/06/11 P F506 ¢

• R300 PSY UO02259

     UP25/03/08 P F503

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.