PH67936: FAILED-PERSISTENT CONNECTIONS TO CSQ_ADMIN STRUCTURE

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• The PLEX was lost due to an unplanned outage and upon recovery
  noticed that the qmgrs in one of the QSGs was experiencing
  issues connecting to the qsgCSQ_ADMIN structure. Noticed the
  qmgrs flipping between 'active' and 'failed-persistent'
  connections.
  
  The following messages were seen:
  
  IXL014I IXLCONN REQUEST FOR STRUCTURE qsgCSQ_ADMIN WAS
  SUCCESSFUL. JOBNAME: XXXXMSTR ASID: nnnn CONNECTOR NAME:
  CSQEqsgXXXXXX CFNAME: XXX ADDITIONAL STATUS INFORMATION:
  CONNECTION HAS BEEN REESTABLISHED
  
  XCSQE021I cpf CSQECONN Structure CSQ_ADMIN connection as
  CSQEqsgXXXXXX warning, RC=00000004 reason=02010407
  codes=00000000 00000000 00000000
  
  IXC567I CONNECTION CSQEqsgXXXXXX TO STRUCTURE qsgCSQ_ADMIN
  FAILED-PERSISTENT: DISCONNECT/FAILURE PROCESSING COMPLETED.
  
  Recover was done by shutting down all the qmgrs, forcing the
  qsgCSQ_ADMIN structure and then successfully restarting the
  qmgrs.
  
  The dumps show that when the queue managers connect to the admin
  structure they are notified of the previous lossconn event for
  the structure and disconnect. At this point CSQESTRD is called
  to delete the admin structure (including any failed persistent
  connections) and rebuild it from the queue managers' logs,
  however this deletion does take place because the structure is
  marked as being in XCF rebuild processing - as IXLFORCE would
  fail if rebuild is in progress, no attempt to delete the
  structure is made.
  
  The DISPLAY XCF output shows that the duplexing rebuild has
  reached the 'DUPLEX ESTABLISHED' state, which according to the
  docs can last indefinitely, and looking at the qmgr it is
  possible to see that the QUASTRSTREBLD flag (which is checked by
  MQ to determine if a rebuild is in progress) is indeed set for a
  duplexed structure - this means that CSQESTRD will always skip
  attempting deletion of the admin structure if it is duplexed,
  leading to the reported looping condition.
  
  This suggests the 'System Outage' caused the queue managers to
  receive an EEPLLOSSCONN event for their connection to the admin
  structure (presumably after the structure stopped being
  duplexed) before failing. When the queue managers were next
  started, the structure was already duplexed again, causing the
  above behavior when attempting to reconnect to the admin
  structure.
  
  Looking at the docs for IXLFORCE, it looks like the delete
  attempt (had the qmgr attempted it) would have succeeded in this
  situation (i.e. the rebuild is a System managed Duplexing
  rebuild in Duplex Established) (
  https://www.ibm.com/docs/en/zos/3.1.0?topic=idpsfc-description)
  
  The QUASTRREBLDDUPLEXESTABLISHED should be considered in this
  case. The QUASTRSTREBLD indicates the rebuild in progress as
  well. However, this bit is on for non-duplexed structures under
  rebuild (ie from CF1 to CF2, for whatever reason). For duplexed
  structures, this bit will be ON as long as the duplex is
  established.
  
  Regarding the IXLFORCE, if the last connection to a
  non-persistent structure is deleted, the structure will also be
  deleted.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 4 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: Queue managers repeatedly connect and   *
  *                      disconnect from the MQ Admin structure  *
  *                      (CSQ_ADMIN) following a loss of         *
  *                      connection event, if the admin          *
  *                      structure is duplexed.                  *
  ****************************************************************
  Following a loss of connection event to the admin structure, all
  queue managers that were connected to the structure disconnected
  from the structure and a rebuild of the admin structure should
  be initiated, including the processing/removal of
  failed-persistent connections to the structure. However, when
  the admin structure is duplexed, CSQESTRD incorrectly determines
  that a rebuild is already in progress. When the structure is
  reconnected, the state of the failed-persistent connections
  causes this processing to repeat.
  

Problem conclusion

• When determining whether to initiate a rebuild of the admin
  structure, the checking has been corrected for duplexed admin
  structures.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UO05367

Modules/Macros

• CSQEQRY6
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R400 PSY UO05367

     UP25/11/12 P F511 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.