PH63327: MQ Z/OS: CFSTRUCT RECOVERY FAILS WITH CSQE101I CSQERCFT UNABLE TO BACKUP OR RECOVER STRUCTURE <NAME>, STRUCTURE IN USE

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• In a Disaster Recovery (DR) or role swap scenario, structure
  recovery is being done as described at
  https://www.ibm.com/docs/en/ibm-mq/9.3?topic=mq-alternative-
  site-recovery-zos#q022650___RecoveringAQueueSharingGroupAtThe
  
  The recovery is failing with:
    CSQE101I CSQERCFT Unable to backup or recover
    structure <structure-name>, structure in use
  
  That message can correctly occur if a BACKUP or
  RECOVER CFSTRUCT command is in progress. However, in this case
  the structure remained in a FAILED state for an extended time.
  
  Prior to the queue manager starting, CSQ5PQSG was run to do
  ADD QMGR for each queue manager in the Queue Sharing Group
  (QSG) as instructed in the link above. The purpose of the job
  is to add the queue manager to XCF--recovered Db2 tables
  already have knowledge of the queue managers. Some of the ADDs
  failed with:
    CSQU527E  No eligible DB2 currently active
  while a couple of the ADDs worked.
  
  When one of the queue managers that had been added to XCF tried
  to do the automatic recover (due to the RECAUTO setting), some
  structures recovered if the BACKUP CFSTRUCT information was
  held in the recovery logs for the queue managers where the ADD
  worked.  Other structures had a failure of:
    CSQJ077E LOG OR BSDS READ ERROR FOR QMGR
     , REASON CODE=00D10901
    CSQE158E CSQERCFT Recovery of structure <structure-name>
    failed, reason=00C5005B
  
  Even after the ADD QMGR worked for the other queue managers in
  the QSG, the CSQE101I message continued.
  
  The problem is that a GRS ENQ for SYSZCSQE was not released
  when the earlier errors occurred.  The queue manager that first
  tried to do the recovery needed to be restarted to release
  the ENQ.
  
  To find information about the ENQ, one option is to use SDSF
  with the command ENQ SYSZCSQE. The ENQ to look for has MAJOR
  and MINOR names as follows:
  
  MAJOR:
  SYSZCSQE
  
  MINOR:
  CSQEStop concurrent structur<qsgname><qsgname><structurename>
  
  For example, for structure "APPLICATION1" in QSG "QSG1" the
  Minor name would be:
  CSQEStop concurrent structurQSG1QSG1APPLICATION1
  
  Note that there will be a number of other ENQs with
  MAJOR=SYSZCSQE but different MINOR names that can be expected
  to be held for the lifetime of the queue manager.
  

Local fix

• Restart the queue manager that first tried to do the recovery
  and that unexpectedly holds the ENQ while no backup or active
  recovery is in progress.
  
  To avoid this situation, be sure that CSQ5PQSG ADD QMGR is
  successful for every queue manager in the QSG before starting
  any queue managers.  Even though Db2 already knows about the
  queue manager, Db2 needs to be active when the utility is run.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 3 Modification 0 and                 *
  *                 Release 4 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: RECOVER CFSTRUCT commands and automatic *
  *                      structure recovery fail, reporting      *
  *                      CSQE101I 'Unable to backup or recover   *
  *                      structure <struc-name>, structure in    *
  *                      use'.                                   *
  ****************************************************************
  While validating the structure(s) that have been scheduled for
  automatic structure recovery, CSQERCFT obtains an ENQ for
  each structure to prevent concurrent attempts to recover them
  taking place. The ENQ is released when structure recovery
  completes (successfully or unsuccessfully).
  
  When multiple structures are being recovered by CSQERCFT, an
  error in cleanup logic following a validation failure (for
  example due to an error locating the required BSDS/log records
  for a queue manager that owns a structure backup) can result in
  the resource name being incorrectly constructed, and the DEQ
  request consequently failing to release the ENQ.
  
  This results in subsequent attempts to recover the affected
  structure failing (on all queue managers in the queue sharing
  group) due to the unreleased ENQ incorrectly indicating that
  recovery of that structure is already in progress.
  

Problem conclusion

• The cleanup logic in CSQERCFT has been corrected so that the
  correct resource name is constructed and the ENQ is correctly
  released.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI98793 UI98794

Modules/Macros

• CSQERCFT
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R300 PSY UI98794

     UP24/11/16 P F411 ¢

• R400 PSY UI98793

     UP24/11/16 P F411 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.