PH62694: MQ Z/OS: THE COMMAND PROCESSOR AND OTHER THREADS ARE HUNG DUE TO A LATCH HELD DURING A LOOPING MQOPEN FOR A SHARED QUEUE

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• The initially reported symptom was that MQ commands timed out
  after 30 seconds (the default RESPTIME).
  
  A dump and trace showed that the command processor thread
  (RTSSRV01) had been waiting for a long time for an internal
  serialization latch.  The console service thread (GCSCN602),
  expiry control thread (EXPCTL01), and checkpoint/restart thread
  (RCRSC) were also waiting for the latch.
  
  Due to the checkpoint delay, the following message later
  occurred:  CSQJ169E Last checkpoint not found in active log
  
  The latch, which should normally only be held for a short time,
  was held for a long time by a thread that was trying to open a
  shared queue.  The MQOPEN processing was looping due to an
  inconsistency in MQ's CSQ.OBJ_B_QUEUE Db2 table.  Trace
  included repeated CsqeRsnCodeSCBAlreadyExists errors for
  CSQEOPEN and CSQEROUT.
  
  CSQ5PQSG VERIFY of the table reported:
  CSQU572E  Usage map T03_LH_ARRAY and DB2 table CSQ.OBJ_B_QUEUE
            inconsistent
  CSQU579E  Queue 2 in usage map for structure <str-name> is in
            use multiple times
  CSQU533E  SQL error. DB2 table=CSQ.OBJ_B_QUEUE, code=-811,
            state=21000, data=-140 00000000 00000000 00000000
            00000000
  CSQU510I  SQL error information
  CSQU511I   DSNT408I SQLCODE = -811, ERROR:  THE RESULT OF AN
                      EMBEDDED SELECT
  CSQU511I            STATEMENT OR A SUBSELECT IN THE SET CLAUSE
                      OF AN UPDATE
  CSQU511I            STATEMENT IS A TABLE OF MORE THAN ONE ROW,
                      OR THE RESULT OF A
  CSQU511I            SUBQUERY OF A BASIC PREDICATE IS MORE THAN
                      ONE VALUE
  CSQU511I   DSNT418I SQLSTATE   = 21000 SQLSTATE RETURN CODE
  CSQU511I   DSNT415I SQLERRP    = DSNXREMS SQL PROCEDURE
                      DETECTING ERROR
  CSQU511I   DSNT416I SQLERRD    = -140  0  0  -1  0  0 SQL
                      DIAGNOSTIC INFORMATION
  CSQU511I   DSNT416I SQLERRD    = X'FFFFFF74'  X'00000000'
                      X'00000000'
  CSQU511I            X'FFFFFFFF'  X'00000000'  X'00000000' SQL
                      DIAGNOSTIC
  CSQU511I            INFORMATION
  ...
  CSQU148I CSQ5PQSG Utility completed, return code=16
  
  Db2 SPUFI output for the table showed that there were two
  list headers with the same number for the same structure in the
  QSG (Queue Sharing Group).
  
  This APAR is to request logic to detect the situation leading
  to the loop. The logic should gather diagnostics to help
  figure out why the inconsistency occurred and should prevent
  the loop.
  

Local fix

• The support center can provide instructions to use CSQUZAP
  to repair the inconsistency in the table. The support center
  will need the CSQ5PQSG VERIFY output, SPUFI output, and a
  dump of the queue manager and the application structure in
  question as described in
  https://www.ibm.com/support/pages/node/409959
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 3 Modification 0, and Release 4      *
  *                 Modification 0.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION: A loop in MQOPEN processing when        *
  *                      attempting to open a Shared Queue       *
  *                      causes an application to hang. This     *
  *                      causes the Queue Manager to not respond *
  *                      to commands.                            *
  ****************************************************************
  An MQOPEN for a shared queue that shares the same list header as
  another shared queue causes a loop. It is not expected that 2
  shared queues will share the same list header in this context.
  
  For the duration of this loop a latch is held and not released
  resulting in other threads remaining stuck waiting for the latch
  to be freed. Threads that process the console, the command
  runtime server and the checkpointing task may be affected.
  

Problem conclusion

• Code has been added to detect and break out of the loop with
  abend code 5C6-00C5105C and take a dump of the Application and
  Queue Manager address spaces, including the structure that the
  shared queue is defined on for analysis.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UO04314 UO04315

Modules/Macros

• CSQEOSQ  CSQESTRT CSQIRECP
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R300 PSY UO04315

     UP25/09/13 P F509

• R400 PSY UO04314

     UP25/09/13 P F509

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.