PH68927: MQ Z/OS: ABENDS SUCH AS 5C6-00D40017 IN CSQMGET, 5C6-00C90D06 IN CSQIMGE9 AND 5C6-00C90202 IN CSQIRBLD

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• Various errors occurred due to a rare situation relating to a
  page set.
  
  Applications received MQ reason code 2195 MQRC_UNEXPECTED_ERROR
  for an MQGET for a couple of different queues. For example, a
  QREP (or InfoSphere Data Replication) Apply task received:
  ASN0575E [...] : The program encountered a WebSphere MQ error
  "2195 (MQRC_UNEXPECTED_ERROR)" while issuing the WebSphere MQ
  command "MQGET" on object [queue-name]
  
  Dumps included:
  
  DUMP TITLE=XXXX,ABN=5C6-00D40017,U=YYYYYYYY,C=MQ900.940.MMC -CS
  QMGET ,M=CSQGFRCV,LOC=CSQMLPLM.CSQMGET +0000360A where 00D40017
  means an internal error occurred, and REG0=00C90900
  (CSQI_BAD_RECORD_TYPE)
  
  DUMP TITLE=XXXX,ABN=5C6-00C90D06,U=YYYYYYYY,C=MQ900.940.DMC -CS
  QIMGE9,M=CSQGFRCV,LOC=CSQILPLM.CSQIMGE9+00004D42 where 00C90D06
  = CSQI_RECORD_INVALID
  
  At queue manager restart, there was also an abend during
  rebuilding of queue indexes:
  
  DUMP TITLE=XXXX,ABN=5C6-00C90202,U=YYYYYYYY,C=MQ900.940.DMC -CS
  QIRBLD,M=CSQGFRCV,LOC=CSQILPLM.CSQIRBLD+000018C4 where 00C90202
  means "A page set page retrieved was not valid. The page was not
  a data page."
  
  The underlying problem is that the last 2 pages of the page set
  have been reused by other messages when they should still have
  been allocated to other messages. The problem goes back to when
  there was a page set expansion request (CSQP017I) for the page
  set despite the page set already being at its maximum size. The
  maximum was 64GB, or as shown in DISPLAY USAGE output, 16776178
  4KB pages. The expansion appeared to be successful with a
  CSQI031I message. There is a sequence of events where the space
  map for the last 2 pages in the page set has bits reset that
  makes them eligible to be reused and overwritten.
  
  Depending on the nature of the overwrite, there can be a number
  of different abends in the DMC (data manager) or MMC (message
  manager) components of the queue manager.
  

Local fix

• It would be very difficult to repair the pages set with the
  CSQUZAP utility. You should make a back-up of the page set
  as-is. Then, you can try to reformat the page set to rebuild it
  a couple of ways:
  
  - use the FORMAT option TYPE(RECOVER) to allow media recovery to
  occur for the page set. Depending on the number of messages and
  how log the page set has been damaged, this option could take a
  considerable amount of time. The MQ command DISPLAY USAGE
  TYPE(ALL) will show you the oldest log that is needed for page
  set recovery.
  
  - use the FORMAT option TYPE(NEW) to format the page set with no
  data saved so that the page set can be quickly made available
  again. In this situation, your applications will need to replay
  the messages (QREP has such an option). If there are other
  undamaged queues, their data could be saved off from the
  original page set with CSQUTIL SCOPY and then reloaded with
  CSQUTIL LOAD.
  
  For more information about CSQUTIL FORMAT, see
  https://www.ibm.com/docs/en/ibm-mq/9.4.x?topic=zos-formatting-
  page-sets-format
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 3 Modification 0 and Release 4       *
  *                 Modification 0.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION: When a page set reaches the 64GB        *
  *                      maximum size and the queue manager is   *
  *                      restarted, applications may receive     *
  *                      MQRC_UNEXPECTED_ERROR (reason code      *
  *                      2195) when attempting to retrieve       *
  *                      messages from queues using the page     *
  *                      set. This results in a 5C6-00C90202     *
  *                      abend and various possible follow on    *
  *                      abends.                                 *
  ****************************************************************
  When a 64GB page set reaches its maximum size and the queue
  manager restarts, a flag to mark the page set as no longer
  extendable is not properly set, causing a spurious page set
  expansion that incorrectly marks the last two pages as free.
  These pages are then reused and overwritten, resulting in page
  set corruption and possible loss of messages.
  

Problem conclusion

• The code has been modified to properly flag the page set as not
  extendable when it reaches the 64GB maximum size, preventing
  spurious expansion attempts. Additionally, the space map
  initialization logic has been corrected to preserve
  already-allocated page bits, preventing corruption of the last
  two pages in the page set.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UO06638 UO06639

Modules/Macros

• CSQIFRMT CSQP1CON
  

Fix information

• Fixed component name

  IBM MQ Z/OS V9

• Fixed component ID

  5655MQ900

Applicable component levels

• R300 PSY UO06639

     UP26/02/14 I 1000 ¢

• R400 PSY UO06638

     UP26/02/14 I 1000 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.