PH58934: IBM MQ Z/OS: HIGH CPU AND I/O FOR READS ON THE ACTIVE LOG FOR INTERNAL "INQUIRE GROUP INDOUBT" REQUESTS

APAR status

• Closed as fixed if next.

Error description

• Symptoms:
  There is high CPU in the MQ MSTR job and the z/OS Catalog and
  GRS address space (ASID). There is also high I/O activity for
  reads on the active log.
  
  The CSQ1LOGP EXTRACT output shows PHASE1 and PHASE1 records for
  a cscorrelator of 212.GURAGN03.  GURAGN03 is the Group Indoubt
  Service task for Group Units of Recovery (GROUPUR).
  
  In a dump from the time of the problem, the internal command
  trace table (NTRC) shows that many MQCMD_INQUIRE_CONNECTION
  commands (the PCF equivalent of DISPLAY CONN) were issued.
  Command Events also showed these commands. MSTR internal trace
  shows that these commands correlated with processing by a
  module CSQMCIGI, which is for Inquire Group Indoubt UOWs.  This
  processing was driven by XA_RECOVER commands from client
  channels that were visible in the CHIN trace.
  
  System trace might show interrupts in modules IGG0CLHA and
  IGDZILLA.
  
  When all the queue managers in the QSG were started, the I/O
  and CPU activity decreased.
  
  
  Cause:
  If an application that did an XA_OPEN with QSG-wide disposition
  does an XA_RECOVER, then the queue manager has to look for
  indoubt UOWs on all queue managers in the QSG. If a peer queue
  manager is not active, then the requesting queue manager has to
  read the peer's logs. If the peer is active, then the inquire
  group indoubt request/reply can instead go via the
  SYSTEM.QSG.UR.RESOLUTION.QUEUE.
  
  Each XA_RECOVER with inactive peers will result in the peers'
  logs being read even if the peer QMGR has not restarted since
  the logs were last read. This processing is inefficient.
  
  This APAR is to investigate whether the impact can be reduced
  for reading a peer queue manager's logs for internal "inquire
  group indoubt" requests.
  
  If there are frequent xa_recover requests, the cause of that
  needs to be investigated. It will not be addressed by this
  APAR.
  

Local fix

• Take steps to prevent the peer logs being read repeatedly.
  Some options:
  - Start all of the queue managers in the QSG
  - Do not use the QSG name for the xa_open
  - Disable GROUPUR. Service parm (internal name SYSPXAQSG) is
    required if GROUPUR is disabled without a change to the
    application to specify the queue manager name rather than
    the QSG name
  - Implement an application change to not issue xa_recover so
    frequently
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of IBM MQ for z/OS Version 9       *
  *                 Release 1 Modification 0, Release 2          *
  *                 Modification 0, Release 3 Modification 0     *
  *                 and Release 4 Modification 0.                *
  ****************************************************************
  * PROBLEM DESCRIPTION: An application that runs XA_OPEN with   *
  *                      QSG-wide disposition causes all         *
  *                      subsequent XA_RECOVER calls to check    *
  *                      the log of all inactive peers in the    *
  *                      group for in-doubt units of work.       *
  ****************************************************************
  The efficiency of the code has been improved here to cache the
  logs of inactive peers when they are read, reducing the
  processing involved in subsequent XA_RECOVER calls where no
  peers have restarted since the last XA_RECOVER call.
  

Problem conclusion

Temporary fix

Comments

• Closed FIN following agreement with customer.
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels