Troubleshooting Hints and Tips for Shared Queues on z/OS

Resolving The Problem

Back to top

• Topology details for all distributed queueing related problems. An inbound channel may put to a shared queue, and an outbound channel may get from a shared xmitq. Include the following information for the queue managers and clients that are having the failure:
• A general description of the distributed queueing environment.
• Operating system version and release.
• WebSphere MQ version, release and maintenance level.
• A full display of the related object definitions:
• display queue
• display group
• display cfstruct
• display cfstatus

• OS/390 and z/OS GTF Trace
• OS/390 and z/OS MSTR Internal Trace
• OS/390 and z/OS CHIN Trace

• Search the WebSphere MQ support site for known problems using symptoms like the message number and error codes.
• Gather the documentation and work with the WebSphere MQ support team to resolve your problem.
• When migrating to a new release of WebSphere MQ, check the customization and migration sections of the WebSphere MQ for z/OS System Setup Guide for instructions regarding CSQ45xxx jobs, which update DB2 resources. Failure to run these jobs may result in various symptoms.
• Likewise, when applying maintenance to WebSphere MQ, check the ACTION HOLDS for any CSQ45xxx jobs that need to be run.
• It is recommended you take a backup of all your CF structures about every hour, to minimize the time it takes to restore a CF structure. Use the BACKUP CFSTRUCT command.
• If the administration structure becomes full, the exact symptoms depend on which processes experience the error. They might range from no responses to CMDSCOPE(GROUP) commands, to queue manager failure as a result of problems during commit processing. ABEND5C6-00C51090 is one possible symptom of a full structure. ABEND5C6-00C51090 is one possible symptom of a full structure. See HIPER APAR PK27360(6.0.0). APAR PK27360 was incorporated into 7.0.0 and 7.0.1 Base code.
• See Planning your structure size and technote 1264321 for minimum recommended structure sizes.
• See technote 1144275 for a symptom of RC2192 MQRC_PAGESET_FULL on an MQPUT to a shared queue.
• See "Performance changes in PK14315(MQ531) and PK15776(MQ600 and Base MQ701) to reduce the CPU used when a message arrives on a Shared IMS Bridge queue".
• See "ENTSUSED and SIZEUSED for a WebSphere MQ application structure are greater than 0 even when there is no depth on the shared queues".

• Coupling Facility failures with MQ for z/OS V5.3.1 and V6.0 and V701:
  
  There are two main types of failure, and MQ handles these in different ways. See Coupling Facility and structure failure for clarification of how failure scenarios are handled in V6.0 and V701.
  
  - Firstly, there is a structure failure, which MQ can tolerate at V6 and V701. At V5, if an administration structure fails, all queue managers in the queue-sharing group terminate.
  
  - Secondly, there is a connection failure to the CF (IXLRSNCODENOCONN or reason X'00000C06'), which MQ does not tolerate. MQ will terminate if it receives a connection failure for either admin or application structures. The reason is that MQ doesn't know if someone else is still connected to the structure (they have not received the connection failure). To maintain data integrity, the queue manager must terminate. Other queue managers in the queue-sharing group might still have connectivity and be able to perform peer level recovery.
  
  - For testing purposes, a tool called INJERROR is available to simulate damage to a structure. It is not advisable to do such testing in a production environment.
  
  In WMQ for z/OS 5.3.1, failure of a queue-sharing group's administration (admin) CF structure will cause all the queue managers in the queue sharing group (QSG) to abnormally terminate. CF duplexing of your admin CF structures will reduce the possibility of CF structure failure. However this has an associated performance cost. MQ does not do any management of duplex CF structures, and relies on the CF services to transparently handle duplexing on behalf of MQ.
  
  For V6 and V7 MQ enables queue managers within a QSG to tolerate failure of that QSG's admin CF structure. On detecting such a failure each queue manager will rebuild its part of the admin structure from its in-memory data. Peer recovery for a queue manager will be suspended until that queue manager has rebuilt its part of the admin structure, meaning that in-flight MQGETs and in-commit MQPUTs messages for the failed queue manager will only be available to other queue managers when the failed queue manager has restarted. All other applications, including shared queue applications, can carry on processing without the admin structure being available. A queue manager at V701 or later in the queue sharing group can build another queue manager's entries in the admin structure.

• Best Practices: WebSphere MQ shared queues and application programs
• Library and Information centers

Back to top

• See Exchanging information with IBM Technical Support for FTP and email instructions using the IBM Enhanced Customer Data Repository (ECuRep).
• Go to the Service Request page to open or update a problem.
  Note: Always update your Service Request (problem record) to indicate that data has been sent.

Back to top