PI66709: WMQ: SHUTDOWN HANGS FOR AN MQ APPLICATION AND THEN THE QUEUE MANAGER AFTER Z/OS MAINTENANCE IS APPLIED.

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• z/OS maintenance for PUT 1604 or RSU 1604 has been applied that
  includes PTFs
    UA81297 z/OS 2.2
    UA81298 z/OS 1.13
    UA81300 z/OS 2.1
  
  Termination of an application and/or queue manager may result
  in a hang or "no work" loop waiting for tasks to end.
  
  The problem occurs for MQ batch applications, including RRS
  batch such as WebSphere Application Server, that end without
  doing an MQDISC and which have been doing MQGETs with MQGMO_WAIT
  or MQGMO_SET_SIGNAL. Those hung applications can in turn affect
  MQ shutdown.
  
  Systrace has a timed wait / post loop.
  
  TCBs for the application address space have SVRBs that are
  waiting with PSWs in CSQMEOTC. They are waiting for the wEndECB
  to be posted by CSQBTEXT.
  
  The expectation in the MQ code is that CSQBTEXT is initially
  scheduled as an exit on a STIMERM call and then continues to
  issue 1 second STIMERM calls to reschedule itself until it is
  either notified that the queue manager is terminated or that
  the application task no longer requires the exit to be running.
  
  In the hang situation, the exit would be driven repeatedly
  until the application address space starts to end, at which
  point STIMERM is issued but the CSQBTEXT exit does not get
  re-driven.
  
  If the application is WebSphere Application Server, the shut
  down of the control region may time out with ABENDSA03 and
  ABENDSE6C and messages:
  
   BPXM020I BPXBATCH FAILED BECAUSE MVSPROCCLP (BPX1MPC) FAILED
   WITH RETURN CODE 0000009D REASON CODE 0B7000B9
  
   BPXP018I THREAD 0000000000000000, IN PROCESS nnnnnnnn, ENDED
   WITHOUT BEING UNDUBBED WITH COMPLETION CODE 80A03000 AND
   REASON CODE 00000000
  
   BPXM020I BPXBATCH FAILED BECAUSE MVSPROCCLP (BPX1MPC) FAILED
   WITH RETURN CODE 0000009D REASON CODE 0B7000B9
  
  The reason code 0B7000B9 means :
  "JRActiveProcess: Multiple processes in the address space
   prevent the termination of the job step process."
  
  For the terminating queue manager, the PSW for the SVRB for the
  STIMER (0A2F) WAIT request is in CSQ3EPX.
  
  STOP QMGR MODE(FORCE) does not end the queue manager.
  
  
  ** This fix is required even if the PTF for OA50970 is **
  ** applied to z/OS.                                    **
  
  Additional Symptom(s) Search Keyword(s):
  ABENDA03 A03 SA03 S0A03
  ABENDEC6 EC6 SEC6 S0EC6
  

Local fix

• To prevent the problem:
   A ++APAR is available from the support center.  The queue
   manager will need to be recycled to pick up the fix.
   .
   As stated in OA50970, if the MQ relief cannot be installed
   promptly, it is advised to remove the z/OS PTF until the time
   the MQ relief can be installed.
  .
  If the problem has occurred:
  Use MVS CANCEL for the hung connected application regions.
  .
  If you need to determine the hung application, you can create a
  dump of the queue manager:
      /DUMP COMM=(dump title)
      /R xx,SDATA=(CSA,RGN,PSA,SQA,LSQA,TRT,SUM,GRSQ),CONT
      /R xx,JOBNAME=(ssidMSTR),END
  In IPCS, run the command VERBX GRSTRACE and search for ERLYOLRH,
  for instance:
   MAJOR NAME: SYSZCSQ3
     MINOR NAME: C5D9D3E8D6D3D9C816078530
     *ERLYOLRH....        *
       SCOPE: SYSTEM    SYSNAME: <id>     STATUS: *SHARED*   /OWN
         ASID: <nnnnnnnn>   TCB: <nnnnnnnn>  JOBNAME: <batch job>
         Critical ENQ Time(s):
           Request:     <date> <time>
           Grant:       <date> <time>
  Compare the ENQ time with the time the dump was created. The
  ERLYOLRH ENQ is not expected to be held for very long.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All users of Websphere MQ for z/OS Version 7 *
  *                 Release 1 Modification 0.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: After application of z/OS maintenance   *
  *                      APAR OA49676 (PTFs UA81297/UA81298/     *
  *                      UA81300) MQ batch                       *
  *                      application may hang and prevent        *
  *                      shutdown of the Queue Manager.          *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
   The z/OS maintenance (OA49676) affects the end of task
  processing for MQ batch applications which do not disconnect
  from MQ prior to ending. This leaves the application in a hung
  state, and prevents the queue manager from completing a
  shutdown. This requires the application to be cancelled to allow
  shutdown to complete correctly. This also prevents Websphere
  Application Server from ending correcting when running MQ
  applications.
  

Problem conclusion

• End of task processing for MQ applications has been updated to
  correctly complete in this scenario.
  100Y
  CSQMEOTC
  

Temporary fix

• *********
  * HIPER *
  *********
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PI66686

• APAR is sysrouted TO one or more of the following:

  UI40127

Modules/Macros

• CSQMEOTC
  

Fix information

• Fixed component name

  WMQ Z/OS V7

• Fixed component ID

  5655R3600

Applicable component levels

• R100 PSY UI40127

     UP16/08/25 P F608 ¢

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.