IT06660: WEBSPHERE APPSERVER ON Z/OS ACTIVATION SPEC DEADLOCKS WHILE BEING PAUSED

APAR status

• Closed as program error.

Error description

• A WebSphere MQ messaging provider activation specification in
  WebSphere Application Server for z/OS may hang while being
  paused due to a logical deadlock between two threads.
  
  In a Javacore you will see the following lock information:
  
  3LKMONOBJECT com/ibm/mq/connector/inbound/
   ServerSessionImpl$IsInUseLock@0x
   000000482E732E40/0x000000482E732E58:
  
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "WebSphere t=00ab8408"
   (0x0000004808CDFC00)
  2LKMONINUSE sys_mon_t:0x0000004812E59AC0 infl_mon_t:
   0x0000004812E59B30:
  
  This is followed by thread information where two threads are
  shown to be in the following state:
  
  WebSphere t=00ab8408" J9VMThread:0x0000004808CDFC00,
   Java callstack:
  at java/lang/Object.wait(Native Method)
  at java/lang/Object.wait(Object.java:167
   (Compiled Code))
  at com/ibm/mq/connector/inbound/
   ServerSessionImpl.close(ServerSessionImpl.java:299)
  at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.
   closeInternal(ServerSessionPoolImpl.java:648)
  at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.
   close(ServerSessionPoolImpl.java:557)
  at com/ibm/mq/connector/inbound/MessageEndpointDeployment.
   stop(MessageEndpointDeployment.java:473)
  at com/ibm/mq/connector/ResourceAdapterImpl.
   endpointDeactivation(ResourceAdapterImpl.java:523)
  
  and
  
  "WebSphere:ORB.thread.pool t=00ac0528"
   Java callstack:
  at com/ibm/ws390/orb/ClientDelegate.
   invokeRequestCFW(Native Method)
  at com/ibm/ws390/orb/ClientDelegate.
   commonInvoke(ClientDelegate.java:1042
   (Compiled Code))
  at com/ibm/ws390/orb/ClientDelegate.
   invoke(ClientDelegate.java:889)
  at org/omg/CORBA/portable/ObjectImpl.
   _invoke(ObjectImpl.java:484)
  at com/ibm/ws390/management/connector/corba/
   _CorbaConnectorStub.invoke
   (_CorbaConnectorStub.java:634)
  at com/ibm/ws390/management/connector/corba/
   CorbaConnectorClient.invoke
   (CorbaConnectorClient.java:903)
  at com/ibm/ws390/management/InterProcessAdminService.
   invoke(InterProcessAdminService.java:371)
  at com/ibm/ws/management/AdminServiceImpl$1.
   run(AdminServiceImpl.java:1432)
  at com/ibm/ws/security/util/AccessController.
   doPrivileged(AccessController.java:118
   (Compiled Code))
  at com/ibm/ws/management/AdminServiceImpl.
   invoke(AdminServiceImpl.java:1224)
  at com/ibm/ws/wmqra/utils/EndpointPauser$1.run
   (EndpointPauser.java:99)
  at com/ibm/ws/sib/security/auth/AuthUtilsImpl$5.
   run(AuthUtilsImpl.java:1454)
  at com/ibm/ws/security/auth/ContextManagerImpl.
   runAs(ContextManagerImpl.java:5365)
  at com/ibm/ws/security/auth/ContextManagerImpl.
   runAsSystem(ContextManagerImpl.java:5453)
  at com/ibm/ws/sib/security/auth/AuthUtilsImpl.
   runAsSystem(AuthUtilsImpl.java:1449)
  at com/ibm/ws/wmqra/utils/EndpointPauser.
   pauseJ2CMessageEndpoint(EndpointPauser.java:96)
  at com/ibm/ws/wmqra/helper/WASRuntimeHelper.
   deliveryFailed(WASRuntimeHelper.java:326)
  at com/ibm/mq/connector/inbound/WorkImpl.
   callDeliveryFailed(WorkImpl.java:511)
  
  
  The Job Log might also report the threads as hung, as per
  WebSphere Application Server hung thread detection processing.
  For example:
  
  BBOO0221W: WSVR0605W: Thread "WebSphere WLM Dispatch Thread
  t=00ac0528" (00000011) has been active for 706600 milliseconds
  and may be hung. There is/are 1 thread(s) in total in the server
  that may be hung.
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of:
  
  - The WebSphere Application Server for z/OS V7 WebSphere MQ
  messaging provider.
  - The WebSphere Application Server for z/OS V8 WebSphere MQ
  messaging provider.
  - The WebSphere Application Server for z/OS V8.5 WebSphere MQ
  messaging provider.
  
  who are using an activation specification to drive a
  Message-Driven Bean (MDB) application which consumes messages
  from WebSphere MQ.
  
  
  Platforms affected:
  z/OS
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  The default configuration of an activation specification defined
  against the WebSphere MQ messaging provider within WebSphere
  Application Server is to stop the messaging endpoint on the
  first failure to deliver a message. This behaviour is
  configurable using the properties:
  
  - Stop endpoint if message delivery fails
  
  and
  
  - Number of sequential delivery failures before suspending
  endpoint
  
  
  Using the default configuration, if a Message-Driven Bean (MDB)
  application rolls a message back because it could not be
  processed, WebSphere Application Server will attempt to stop the
  activation specification. When this is requested, the activation
  specification waits for all current activity (included queued up
  work) to complete, either successfully or due to an error,
  before stopping.
  
  It was observed that when the activation specification on
  WebSphere Application Server for z/OS attempted to pause, it
  ended up hanging because of two Java threads were in a logical
  deadlock processing the stop request.
  
  The scenario was as follows.
  
  A thread that was running a Server Session and had driven the
  MDB application with a message requested the endpoint to stop.
  For WebSphere Application Server for z/OS, this thread was not
  necessarily the thread that performed the actual stop of the
  endpoint and the subsequent closure of the Server Session Pool
  associated with the activation specification. In this case, the
  thread associated with the Server Session waited for the
  deactivation of the endpoint to complete, while the Server
  Session was still marked as "in use". However, the thread that
  was processing the endpoint deactivation waited for all Server
  Sessions from the activation specification's Server Session Pool
  to complete their processing and to be returned to the pool.
  
  This resulted in the thread stopping the endpoint waiting for a
  Server Session to be returned to the pool, which was busy
  waiting for the processing of stopping the endpoint to complete.
  

Problem conclusion

• The WebSphere MQ resource adapter code has been changed to
  ensure when WebSphere Application Server for z/OS requests a
  stop of the messaging endpoint (activation specification), the
  thread processing the stop request does not wait for the Server
  Session that requested the stop to complete before pausing.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v7.0       7.0.1.14
  v7.1       7.1.0.7
  v7.5       7.5.0.6
  v8.0       8.0.0.3
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  PI35752

Fix information

• Fixed component name

  WMQ WINDOWS V7

• Fixed component ID

  5724H7220

Applicable component levels

• R710 PSY

     UP