IT18624: Activation specification deadlocks when server session is renewed

APAR status

• Closed as program error.

Error description

• A WebSphere MQ Resource Adapter (WMQ-RA) is being used within a
  WebSphere Application Server environment to drive an Activation
  Specification, consuming messages from a queue where the queue
  manager is being communicated with using CLIENT mode transport.
  
  The Activation Specification is observed to stop consuming
  messages from the queue, which in turn stops the application's
  MDB threads from running, even though there are messages
  available for consumption on the queue.  A thread dump taken of
  the JVM when it is in state shows the following pair of threads:
  
  DispatchThread
  --------------
  com/ibm/mq/connector/inbound/ServerSessionPoolImpl.getServerSess
  ion
  at
  com/ibm/msg/client/jms/internal/JmsConnectionConsumerImpl$JmsMes
  sageReferenceHandlerImpl.endDeliverInternal
  at
  com/ibm/msg/client/jms/internal/JmsConnectionConsumerImpl$JmsMes
  sageReferenceHandlerImpl.handleMessageReference
  at
  com/ibm/msg/client/jms/internal/JmsConnectionImpl$JmsProviderMes
  sageRefHandler.handleMessageReference
  at
  com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti
  onBrowserShadow.consumer
  at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.driveConsumer
  at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.deliverMsgs
  at
  com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgsReco
  nnectable
  at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgs
  at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.run
  at
  com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas
  k
  at
  com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
  runItem
  at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run
  at
  com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
  le.run
  at java/lang/Thread.run
  
  
  "WMQJCAResourceAdapter : nnnnn"
  -------------------------------
  at java/lang/Object.wait
  at java/lang/Object.wait
  at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire
     (entered lock:
  com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x00000007A6
  EE7058, entry count: 2)
  at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire
     (entered lock:
  com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x00000007A6
  EE7058, entry count: 1)
  at com/ibm/mq/jmqi/remote/api/RemoteHconn.requestDispatchLock
  at com/ibm/mq/jmqi/remote/api/RemoteFAP.MQCTL
  at
  com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.suspendAs
  yncService
  at
  com/ibm/msg/client/wmq/internal/WMQConnection.suspendAsyncServic
  e
  at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close
    (entered lock: java/lang/Object@0x00000007A3FD90A8, entry
  count: 1)
  at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close
  at com/ibm/msg/client/wmq/internal/WMQSession.stop
     (entered lock: java/lang/Object@0x00000007A80C3468, entry
  count: 1)
  at com/ibm/msg/client/jms/internal/JmsSessionImpl.stop
  at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
  at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
  at com/ibm/mq/jms/MQSession.close
  at com/ibm/mq/connector/inbound/ServerSessionImpl.close
  at
  com/ibm/mq/connector/inbound/ServerSessionPoolImpl.sessionReleas
  ed
     (entered lock: java/util/ArrayList@0x00000007A6EE83E8, entry
  count: 1)
  at
  com/ibm/mq/connector/inbound/MessageEndpointDeployment.releaseSe
  ssion
  at
  com/ibm/mq/connector/inbound/MessageEndpointDeployment.workCompl
  eted
  at com/ibm/ejs/j2c/work/WorkProxy.run
  at com/ibm/ws/util/ThreadPool$Worker.run
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of the:
  
    - WebSphere MQ V7.1 JCA Resource Adapter
    - WebSphere MQ V7.5 JCA Resource Adapter
    - IBM MQ V8 JCA Resource Adapter
    - IBM MQ V9 JCA Resource Adapter
  
  who are using an Activation Specification to consume messages
  from an MQ Queue or Topic when connected to the queue manager
  either in CLIENT transport (using TCP/IP) or BINDINGs transport,
  where there are repeated failures which occur during the
  internal work associated with processing the message within the
  MDB before the MDB is driven, for example caused by repeated
  network failures which disconnect the application server from
  the queue manager.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  This problem is broadly similar to that documented under APAR
  IT15559:
  
  http://www.ibm.com/support/docview.wss?uid=swg1IT15559
  
  in that a deadlock occurs between two threads associated with
  the MQ Activation Specification as a result of attempting to
  close a ServerSession, while simultaneously driving new work
  through the Activation Specification.
  
  In IT15559, the ServerSession was being closed by the Pool
  Scavenger Thread as a result of a ServerSession being unused in
  the pool for a time which exceeded the pool timeout value.
  
  The distinction between that APAR and this one is that here, a
  ServerSession is being closed as a result of the following JEE
  method call throwing an exception to indicate a problem had
  occurred in the pre-processing of an MDB's message:
  
  
  javax.resource.spi.endpoint.MessageEndpoint.beforeDelivery(Metho
  d)
  
  For example, if the ServerSession's associated javax.jms.Session
  was no longer connected to the queue manager as a result of a
  network disconnection event, and two-phase commit (XA) was being
  used to run the transaction for the MDB, the application server
  would be unable to start the XA transaction to enlist the queue
  manager and the above method would fail.
  
  In this circumstance, the code design is to dispose the
  ServerSession and associated JMS Session and create a
  replacement to drive future work under the MDB.  The subsequent
  close() call on the ServerSession could result in deadlock with
  the Activation Specification's DispatchThread, if in parallel
  more work was being driven by another message being available on
  the queue.
  

Problem conclusion

• Similar to design change associated with IT15559, the code has
  been updated such that Activation Specification pool lock is no
  longer held when closing the ServerSession, which circumvents
  the deadlock.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v7.5       7.5.0.8
  v8.0       8.0.0.7
  v9.0 CD    9.0.2
  v9.0 LTS   9.0.0.1
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WMQ WINDOWS V7

• Fixed component ID

  5724H7220

Applicable component levels

• R710 PSY

     UP