IBM Support

IT22327: MQ-JMS deadlock when MDBs share the same ServerSessionPool between multiple Destinations

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • An message-driven bean (MDB) application, which is configured to
    run from an MQ queue using the IBM MQ classes for JMS, stops
    consuming messages after a period of time.
    
    The MDB shares a "javax.jms.ServerSessionPool" object with other
    MDBs which are running in the same JVM, which are consuming
    messages from different MQ queues.
    
    
    A Javacore taken when the messages had stopped processing
    messages shows Java call stacks of the form:
    
    
    "Thread-54" (id: 84, state: WAITING) priority=5,
    interrupted=false, daemon=false
      at java.lang.Object.wait
      at java.lang.Object.wait
      at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire
      at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire
      at com.ibm.mq.jmqi.remote.api.RemoteHconn.requestDispatchLock
      at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQCTL
      at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.MQCTL
      at
    com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs
    yncService
      at
    com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic
    e
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at
    com.ibm.msg.client.wmq.internal.WMQSession.loadMessageReference
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.consume
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.run
      at com.ibm.mq.jms.MQSession.run
      at myApplication.run
      at java.lang.Thread.run
    
    
    "DispatchThread:
    [com.ibm.mq.jmqi.remote.impl.RemoteSession[:/66477451][connectio
    nId=414D51436D7166766A6D73312020202037B6B0593A0C3822]]" (id: 88,
    state: WAITING) priority=5, interrupted=false, daemon=true
      at java.lang.Object.wait
      at java.lang.Object.wait
      at my.ServerSessionPool.implementation.getServerSession
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.endDeliverInternal
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.endDeliver
      at
    com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes
    sageRefHandler.endDeliver
      at
    com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti
    onBrowserShadow.consumer
      at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.callConsumer
      at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run
      at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas
    k
      at
    com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.
    runItem
      at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run
      at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.run
    WorkQueueItem
      at
    com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManage
    rImplementation$ThreadPoolWorker.run
    

Local fix

  • Do not share a:
    
      javax.jms.ServerSessionPool
    
    between MDBs.  Create a new instance for each MDB which is
    running within the JVM.
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects message-driven bean (MDB) applications
    deployed into application server environments that create
    multiple JMS "javax.jms.ConnectionConsumer" objects to consume
    messages from different destinations, where the JMS
    ConnectionConsumers share the same "javax.jms.ServerSessionPool"
    object instance.
    
    Activation Specification implementations which are provided by
    the IBM MQ JCA Resource Adapter are not affected by this
    problem, as this Resource Adapter does not share a
    "javax.jms.ServerSessionPool" between different Activation
    Specifications.
    
    This means that MQ Activation Specifications running within
    IBM's WebSphere Application Server are not affected by this
    problem.  However it can affect a Listener Port running within
    the WebSphere Application Server, if a ListenerPort for one MDB
    is stopped, and another one is started to consume messages from
    a different Destination on the same queue manager, due to the
    way in which the WebSphere Application Server pools JMS
    Sessions.
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    A deadlock may occur in an application using a message-driven
    bean (MDB) which uses a "javax.jms.ServerSessionPool"
    implementation which is shared across multiple MDBs which are
    configured to consume messages from different MQ destinations.
    
    Prior to APAR IT10205:
    
    http://www.ibm.com/support/docview.wss?uid=swg1IT10205
    
    in order to encounter the deadlock, you must have a
    ServerSessionPool maximum size of a single ServerSession.  A new
    lock introduced under APAR IT10205 results in you being able to
    encounter this problem with any ServerSessionPool size.  This
    APAR was introduced in MQ versions:
    
    MQ 8.0.0.5
    MQ 9.0.0.0
    
    
    The problem occurred when the MQ "DispatchThread" (which is a
    named thread associated with the MDB that browses for messages
    on the MQ Destination, configures a ServerSession with
    information about how to identify the browsed message, and
    requests that it initiates its MDB processing) attempted to get
    a ServerSession from the implemented
    "javax.jms.ServerSessionPool".  However the DispatchThread was
    unable to obtain a free ServerSession instance from the pool as
    all ServerSessions were in use.  In this scenario, the
    DispatchThread would block at that point on the
    "javax.jms.ServerSessionPool" implementation, waiting for a
    ServerSession to be returned to the pool following the
    completion of an running MDB onMessage() instance.
    
    This is seen in a Javacore with a stack of the form:
    
    "DispatchThread" (id: 88, state: WAITING) priority=5,
    interrupted=false, daemon=true
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:502)
      at my.ServerSessionPoolImplementation.getServerSession
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.endDeliverInternal
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.endDeliver
      at
    com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes
    sageRefHandler.endDeliver
      at
    com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti
    onBrowserShadow.consumer
      at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.callConsumer
      at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run
      ... ... ...
    
    
    However, the running MDB instance thread was currently blocked
    waiting for a lock which the above DispatchThread holds.  This
    thread had been asked to start its MDB processing cycle, which
    involves destructively consuming the message from the MQ
    Destination.
    
    When consuming messages to deliver to MDB instances in this way
    (browse-then-get), which is referred to as the Application
    Server Facilities mode of operation, IBM MQ's implementation of
    a JMS Session that is associated with a ServerSession uses a
    cached "javax.jms.MessageConsumer".  This is in order to
    optimise the consumption of messages from a particular queue by
    preventing the need to perform a new MQOPEN request each time
    the ServerSession runs.
    
    Because a "javax.jms.ServerSessionPool" implementation may
    contain ServerSessions which are being used to consume messages
    from multiple destinations, the MQ classes for JMS must verify
    that the message it has been told to go and consume to drive the
    MDB's "onMessage(javax.jms.Message)" method, is on the same
    Destination as the cached MessageConsumer is configured to
    consume messages from.  If the MessageConsumer does not match
    the Destination, the ServerSession closes the existing
    MessageConsumer, and creates a new one for the new Destination.
    
    However in order to do this, the thread executing the
    ServerSession processing required a lock owned by the
    "DispatchThread".  This result was that this thread gets blocked
    with a Java stack of the form:
    
    "Thread-1" (id: 84, state: WAITING) priority=5,
    interrupted=false, daemon=false
      at java.lang.Object.wait
      at java.lang.Object.wait
      at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire
      at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire
      at com.ibm.mq.jmqi.remote.api.RemoteHconn.requestDispatchLock
      at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQCTL
      at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.MQCTL
      at
    com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs
    yncService
      at
    com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic
    e
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at
    com.ibm.msg.client.wmq.internal.WMQSession.loadMessageReference
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.consume
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.run
      at com.ibm.mq.jms.MQSession.run
      at myApplication.run
      at java.lang.Thread.run
    
    
    This thread had also taken a new lock which was introduced under
    APAR IT10205, which prevents other ServerSession threads who are
    also trying to close their respective MessageConsumers to
    replace with one for associated with the correct MQ Destination.
     These threads have the form:
    
      at sun.misc.Unsafe.park
      at java.util.concurrent.locks.LockSupport.park
      at
    java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndChe
    ckInterrupt
      at
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue
    ued
      at
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQue
    ued
      at
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
      at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock
      at java.util.concurrent.locks.ReentrantLock.lock
      at
    com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs
    yncService
      at
    com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic
    e
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at
    com.ibm.msg.client.wmq.internal.WMQSession.loadMessageReference
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.consume
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.run
      at com.ibm.mq.jms.MQSession.run
      at myApplication.run
      at java.lang.Thread.run
    
    and would be observed if the ServerSessionPool size is greater
    than 1 when the problem is encountered.
    
    The net result of this was that the "DispatchThread" is stuck
    waiting for a ServerSession to be returned to the pool.  However
    all of the ServerSessions were stuck on a thread which is
    attempting to get a lock which the "DispatchThread" was holding.
     The result is that the MDB is deadlocked.
    

Problem conclusion

  • The IBM MQ classes for JMS has been updated such that when
    closing the "javax.jms.MessageConsumer" associated with the
    ServerSession, the lock held by the "DispatchThread" is no
    longer requested, which prevents the deadlock from happening.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v8.0       8.0.0.9
    v9.0 LTS   9.0.0.4
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT22327

  • Reported component name

    IBM MQ BASE M/P

  • Reported component ID

    5724H7261

  • Reported release

    900

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-09-08

  • Closed date

    2018-01-29

  • Last modified date

    2018-05-09

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    IBM MQ BASE M/P

  • Fixed component ID

    5724H7261

Applicable component levels

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"9.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
09 May 2018