IBM Support

IT18624: Activation specification deadlocks when server session is renewed

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A WebSphere MQ Resource Adapter (WMQ-RA) is being used within a
    WebSphere Application Server environment to drive an Activation
    Specification, consuming messages from a queue where the queue
    manager is being communicated with using CLIENT mode transport.
    
    The Activation Specification is observed to stop consuming
    messages from the queue, which in turn stops the application's
    MDB threads from running, even though there are messages
    available for consumption on the queue.  A thread dump taken of
    the JVM when it is in state shows the following pair of threads:
    
    DispatchThread
    --------------
    com/ibm/mq/connector/inbound/ServerSessionPoolImpl.getServerSess
    ion
    at
    com/ibm/msg/client/jms/internal/JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.endDeliverInternal
    at
    com/ibm/msg/client/jms/internal/JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.handleMessageReference
    at
    com/ibm/msg/client/jms/internal/JmsConnectionImpl$JmsProviderMes
    sageRefHandler.handleMessageReference
    at
    com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti
    onBrowserShadow.consumer
    at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.driveConsumer
    at com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.deliverMsgs
    at
    com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgsReco
    nnectable
    at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgs
    at com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.run
    at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas
    k
    at
    com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
    runItem
    at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run
    at
    com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
    le.run
    at java/lang/Thread.run
    
    
    "WMQJCAResourceAdapter : nnnnn"
    -------------------------------
    at java/lang/Object.wait
    at java/lang/Object.wait
    at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire
       (entered lock:
    com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x00000007A6
    EE7058, entry count: 2)
    at com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire
       (entered lock:
    com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x00000007A6
    EE7058, entry count: 1)
    at com/ibm/mq/jmqi/remote/api/RemoteHconn.requestDispatchLock
    at com/ibm/mq/jmqi/remote/api/RemoteFAP.MQCTL
    at
    com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.suspendAs
    yncService
    at
    com/ibm/msg/client/wmq/internal/WMQConnection.suspendAsyncServic
    e
    at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close
      (entered lock: java/lang/Object@0x00000007A3FD90A8, entry
    count: 1)
    at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close
    at com/ibm/msg/client/wmq/internal/WMQSession.stop
       (entered lock: java/lang/Object@0x00000007A80C3468, entry
    count: 1)
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.stop
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
    at com/ibm/mq/jms/MQSession.close
    at com/ibm/mq/connector/inbound/ServerSessionImpl.close
    at
    com/ibm/mq/connector/inbound/ServerSessionPoolImpl.sessionReleas
    ed
       (entered lock: java/util/ArrayList@0x00000007A6EE83E8, entry
    count: 1)
    at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.releaseSe
    ssion
    at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.workCompl
    eted
    at com/ibm/ejs/j2c/work/WorkProxy.run
    at com/ibm/ws/util/ThreadPool$Worker.run
    

Local fix

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of the:
    
      - WebSphere MQ V7.1 JCA Resource Adapter
      - WebSphere MQ V7.5 JCA Resource Adapter
      - IBM MQ V8 JCA Resource Adapter
      - IBM MQ V9 JCA Resource Adapter
    
    who are using an Activation Specification to consume messages
    from an MQ Queue or Topic when connected to the queue manager
    either in CLIENT transport (using TCP/IP) or BINDINGs transport,
    where there are repeated failures which occur during the
    internal work associated with processing the message within the
    MDB before the MDB is driven, for example caused by repeated
    network failures which disconnect the application server from
    the queue manager.
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    This problem is broadly similar to that documented under APAR
    IT15559:
    
    http://www.ibm.com/support/docview.wss?uid=swg1IT15559
    
    in that a deadlock occurs between two threads associated with
    the MQ Activation Specification as a result of attempting to
    close a ServerSession, while simultaneously driving new work
    through the Activation Specification.
    
    In IT15559, the ServerSession was being closed by the Pool
    Scavenger Thread as a result of a ServerSession being unused in
    the pool for a time which exceeded the pool timeout value.
    
    The distinction between that APAR and this one is that here, a
    ServerSession is being closed as a result of the following JEE
    method call throwing an exception to indicate a problem had
    occurred in the pre-processing of an MDB's message:
    
    
    javax.resource.spi.endpoint.MessageEndpoint.beforeDelivery(Metho
    d)
    
    For example, if the ServerSession's associated javax.jms.Session
    was no longer connected to the queue manager as a result of a
    network disconnection event, and two-phase commit (XA) was being
    used to run the transaction for the MDB, the application server
    would be unable to start the XA transaction to enlist the queue
    manager and the above method would fail.
    
    In this circumstance, the code design is to dispose the
    ServerSession and associated JMS Session and create a
    replacement to drive future work under the MDB.  The subsequent
    close() call on the ServerSession could result in deadlock with
    the Activation Specification's DispatchThread, if in parallel
    more work was being driven by another message being available on
    the queue.
    

Problem conclusion

  • Similar to design change associated with IT15559, the code has
    been updated such that Activation Specification pool lock is no
    longer held when closing the ServerSession, which circumvents
    the deadlock.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v7.5       7.5.0.8
    v8.0       8.0.0.7
    v9.0 CD    9.0.2
    v9.0 LTS   9.0.0.1
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT18624

  • Reported component name

    WMQ WINDOWS V7

  • Reported component ID

    5724H7220

  • Reported release

    710

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-01-03

  • Closed date

    2017-01-30

  • Last modified date

    2017-06-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WMQ WINDOWS V7

  • Fixed component ID

    5724H7220

Applicable component levels

  • R710 PSY

       UP

[{"Line of Business":{"code":"LOB36","label":"IBM Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]

Document Information

Modified date:
08 March 2021