IBM Support

IT15559: Activation specification deadlocks when pool scavenger thread closes a server session

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A WebSphere MQ Resource Adapter (WMQ-RA) is being used within a
    JBoss environment to drive an Activation Specification,
    consuming messages from a queue where the WMQ-RA is connected to
    the queue manager using the BINDINGs mode transport.
    
    The Activation Specification is observed to stop consuming
    messages from the queue and driving the application's MDB, even
    though there are messages available for consumption.
    
    A thread dump (Javacore) taken of the JVM when in this state
    shows the following two threads which are not moving over time:
    
    Thread 1:
    java.lang.Thread.State: RUNNABLE
      at com.ibm.mq.jmqi.local.internal.base.Native.MQCTL
      at com.ibm.mq.jmqi.local.LocalMQ.MQCTL
      at
    com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs
    yncService
      at
    com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic
    e
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
        - locked <0x0000000733373260> (a java.lang.Object)
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at com.ibm.msg.client.wmq.internal.WMQSession.stop
        - locked <0x00000007333731e8> (a java.lang.Object)
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.stop
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.close
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.close
      at com.ibm.mq.jms.MQSession.close
      at com.ibm.mq.connector.inbound.ServerSessionImpl.close
      at
    com.ibm.mq.connector.inbound.ServerSessionPoolImpl.poolCleanup
        - locked <0x00000007327895e8> (a java.util.ArrayList)
      at com.ibm.mq.connector.inbound.PoolScavengerThread.run
    
    Thread 2:
    java.lang.Thread.State: BLOCKED (on object monitor)
      at
    com.ibm.mq.connector.inbound.ServerSessionPoolImpl.getServerSess
    ion
      - waiting to lock <0x00000007327895e8> (a java.util.ArrayList)
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.endDeliverInternal
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.handleMessageReference
      at
    com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes
    sageRefHandler.handleMessageReference
      at
    com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti
    onBrowserShadow.consumer
      at
    com.ibm.mq.jmqi.local.internal.LocalProxyConsumer.jmqiConsumerMe
    thod
    
    
    The JVM process must be killed in order to shut down the
    application server once it has reached this state.
    

Local fix

  • Increase the value of the WebSphere MQ JCA Resource Adapter
    inbound property "poolTimeout" from the default value of 300000
    ms (5 minutes), to the maximum 'int' value:
    
        2147483647 ms
    
    which is equivalent to the time value of 24.9 days.   This
    minimises the frequency at which the problem might occur but
    does not fix the issue in its entirety.  Using the workaround
    may be useful until an interim fix can be applied to the system.
    
    When using the WebSphere Application Server (WSAS)
    Administration Console, this property is accessible via two
    paths.  The first is:
    
      Resources
      --> Resource Adapters
        --> Resource adapters
          --> WebSphere MQ Resource Adapter
                 (for the scope at which the Activation
    Specification is defined)
            --> J2C activation specifications
              --> <Activation Specification Name>
                --> J2C activation specification custom properties
                  --> the "poolTimeout" property is normally listed
    on the second page of properties
    
    
    The second path is:
    
      Resources
      --> JMS
        --> Activation specifications
          --> <Activation Specification Name>
            --> Custom properties
              --> The "poolTimeout" property is normally listed on
    the second page of properties
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of the:
    
      - WebSphere MQ V7.1 JCA Resource Adapter
      - WebSphere MQ V7.5 JCA Resource Adapter
      - IBM MQ V8 JCA Resource Adapter
      - IBM MQ V9 JCA Resource Adapter
    
    who are using an Activation Specification to consume messages
    from a queue with varying work loads on the queue, such that
    ServerSessions will be created during busier periods to drive
    the MDB, but then closed during quieter periods when there is
    insufficient workload such that the ServerSession remains idle
    for longer than the WMQ-RA 'poolTimeout' value, which defaults
    to 5 minutes.
    
    This includes users of WebSphere Application Server V8.5 and V9.
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    For 'inbound' messaging (where an Activation Specification is
    used to consume messages from a queue and drive MDBs), the
    WebSphere MQ Resource Adapter maintains a pool of ServerSession
    objects.
    
    These ServerSessions are the objects which are used to get a
    specific message from a queue and drive an MDB application's:
    
       javax.jms.MessageListener.onMessage(javax.jms.Message)"
    
    method.
    
    
    As the workload increases with more messages arriving on the
    source queue, the number of ServerSessions associated with a
    particular Activation Specification are increased up to the
    limit specified by the WMQ-RA property "maxPoolDepth", which
    defaults to the value of "10".
    
    If the workload then drops off, some of these ServerSessions may
    sit idle for a period of time.  Every 5 minutes, a
    "PoolScavengerThread" checks the idle time of each of the
    ServerSessions in the pool, and if they have exceeded the
    "poolTimeout" property (which defaults to 300,000ms, or 5
    minutes), the ServerSession is closed releasing resources both
    within the JVM and within the queue manager.
    
    
    If this PoolScavengerThread process should attempt to close a
    ServerSession from the pool while a message becomes available to
    process within the MDB, there is the possibility that a deadlock
    can occur where the PoolScavengerThread is blocked waiting for
    the MDB processing to complete, and the MDB is blocked trying to
    get a ServerSession from the pool.
    
    The Java stacks which are seen when this happens vary depending
    on which transport mode is being used.  If a BINDINGs mode
    transport is in operation (that is where the WMQ-RA communicates
    with the queue manager using Interprocess Communication (IPC)),
    the two relevant thread stacks are as listed above, with the
    PoolScavengerThread within the native code method:
    
      com.ibm.mq.jmqi.local.internal.base.Native.MQCTL
    
    and the MDB thread blocked trying to get a ServerSession from
    the pool:
    
    
    com.ibm.mq.connector.inbound.ServerSessionPoolImpl.getServerSess
    ion
    
    
    When CLIENT mode transport is being used, that is to say the
    WMQ-RA is communicating with the queue manager over the TCP
    layer, then the relevant thread stacks which are seen are:
    
    
    PoolScavengerThread:
    java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait
      at java.lang.Object.wait
      at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire
      - locked <0xfffffd7fb1ce6730> (a
    com.ibm.mq.jmqi.remote.api.RemoteHconn$DispatchLock)
      at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire
      - locked <0xfffffd7fb1ce6730> (a
    com.ibm.mq.jmqi.remote.api.RemoteHconn$DispatchLock)
      at com.ibm.mq.jmqi.remote.api.RemoteHconn.requestDispatchLock
      at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQCTL
      at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.MQCTL
      at
    com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs
    yncService
      at
    com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic
    e
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close
      at com.ibm.msg.client.wmq.internal.WMQSession.stop
      - locked <0xfffffd7fb4450418> (a java.lang.Object)
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.stop
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.close
      at com.ibm.msg.client.jms.internal.JmsSessionImpl.close
      at com.ibm.mq.jms.MQSession.close
      at com.ibm.mq.connector.inbound.ServerSessionImpl.close
      at
    com.ibm.mq.connector.inbound.ServerSessionPoolImpl.poolCleanup
      - locked <0xfffffd7fb1cde548> (a java.util.ArrayList)
      at com.ibm.mq.connector.inbound.PoolScavengerThread.run
    
    
    DispatchThread:
    java.lang.Thread.State: BLOCKED (on object monitor)
      at
    com.ibm.mq.connector.inbound.ServerSessionPoolImpl.getServerSess
    ion
      - waiting to lock <0xfffffd7fb1cde548> (a java.util.ArrayList)
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.endDeliverInternal
      at
    com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes
    sageReferenceHandlerImpl.handleMessageReference
      at
    com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes
    sageRefHandler.handleMessageReference
      at
    com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti
    onBrowserShadow.consumer
      at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.driveConsumer
      at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.deliverMsgs
      at
    com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgsReco
    nnectable
      at
    com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgs
      at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run
      at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas
    k
      at
    com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.
    runItem
      at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run
      at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.run
    WorkQueueItem
      at
    com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManage
    rImplementation$ThreadPoolWorker.run
    
    
    
    There are two other common Java call stacks that indicate this
    problem has been encountered and can be seen alongside the above
    two threads in the same Javacore / thread dump file.  These are:
    
    Thread 3:
    java.lang.Thread.State: BLOCKED
    at
    com/ibm/mq/connector/inbound/ServerSessionPoolImpl.sessionReleas
    ed
    at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.releaseSe
    ssion
    at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.workCompl
    eted
    at com/ibm/ejs/j2c/work/WorkProxy.run
    
    Thread 4:
    java.lang.Thread.State: BLOCKED
    at
    com/ibm/mq/connector/inbound/ServerSessionPoolImpl.setIsClosing
    at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv
    eryASF
    at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop
    at com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation
    at
    com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass
    LoaderContext
    at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint
    at com/ibm/ejs/j2c/RAWrapperImpl.deactivateEndpoint
    at com/ibm/ejs/j2c/RALifeCycleManagerImpl.deactivateEndPoint
    
    
    The blocked threads may also be reported as hung in the
    WebSphere Application Server SystemOut.log file. For example:
    
    ThreadMonitor W   WSVR0605W: Thread "Thread 3" (0000cd2a) has
    been active for 760129 milliseconds and may be hung.  There
    is/are 1 thread(s) in total in the server that may be hung.
    

Problem conclusion

  • The code has been updated such that the PoolScavengerThread no
    longer holds a lock on the ServerSession pool when closing
    ServerSessions, which avoids the deadlock in both the CLIENT and
    BINDINGs transport modes.
    
    Note that this second CLIENT transport issue was already
    documented and resolved in IBM MQ v8.0.0.4 by APAR IT11876.
    This APAR (IT15559) improves on the change implemented under
    APAR IT11876 by changing the pool locking model.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v7.1       7.1.0.8
    v7.5       7.5.0.8
    v8.0       8.0.0.6
    v9.0 CD    9.0.1
    v9.0 LTS   9.0.0.1
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT15559

  • Reported component name

    WMQ BASE MULTIP

  • Reported component ID

    5724H7241

  • Reported release

    750

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    YesHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-06-02

  • Closed date

    2016-06-21

  • Last modified date

    2017-06-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WMQ BASE MULTIP

  • Fixed component ID

    5724H7241

Applicable component levels

  • R750 PSY

       UP

[{"Line of Business":{"code":"LOB36","label":"IBM Automation"},"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.5"}]

Document Information

Modified date:
26 August 2021