IBM Support

IV85254: MQ Resource Adapter - listener ports connecting using bindings transport hang when being shutdown

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • When using the WMQ 7.1 Resource Adapter (supplied with
    WebSphere Application Server v8.5.5.9), listener ports
    connecting using BINDINGS transport hang when being
    shut down.
    
    The WebSphere Application Server SystemOut.log shows:
    
    [5/8/16 1:25:57:383 CDT] 000000a2 ThreadMonitor W  WSVR0605W:
    Thread "MessageListenerThreadPool : 3" (000000f5) has been
    active for 622751 milliseconds and may be hung. There is/are
    1 thread(s) in total in the server that may be hung.
     at
    com.ibm.msg.client.wmq.internal.WMQSession
    .loadMessageReference (WMQSession.java:1207)
    
    A Javacore taken at the time of the hang shows the following
    four threads of interest:
    
    ****************************************************************
    *******
    Thread 1:
    ------------
    "Jmqi AsyncConsume Thread. tid: <number>
    Waiting on: java/util/Vector@0x00000000D1170F98 Owned by:
    <unowned>
    
    Java callstack:
    at  java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:172(Compiled Code))
    at com/ibm/ejs/jms/listener/ServerSessionPool.
          getServerSession(ServerSessionPool.java:452(Compiled
    Code))
        (entered lock: java/util/Vector@0x00000000D1170F98, entry
    count: 1)
    at com/ibm/msg/client/jms/internal/
          JmsConnectionConsumerImpl$JmsMessageReferenceHandlerImpl.
    	  endDeliverInternal
    	  (JmsConnectionConsumerImpl.java:338(Compiled Code))
    at com/ibm/msg/client/jms/internal/
          JmsConnectionConsumerImpl$JmsMessageReferenceHandlerImpl.
    	  handleMessageReference
    	  (JmsConnectionConsumerImpl.java:456(Compiled Code))
    at com/ibm/msg/client/jms/internal/
          JmsConnectionImpl$JmsProviderMessageRefHandler.
    	  handleMessageReference
    	  (JmsConnectionImpl.java:1593(Compiled Code))
    at com/ibm/msg/client/wmq/internal/
          WMQConnectionBrowser$WMQConnectionBrowserShadow.
    	  consumer(WMQConnectionBrowser.java:872(Compiled Code))
    at com/ibm/mq/jmqi/local/internal/LocalProxyConsumer.
          jmqiConsumerMethod(LocalProxyConsumer.java:222(Compiled
    Code))
    
    
    ****************************************************************
    *******
    Thread 2:
    ----------
    "Thread-<number>"
    Waiting on: com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38
    Owned by: "JmqiWorkerThread"
    
    Java callstack:
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:172(Compiled Code))
    at com/ibm/mq/jmqi/JmqiWorkerThread.
          syncExec(JmqiWorkerThread.java:179(Compiled Code))
       (entered lock:
    com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38,
          entry count: 1)
       (entered lock: java/lang/Object@0x00000000D117C3D8, entry
    count: 1)
    at com/ibm/mq/jmqi/local/LocalHconn.syncExec
         (LocalHconn.java:501(Compiled Code))
    at com/ibm/mq/jmqi/local/LocalMQ.MQCTL(LocalMQ.java:2933)
    at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
    
    suspendAsyncService(WMQConsumerOwnerShadow.java:448(Compiled
    Code))
    at com/ibm/msg/client/wmq/internal/WMQConnection.
           suspendAsyncService(WMQConnection.java:2235)
    at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.
           close(WMQConsumerShadow.java:325)
       (entered lock: java/lang/Object@0x00000000D15088A0, entry
    count: 1)
    at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.
           close(WMQConsumerShadow.java:279)
    at com/ibm/msg/client/wmq/internal/
           WMQSession.stop(WMQSession.java:1544(Compiled Code))
       (entered lock: java/lang/Object@0x00000000D15082E0, entry
    count: 1)
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.
           stop(JmsSessionImpl.java:2227(Compiled Code))
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.
           stop(JmsSessionImpl.java:2212(Compiled Code))
    at com/ibm/msg/client/jms/internal/JmsConnectionImpl.
           stop(JmsConnectionImpl.java:928(Compiled Code))
       (entered lock:
           com/ibm/msg/client/jms/internal/State@0x00000000D11F4198,
    	   entry count: 1)
    at
    com/ibm/mq/jms/MQConnection.stop(MQConnection.java:459(Compiled
    Code))
    at
    com/ibm/ejs/jms/JMSConnectionHandle.stop(JMSConnectionHandle.jav
    a:774)
       (entered lock: java/lang/String@0x00000000D11737C8, entry
    count: 1)
    at com/ibm/ejs/jms/listener/MDBListenerImpl$2.
           run(MDBListenerImpl.java:1182)
    
    ****************************************************************
    *******
    Thread 3:
    -----------
    "JmqiWorkerThread"
    Java callstack:
    at com/ibm/mq/jmqi/local/internal/base/Native.MQCTL(Native
    Method)
    at com/ibm/mq/jmqi/local/LocalMQ$10.run(LocalMQ.java:2948)
    at
    com/ibm/mq/jmqi/JmqiWorkerThread.run(JmqiWorkerThread.java:245)
       (entered lock:
    com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38,
           entry count: 1)
    at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.
           runTask(WorkQueueItem.java:209)
    at
    com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
           runItem(SimpleWorkQueueItem.java:100)
    at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.
           run(WorkQueueItem.java:224)
    at
    com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
    le.
            run(WorkQueueManagerImpl.java:550)
    at java/lang/Thread.run(Thread.java:809)
    
    ****************************************************************
    *******
    Thread 4
    -----------
    "MessageListenerThreadPool : <number"
    Blocked on: java/lang/Object@0x00000000D15082E0
    Owned by: "<number>"
    
    Java callstack:
    at com/ibm/msg/client/wmq/internal/WMQSession.
          loadMessageReference(WMQSession.java:1207(Compiled Code))
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.
          consume(JmsSessionImpl.java:3268(Compiled Code))
    at com/ibm/msg/client/jms/internal/JmsSessionImpl.
          run(JmsSessionImpl.java:2903(Compiled Code))
    at com/ibm/msg/client/jms/internal/JmsXAQueueSessionImpl$1.
          run(JmsXAQueueSessionImpl.java:1007(Compiled Code))
    at com/ibm/mq/jms/MQSession.run(MQSession.java:958(Compiled
    Code))
    at com/ibm/ejs/jms/JMSSessionHandle.
           run(JMSSessionHandle.java:1056(Compiled Code))
    at com/ibm/ejs/jms/listener/ServerSession.
           connectionConsumerOnMessage
    	   (ServerSession.java:1090(Compiled Code))
    at com/ibm/ejs/jms/listener/ServerSession.
           onMessage(ServerSession.java:760(Compiled Code))
    at com/ibm/ejs/jms/listener/ServerSession.
           dispatch(ServerSession.java:726(Compiled Code))
    at sun/reflect/GeneratedMethodAccessor73.
           invoke(Bytecode PC:32(Compiled Code))
    at sun/reflect/DelegatingMethodAccessorImpl.
           invoke(DelegatingMethodAccessorImpl.java:56(Compiled
    Code))
    at java/lang/reflect/Method.invoke(Method.java:620(Compiled
    Code))
    at com/ibm/ejs/jms/listener/ServerSessionDispatcher.
           dispatch(ServerSessionDispatcher.java:47(Compiled Code))
    at com/ibm/ejs/container/MDBWrapper.
           onMessage(MDBWrapper.java:100(Compiled Code))
    at com/ibm/ejs/container/MDBWrapper.
           onMessage(MDBWrapper.java:132(Compiled Code))
    at com/ibm/ejs/jms/listener/
           ServerSession.run(ServerSession.java:581(Compiled Code))
    at com/ibm/ws/util/
           ThreadPool$Worker.run(ThreadPool.java:1881(Compiled
    Code))
    
    ****************************************************************
    *******
    

Local fix

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of:
    
    - The WebSphere Application Server V8.5.5.9 WebSphere MQ
    messaging provider
    
    who have message-driven bean applications that have been
    deployed against a a Listener Port which has:
    
    - Been configured to connect to a queue manager using the
    BINDINGS transport.
    - The Maximum sessions property set to the value 1.
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    When a Listener Port that has been configured to connect to a
    queue manager using the BINDINGS transport starts up, it will
    register a callback with the queue manager on the queue or topic
    it has been configured to monitor. When a message arrives on the
    JMS Destination, the queue manager calls back to the main
    Listener Port thread, passing it a message token representing
    the message. The main Listener Port thread will then:
    
    - Get a Server Session from it's Server Session Pool.
    - Construct an internal message reference object, that
    represents the WebSphere MQ message that has been detected.
    - Load the message reference into the Server Session.
    - Run the Server Session on a new thread.
    
    The Server Session uses the information in the message reference
    to get the message, and pass it to an instance of a
    message-driven bean application for processing. When the message
    has been processed, and the message-driven bean's onMessage()
    method has completed, the Server Session returns itself to the
    Server Session Pool where it is available for reuse.
    
    
    When the hang reported by this APAR occurred, the following
    sequence of events had taken place:
    
    Thread 1:
    -----------
    - The queue manager finds a message on the queue being monitored
    by the Listener Port, and calls back to the main Listener Port
    thread passing it a message token for the message.
    - The Listener Port thread constructs a message reference
    containing that message token, gets a Server Session from it's
    Server Session Pool and then starts that Server Session.
    - The main Listener Port thread then waits for the queue manager
    to call it back again.
    
    - A few milliseconds later, the queue manager finds a second
    message on the queue, and calls back to the main Listener Port
    thread again.
    - The Listener Port thread constructs a message reference for
    this message, and tries to get a Server Session from the Server
    Session Pool. However, only Server Session in the Server Session
    Pool is in use, so Thread 1 becomes blocked.
    
    
    Thread 2:
    ---------
    - At this point, a request is made to stop the application
    server. While processing this request, the Listener Port starts
    shutting down.
    - As part of the processing of this method, the thread takes an
    internal lock.
    - The thread then issues a WebSphere MQ MQCTL API call, passing
    in the option MQOP_SUSPEND, to suspend the call back that has
    been registered with the queue manager for the Listener Port.
    - Because the Listener Port is connecting to the queue manager
    using the BINDINGS transport, a new worker thread is started to
    perform the MQCTL call.
    - This thread then waits for the worker thread to complete.
    
    Thread 3:
    ----------
    - The worker thread starts, and calls into some native queue
    manager code to perform the actual MQCTL processing.
    - This thread has to wait for any call backs that are currently
    in progress to complete before the MQCTL API call can return.
    - There is one outstanding call back, which is currently running
    on Thread 1. As a result, Thread 3 has to wait for Thread 1 to
    finish before it can continue.
    
    
    Thread 4:
    ----------
    - While Thread 2 was performing it's processing, the JMS Server
    Session started by Thread 1 finally starts.
    - This thread starts processing the message reference, and then
    becomes blocked waiting for the lock held by Thread 2.
    
    
    At this point, a deadlock occurs:
    
    - Thread 1 is waiting for the Server Session being used by
    Thread 4 to be put back in the Server Session Pool.
    - Thread 4 is waiting for an internal lock held by Thread 2.
    - Thread 2 is waiting for the worker thread Thread 3 to finish
    processing the MQCTL API call.
    - Thread 3 is waiting for the call back running on Thread 1 to
    complete.
    

Problem conclusion

  • The WebSphere MQ Resource Adapter (the component of WebSphere
    Application Server that handles all of the communication with
    WebSphere MQ) has been updated to ensure that, when a Listener
    Port that is connecting to a queue manager using the BINDINGS
    transport is stopped, any message references which have not yet
    been processed will be discarded. This allows the Listener Port
    to shut down as expected, and prevents the deadlock mentioned in
    this APAR from occurring.
    
    The messages mentioned within the message references will remain
    on the destination that the Listener Port was monitoring, and
    will be available to be processed when the Listener Port is
    restarted.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v7.1       7.1.0.9
    v7.5       7.5.0.8
    v8.0       8.0.0.6
    v9.0 CD    9.0.2
    v9.0 LTS   9.0.0.1
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IV85254

  • Reported component name

    WMQ LIN X86 V7

  • Reported component ID

    5724H7224

  • Reported release

    710

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-05-27

  • Closed date

    2016-11-25

  • Last modified date

    2017-08-02

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WMQ LIN X86 V7

  • Fixed component ID

    5724H7224

Applicable component levels

[{"Line of Business":{"code":"LOB36","label":"IBM Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]

Document Information

Modified date:
08 March 2021