IV85254: MQ Resource Adapter - listener ports connecting using bindings transport hang when being shutdown

APAR status

• Closed as program error.

Error description

• When using the WMQ 7.1 Resource Adapter (supplied with
  WebSphere Application Server v8.5.5.9), listener ports
  connecting using BINDINGS transport hang when being
  shut down.
  
  The WebSphere Application Server SystemOut.log shows:
  
  [5/8/16 1:25:57:383 CDT] 000000a2 ThreadMonitor W  WSVR0605W:
  Thread "MessageListenerThreadPool : 3" (000000f5) has been
  active for 622751 milliseconds and may be hung. There is/are
  1 thread(s) in total in the server that may be hung.
   at
  com.ibm.msg.client.wmq.internal.WMQSession
  .loadMessageReference (WMQSession.java:1207)
  
  A Javacore taken at the time of the hang shows the following
  four threads of interest:
  
  ****************************************************************
  *******
  Thread 1:
  ------------
  "Jmqi AsyncConsume Thread. tid: <number>
  Waiting on: java/util/Vector@0x00000000D1170F98 Owned by:
  <unowned>
  
  Java callstack:
  at  java/lang/Object.wait(Native Method)
  at java/lang/Object.wait(Object.java:172(Compiled Code))
  at com/ibm/ejs/jms/listener/ServerSessionPool.
        getServerSession(ServerSessionPool.java:452(Compiled
  Code))
      (entered lock: java/util/Vector@0x00000000D1170F98, entry
  count: 1)
  at com/ibm/msg/client/jms/internal/
        JmsConnectionConsumerImpl$JmsMessageReferenceHandlerImpl.
  	  endDeliverInternal
  	  (JmsConnectionConsumerImpl.java:338(Compiled Code))
  at com/ibm/msg/client/jms/internal/
        JmsConnectionConsumerImpl$JmsMessageReferenceHandlerImpl.
  	  handleMessageReference
  	  (JmsConnectionConsumerImpl.java:456(Compiled Code))
  at com/ibm/msg/client/jms/internal/
        JmsConnectionImpl$JmsProviderMessageRefHandler.
  	  handleMessageReference
  	  (JmsConnectionImpl.java:1593(Compiled Code))
  at com/ibm/msg/client/wmq/internal/
        WMQConnectionBrowser$WMQConnectionBrowserShadow.
  	  consumer(WMQConnectionBrowser.java:872(Compiled Code))
  at com/ibm/mq/jmqi/local/internal/LocalProxyConsumer.
        jmqiConsumerMethod(LocalProxyConsumer.java:222(Compiled
  Code))
  
  
  ****************************************************************
  *******
  Thread 2:
  ----------
  "Thread-<number>"
  Waiting on: com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38
  Owned by: "JmqiWorkerThread"
  
  Java callstack:
  at java/lang/Object.wait(Native Method)
  at java/lang/Object.wait(Object.java:172(Compiled Code))
  at com/ibm/mq/jmqi/JmqiWorkerThread.
        syncExec(JmqiWorkerThread.java:179(Compiled Code))
     (entered lock:
  com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38,
        entry count: 1)
     (entered lock: java/lang/Object@0x00000000D117C3D8, entry
  count: 1)
  at com/ibm/mq/jmqi/local/LocalHconn.syncExec
       (LocalHconn.java:501(Compiled Code))
  at com/ibm/mq/jmqi/local/LocalMQ.MQCTL(LocalMQ.java:2933)
  at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
  
  suspendAsyncService(WMQConsumerOwnerShadow.java:448(Compiled
  Code))
  at com/ibm/msg/client/wmq/internal/WMQConnection.
         suspendAsyncService(WMQConnection.java:2235)
  at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.
         close(WMQConsumerShadow.java:325)
     (entered lock: java/lang/Object@0x00000000D15088A0, entry
  count: 1)
  at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.
         close(WMQConsumerShadow.java:279)
  at com/ibm/msg/client/wmq/internal/
         WMQSession.stop(WMQSession.java:1544(Compiled Code))
     (entered lock: java/lang/Object@0x00000000D15082E0, entry
  count: 1)
  at com/ibm/msg/client/jms/internal/JmsSessionImpl.
         stop(JmsSessionImpl.java:2227(Compiled Code))
  at com/ibm/msg/client/jms/internal/JmsSessionImpl.
         stop(JmsSessionImpl.java:2212(Compiled Code))
  at com/ibm/msg/client/jms/internal/JmsConnectionImpl.
         stop(JmsConnectionImpl.java:928(Compiled Code))
     (entered lock:
         com/ibm/msg/client/jms/internal/State@0x00000000D11F4198,
  	   entry count: 1)
  at
  com/ibm/mq/jms/MQConnection.stop(MQConnection.java:459(Compiled
  Code))
  at
  com/ibm/ejs/jms/JMSConnectionHandle.stop(JMSConnectionHandle.jav
  a:774)
     (entered lock: java/lang/String@0x00000000D11737C8, entry
  count: 1)
  at com/ibm/ejs/jms/listener/MDBListenerImpl$2.
         run(MDBListenerImpl.java:1182)
  
  ****************************************************************
  *******
  Thread 3:
  -----------
  "JmqiWorkerThread"
  Java callstack:
  at com/ibm/mq/jmqi/local/internal/base/Native.MQCTL(Native
  Method)
  at com/ibm/mq/jmqi/local/LocalMQ$10.run(LocalMQ.java:2948)
  at
  com/ibm/mq/jmqi/JmqiWorkerThread.run(JmqiWorkerThread.java:245)
     (entered lock:
  com/ibm/mq/jmqi/JmqiWorkerThread@0x00000000D1170F38,
         entry count: 1)
  at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.
         runTask(WorkQueueItem.java:209)
  at
  com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
         runItem(SimpleWorkQueueItem.java:100)
  at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.
         run(WorkQueueItem.java:224)
  at
  com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
  le.
          run(WorkQueueManagerImpl.java:550)
  at java/lang/Thread.run(Thread.java:809)
  
  ****************************************************************
  *******
  Thread 4
  -----------
  "MessageListenerThreadPool : <number"
  Blocked on: java/lang/Object@0x00000000D15082E0
  Owned by: "<number>"
  
  Java callstack:
  at com/ibm/msg/client/wmq/internal/WMQSession.
        loadMessageReference(WMQSession.java:1207(Compiled Code))
  at com/ibm/msg/client/jms/internal/JmsSessionImpl.
        consume(JmsSessionImpl.java:3268(Compiled Code))
  at com/ibm/msg/client/jms/internal/JmsSessionImpl.
        run(JmsSessionImpl.java:2903(Compiled Code))
  at com/ibm/msg/client/jms/internal/JmsXAQueueSessionImpl$1.
        run(JmsXAQueueSessionImpl.java:1007(Compiled Code))
  at com/ibm/mq/jms/MQSession.run(MQSession.java:958(Compiled
  Code))
  at com/ibm/ejs/jms/JMSSessionHandle.
         run(JMSSessionHandle.java:1056(Compiled Code))
  at com/ibm/ejs/jms/listener/ServerSession.
         connectionConsumerOnMessage
  	   (ServerSession.java:1090(Compiled Code))
  at com/ibm/ejs/jms/listener/ServerSession.
         onMessage(ServerSession.java:760(Compiled Code))
  at com/ibm/ejs/jms/listener/ServerSession.
         dispatch(ServerSession.java:726(Compiled Code))
  at sun/reflect/GeneratedMethodAccessor73.
         invoke(Bytecode PC:32(Compiled Code))
  at sun/reflect/DelegatingMethodAccessorImpl.
         invoke(DelegatingMethodAccessorImpl.java:56(Compiled
  Code))
  at java/lang/reflect/Method.invoke(Method.java:620(Compiled
  Code))
  at com/ibm/ejs/jms/listener/ServerSessionDispatcher.
         dispatch(ServerSessionDispatcher.java:47(Compiled Code))
  at com/ibm/ejs/container/MDBWrapper.
         onMessage(MDBWrapper.java:100(Compiled Code))
  at com/ibm/ejs/container/MDBWrapper.
         onMessage(MDBWrapper.java:132(Compiled Code))
  at com/ibm/ejs/jms/listener/
         ServerSession.run(ServerSession.java:581(Compiled Code))
  at com/ibm/ws/util/
         ThreadPool$Worker.run(ThreadPool.java:1881(Compiled
  Code))
  
  ****************************************************************
  *******
  

Local fix

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of:
  
  - The WebSphere Application Server V8.5.5.9 WebSphere MQ
  messaging provider
  
  who have message-driven bean applications that have been
  deployed against a a Listener Port which has:
  
  - Been configured to connect to a queue manager using the
  BINDINGS transport.
  - The Maximum sessions property set to the value 1.
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  When a Listener Port that has been configured to connect to a
  queue manager using the BINDINGS transport starts up, it will
  register a callback with the queue manager on the queue or topic
  it has been configured to monitor. When a message arrives on the
  JMS Destination, the queue manager calls back to the main
  Listener Port thread, passing it a message token representing
  the message. The main Listener Port thread will then:
  
  - Get a Server Session from it's Server Session Pool.
  - Construct an internal message reference object, that
  represents the WebSphere MQ message that has been detected.
  - Load the message reference into the Server Session.
  - Run the Server Session on a new thread.
  
  The Server Session uses the information in the message reference
  to get the message, and pass it to an instance of a
  message-driven bean application for processing. When the message
  has been processed, and the message-driven bean's onMessage()
  method has completed, the Server Session returns itself to the
  Server Session Pool where it is available for reuse.
  
  
  When the hang reported by this APAR occurred, the following
  sequence of events had taken place:
  
  Thread 1:
  -----------
  - The queue manager finds a message on the queue being monitored
  by the Listener Port, and calls back to the main Listener Port
  thread passing it a message token for the message.
  - The Listener Port thread constructs a message reference
  containing that message token, gets a Server Session from it's
  Server Session Pool and then starts that Server Session.
  - The main Listener Port thread then waits for the queue manager
  to call it back again.
  
  - A few milliseconds later, the queue manager finds a second
  message on the queue, and calls back to the main Listener Port
  thread again.
  - The Listener Port thread constructs a message reference for
  this message, and tries to get a Server Session from the Server
  Session Pool. However, only Server Session in the Server Session
  Pool is in use, so Thread 1 becomes blocked.
  
  
  Thread 2:
  ---------
  - At this point, a request is made to stop the application
  server. While processing this request, the Listener Port starts
  shutting down.
  - As part of the processing of this method, the thread takes an
  internal lock.
  - The thread then issues a WebSphere MQ MQCTL API call, passing
  in the option MQOP_SUSPEND, to suspend the call back that has
  been registered with the queue manager for the Listener Port.
  - Because the Listener Port is connecting to the queue manager
  using the BINDINGS transport, a new worker thread is started to
  perform the MQCTL call.
  - This thread then waits for the worker thread to complete.
  
  Thread 3:
  ----------
  - The worker thread starts, and calls into some native queue
  manager code to perform the actual MQCTL processing.
  - This thread has to wait for any call backs that are currently
  in progress to complete before the MQCTL API call can return.
  - There is one outstanding call back, which is currently running
  on Thread 1. As a result, Thread 3 has to wait for Thread 1 to
  finish before it can continue.
  
  
  Thread 4:
  ----------
  - While Thread 2 was performing it's processing, the JMS Server
  Session started by Thread 1 finally starts.
  - This thread starts processing the message reference, and then
  becomes blocked waiting for the lock held by Thread 2.
  
  
  At this point, a deadlock occurs:
  
  - Thread 1 is waiting for the Server Session being used by
  Thread 4 to be put back in the Server Session Pool.
  - Thread 4 is waiting for an internal lock held by Thread 2.
  - Thread 2 is waiting for the worker thread Thread 3 to finish
  processing the MQCTL API call.
  - Thread 3 is waiting for the call back running on Thread 1 to
  complete.
  

Problem conclusion

• The WebSphere MQ Resource Adapter (the component of WebSphere
  Application Server that handles all of the communication with
  WebSphere MQ) has been updated to ensure that, when a Listener
  Port that is connecting to a queue manager using the BINDINGS
  transport is stopped, any message references which have not yet
  been processed will be discarded. This allows the Listener Port
  to shut down as expected, and prevents the deadlock mentioned in
  this APAR from occurring.
  
  The messages mentioned within the message references will remain
  on the destination that the Listener Port was monitoring, and
  will be available to be processed when the Listener Port is
  restarted.
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v7.1       7.1.0.9
  v7.5       7.5.0.8
  v8.0       8.0.0.6
  v9.0 CD    9.0.2
  v9.0 LTS   9.0.0.1
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WMQ LIN X86 V7

• Fixed component ID

  5724H7224

Applicable component levels