A fix is available
APAR status
Closed as program error.
Error description
A WebSphere MQ Resource Adapter (WMQ-RA) is being used within a JBoss environment to drive an Activation Specification, consuming messages from a queue where the WMQ-RA is connected to the queue manager using the BINDINGs mode transport. The Activation Specification is observed to stop consuming messages from the queue and driving the application's MDB, even though there are messages available for consumption. A thread dump (Javacore) taken of the JVM when in this state shows the following two threads which are not moving over time: Thread 1: java.lang.Thread.State: RUNNABLE at com.ibm.mq.jmqi.local.internal.base.Native.MQCTL at com.ibm.mq.jmqi.local.LocalMQ.MQCTL at com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs yncService at com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic e at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close - locked <0x0000000733373260> (a java.lang.Object) at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQSession.stop - locked <0x00000007333731e8> (a java.lang.Object) at com.ibm.msg.client.jms.internal.JmsSessionImpl.stop at com.ibm.msg.client.jms.internal.JmsSessionImpl.close at com.ibm.msg.client.jms.internal.JmsSessionImpl.close at com.ibm.mq.jms.MQSession.close at com.ibm.mq.connector.inbound.ServerSessionImpl.close at com.ibm.mq.connector.inbound.ServerSessionPoolImpl.poolCleanup - locked <0x00000007327895e8> (a java.util.ArrayList) at com.ibm.mq.connector.inbound.PoolScavengerThread.run Thread 2: java.lang.Thread.State: BLOCKED (on object monitor) at com.ibm.mq.connector.inbound.ServerSessionPoolImpl.getServerSess ion - waiting to lock <0x00000007327895e8> (a java.util.ArrayList) at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.endDeliverInternal at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.handleMessageReference at com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes sageRefHandler.handleMessageReference at com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti onBrowserShadow.consumer at com.ibm.mq.jmqi.local.internal.LocalProxyConsumer.jmqiConsumerMe thod The JVM process must be killed in order to shut down the application server once it has reached this state.
Local fix
Increase the value of the WebSphere MQ JCA Resource Adapter inbound property "poolTimeout" from the default value of 300000 ms (5 minutes), to the maximum 'int' value: 2147483647 ms which is equivalent to the time value of 24.9 days. This minimises the frequency at which the problem might occur but does not fix the issue in its entirety. Using the workaround may be useful until an interim fix can be applied to the system. When using the WebSphere Application Server (WSAS) Administration Console, this property is accessible via two paths. The first is: Resources --> Resource Adapters --> Resource adapters --> WebSphere MQ Resource Adapter (for the scope at which the Activation Specification is defined) --> J2C activation specifications --> <Activation Specification Name> --> J2C activation specification custom properties --> the "poolTimeout" property is normally listed on the second page of properties The second path is: Resources --> JMS --> Activation specifications --> <Activation Specification Name> --> Custom properties --> The "poolTimeout" property is normally listed on the second page of properties
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the: - WebSphere MQ V7.1 JCA Resource Adapter - WebSphere MQ V7.5 JCA Resource Adapter - IBM MQ V8 JCA Resource Adapter - IBM MQ V9 JCA Resource Adapter who are using an Activation Specification to consume messages from a queue with varying work loads on the queue, such that ServerSessions will be created during busier periods to drive the MDB, but then closed during quieter periods when there is insufficient workload such that the ServerSession remains idle for longer than the WMQ-RA 'poolTimeout' value, which defaults to 5 minutes. This includes users of WebSphere Application Server V8.5 and V9. Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: For 'inbound' messaging (where an Activation Specification is used to consume messages from a queue and drive MDBs), the WebSphere MQ Resource Adapter maintains a pool of ServerSession objects. These ServerSessions are the objects which are used to get a specific message from a queue and drive an MDB application's: javax.jms.MessageListener.onMessage(javax.jms.Message)" method. As the workload increases with more messages arriving on the source queue, the number of ServerSessions associated with a particular Activation Specification are increased up to the limit specified by the WMQ-RA property "maxPoolDepth", which defaults to the value of "10". If the workload then drops off, some of these ServerSessions may sit idle for a period of time. Every 5 minutes, a "PoolScavengerThread" checks the idle time of each of the ServerSessions in the pool, and if they have exceeded the "poolTimeout" property (which defaults to 300,000ms, or 5 minutes), the ServerSession is closed releasing resources both within the JVM and within the queue manager. If this PoolScavengerThread process should attempt to close a ServerSession from the pool while a message becomes available to process within the MDB, there is the possibility that a deadlock can occur where the PoolScavengerThread is blocked waiting for the MDB processing to complete, and the MDB is blocked trying to get a ServerSession from the pool. The Java stacks which are seen when this happens vary depending on which transport mode is being used. If a BINDINGs mode transport is in operation (that is where the WMQ-RA communicates with the queue manager using Interprocess Communication (IPC)), the two relevant thread stacks are as listed above, with the PoolScavengerThread within the native code method: com.ibm.mq.jmqi.local.internal.base.Native.MQCTL and the MDB thread blocked trying to get a ServerSession from the pool: com.ibm.mq.connector.inbound.ServerSessionPoolImpl.getServerSess ion When CLIENT mode transport is being used, that is to say the WMQ-RA is communicating with the queue manager over the TCP layer, then the relevant thread stacks which are seen are: PoolScavengerThread: java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait at java.lang.Object.wait at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire - locked <0xfffffd7fb1ce6730> (a com.ibm.mq.jmqi.remote.api.RemoteHconn$DispatchLock) at com.ibm.mq.jmqi.remote.util.ReentrantMutex.acquire - locked <0xfffffd7fb1ce6730> (a com.ibm.mq.jmqi.remote.api.RemoteHconn$DispatchLock) at com.ibm.mq.jmqi.remote.api.RemoteHconn.requestDispatchLock at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQCTL at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.MQCTL at com.ibm.msg.client.wmq.internal.WMQConsumerOwnerShadow.suspendAs yncService at com.ibm.msg.client.wmq.internal.WMQConnection.suspendAsyncServic e at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQConsumerShadow.close at com.ibm.msg.client.wmq.internal.WMQSession.stop - locked <0xfffffd7fb4450418> (a java.lang.Object) at com.ibm.msg.client.jms.internal.JmsSessionImpl.stop at com.ibm.msg.client.jms.internal.JmsSessionImpl.close at com.ibm.msg.client.jms.internal.JmsSessionImpl.close at com.ibm.mq.jms.MQSession.close at com.ibm.mq.connector.inbound.ServerSessionImpl.close at com.ibm.mq.connector.inbound.ServerSessionPoolImpl.poolCleanup - locked <0xfffffd7fb1cde548> (a java.util.ArrayList) at com.ibm.mq.connector.inbound.PoolScavengerThread.run DispatchThread: java.lang.Thread.State: BLOCKED (on object monitor) at com.ibm.mq.connector.inbound.ServerSessionPoolImpl.getServerSess ion - waiting to lock <0xfffffd7fb1cde548> (a java.util.ArrayList) at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.endDeliverInternal at com.ibm.msg.client.jms.internal.JmsConnectionConsumerImpl$JmsMes sageReferenceHandlerImpl.handleMessageReference at com.ibm.msg.client.jms.internal.JmsConnectionImpl$JmsProviderMes sageRefHandler.handleMessageReference at com.ibm.msg.client.wmq.internal.WMQConnectionBrowser$WMQConnecti onBrowserShadow.consumer at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.driveConsumer at com.ibm.mq.jmqi.remote.impl.RemoteProxyQueue.deliverMsgs at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgsReco nnectable at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.deliverMsgs at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas k at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem. runItem at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.run WorkQueueItem at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManage rImplementation$ThreadPoolWorker.run There are two other common Java call stacks that indicate this problem has been encountered and can be seen alongside the above two threads in the same Javacore / thread dump file. These are: Thread 3: java.lang.Thread.State: BLOCKED at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.sessionReleas ed at com/ibm/mq/connector/inbound/MessageEndpointDeployment.releaseSe ssion at com/ibm/mq/connector/inbound/MessageEndpointDeployment.workCompl eted at com/ibm/ejs/j2c/work/WorkProxy.run Thread 4: java.lang.Thread.State: BLOCKED at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.setIsClosing at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv eryASF at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop at com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass LoaderContext at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint at com/ibm/ejs/j2c/RAWrapperImpl.deactivateEndpoint at com/ibm/ejs/j2c/RALifeCycleManagerImpl.deactivateEndPoint The blocked threads may also be reported as hung in the WebSphere Application Server SystemOut.log file. For example: ThreadMonitor W WSVR0605W: Thread "Thread 3" (0000cd2a) has been active for 760129 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
Problem conclusion
The code has been updated such that the PoolScavengerThread no longer holds a lock on the ServerSession pool when closing ServerSessions, which avoids the deadlock in both the CLIENT and BINDINGs transport modes. Note that this second CLIENT transport issue was already documented and resolved in IBM MQ v8.0.0.4 by APAR IT11876. This APAR (IT15559) improves on the change implemented under APAR IT11876 by changing the pool locking model. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v7.1 7.1.0.8 v7.5 7.5.0.8 v8.0 8.0.0.6 v9.0 CD 9.0.1 v9.0 LTS 9.0.0.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT15559
Reported component name
WMQ BASE MULTIP
Reported component ID
5724H7241
Reported release
750
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-06-02
Closed date
2016-06-21
Last modified date
2017-06-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ BASE MULTIP
Fixed component ID
5724H7241
Applicable component levels
R750 PSY
UP
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.5"}]
Document Information
Modified date:
26 August 2021