APAR status
Closed as program error.
Error description
If an application using the IBM MQ classes for JMS configures a JMS MessageListener to receive messages asynchronously, a deadlock can occur between the MQ Java client and the queue manager. This has been observed when the following conditions occur: a) a poison message is being passed to the asynchronous consume callback function in the MQ classes for JMS from the queue manager, b) that message cannot be requeued to either the backout requeue queue or the dead-letter queue, c) at the same time, a separate application thread attempts to close the JMS MessageConsumer, Session or Connection (or for the JMS 2.0 API, a JMSConsumer or JMSContext) associated with the MessageListener. A Javacore (thread dump) of the application will show the two threads that are deadlocked. The threads below are taken from an MQ classes for JMS application that connects to a queue manager using the BINDINGS transport mode: Thread 1 - Closing a JMS Session: ---------------------------------- "main" J9VMThread:0x0000000000010500, j9thread_t:0x000000001004F540, java/lang/Thread:0x00000000A00372C0, state:R, prio=5 (java/lang/Thread getId:0x1, isDaemon:false) (native thread ID:0x443, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000001) (native stack address range from:0x00000FFF8B610000, to:0x00000FFF8BE10000, size:0x800000) CPU usage total: 4.614949438 secs, current category="Application" Heap bytes allocated since last GC cycle=0 (0x0) Java callstack: at com/ibm/mq/jmqi/local/internal/base/Native.MQCTL (Native Method) at com/ibm/mq/jmqi/local/LocalMQ.MQCTL( LocalMQ.java:3125) at com/ibm/mq/jmqi/monitoring/JmqiInterceptAdapter. MQCTL(JmqiInterceptAdapter.java:333) at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow. controlAsyncService(WMQConsumerOwnerShadow.java:163) at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow. stop(WMQConsumerOwnerShadow.java:463) at com/ibm/msg/client/wmq/internal/WMQSession.stop (WMQSession.java:1881) at com/ibm/msg/client/jms/internal/JmsSessionImpl.stop (JmsSessionImpl.java:2537) at com/ibm/msg/client/jms/internal/JmsSessionImpl.close (JmsSessionImpl.java:423) at com/ibm/msg/client/jms/internal/JmsSessionImpl.close (JmsSessionImpl.java:342) at com/ibm/mq/jms/MQSession.close(MQSession.java:275) Thread 2 - AsyncConsume Thread: -------------------------------- "Jmqi AsyncConsume Thread. tid: 40" J9VMThread:0x0000000000394B00, j9thread_t:0x000000001051E120, java/lang/Thread:0x00000000A1FA94D0, state:P, prio=5 (java/lang/Thread getId:0x3C, isDaemon:true) (native thread ID:0x457, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001) (native stack address range from:0x00000FFF74020000, to:0x00000FFF740A0000, size:0x80000) CPU usage total: 0.815008996 secs, current category="Application" Parked on: java/util/concurrent/locks/ReentrantLock$NonfairSync@0x00000000A 1EB4668 Owned by: "main" (J9VMThread:0x0000000000010500, java/lang/Thread:0x00000000A00372C0) Heap bytes allocated since last GC cycle=524640 (0x80160) Java callstack: at sun/misc/Unsafe.park(Native Method) at java/util/concurrent/locks/LockSupport.park (LockSupport.java:198) at java/util/concurrent/locks/AbstractQueuedSynchronizer. parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:846) at java/util/concurrent/locks/AbstractQueuedSynchronizer. acquireQueued(AbstractQueuedSynchronizer.java:879) at java/util/concurrent/locks/AbstractQueuedSynchronizer. acquire(AbstractQueuedSynchronizer.java:1209) at java/util/concurrent/locks/ReentrantLock$NonfairSync. lock(ReentrantLock.java:226) at java/util/concurrent/locks/ReentrantLock.lock (ReentrantLock.java:302) at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow. controlAsyncService(WMQConsumerOwnerShadow.java:145) at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow. removeAsyncConsumer(WMQConsumerOwnerShadow.java:305) at com/ibm/msg/client/wmq/internal/WMQSession. removeAsyncConsumer(WMQSession.java:1739) at com/ibm/msg/client/wmq/internal/WMQAsyncConsumerShadow. poisonous(WMQAsyncConsumerShadow.java:795) at com/ibm/msg/client/wmq/internal/WMQAsyncConsumerShadow. consumer(WMQAsyncConsumerShadow.java:501) at com/ibm/mq/jmqi/local/internal/LocalProxyConsumer. jmqiConsumerMethod(LocalProxyConsumer.java:207(Compiled Code)) If this scenario occurs, the MQ classes for JMS log file (mqjms.log.0) will contain an error entry with following information: com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0032: There is no dead letter queue defined. WebSphere MQ classes for JMS attempted to put a poisonous message to the dead letter queue because it could not be routed to the backout retry queue. This failed because there was no dead letter queue defined to the queue manager. Define a dead letter queue. Class : class com.ibm.msg.client.jms.DetailedJMSException Stack : sun.reflect.NativeConstructorAccessorImpl.newInstance0 (NativeConstructorAccessorImpl.java) sun.reflect.NativeConstructorAccessorImpl.newInstance (NativeConstructorAccessorImpl.java) sun.reflect.DelegatingConstructorAccessorImpl.newInstance (DelegatingConstructorAccessorImpl.java) java.lang.reflect.Constructor.newInstance (Constructor.java) com.ibm.msg.client.commonservices.j2se.NLSServices. createException(NLSServices.java) com.ibm.msg.client.commonservices.nls.NLSServices. createException(NLSServices.java:226) com.ibm.msg.client.wmq.internal.WMQPoison.deadletter (WMQPoison.java:834) com.ibm.msg.client.wmq.internal.WMQPoison. handlePoisonMessage(WMQPoison.java:483) com.ibm.msg.client.wmq.internal.WMQPoison. handlePoisonMessage(WMQPoison.java:264) com.ibm.msg.client.wmq.internal.WMQPoison. handlePoisonMessage(WMQPoison.java:237) com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow. poisonous(WMQAsyncConsumerShadow.java:749) com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow. consumer(WMQAsyncConsumerShadow.java:501) com.ibm.mq.jmqi.local.internal.LocalProxyConsumer. jmqiConsumerMethod(LocalProxyConsumer.java:207) EXPLANATION: The WebSphere MQ classes for JMS connection browser caught an exception. If the JMS ConnectionFactory used by the application set the ASYNC_EXCEPTIONS property to the value WMQConstants.ASYNC_EXCEPTIONS_ALL, the the JMS ExceptionListener for the application will be invoked with the following exception: com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0032: There is no dead letter queue defined. WebSphere MQ classes for JMS attempted to put a poisonous message to the dead letter queue because it could not be routed to the backout retry queue. This failed because there was no dead letter queue defined to the queue manager. Define a dead letter queue.
Local fix
Ensure that poison messages can be requeued to either a backout requeue queue or the dead-letter queue.
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the IBM MQ classes for JMS who use JMS MessageListener's to receive messages asynchronously and are using a version of the MQ classes for JMS that have the fix for APAR IT10205 (e.g., IBM MQ V8.0.0.5, V9.0.0.0 or V9.0.1). Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: After APAR IT10205, a new internal lock (MessageListener MQCTL lock) was added to the MQ classes for JMS to control the flow of MQCTL calls to a queue manager for JMS MessageListeners. The new lock could have caused a deadlock if an application thread was closing either the MessageConsumer, JMSConsumer, Session, Connection or Context that was associated with the MessageListener at the same time as an asynchronous consume thread had been given a poison message to process that it could not move to the defined backout requeue queue or the queue manager's dead-letter queue (for example because it wasn't defined, was full or the application did not have permission). This was because the application thread stopping the JMS MessageListener would obtain the new MessageListener MQCTL lock and then make the MQCTL call as required. The queue manager would not complete this MQCTL call until any outstanding messages have been acknowledged. The asynchronous consume thread (the thread that delivers messages to a JMS MessageListener), after failing to requeue the outstanding poison message to either the backout requeue queue or the dead-letter queue, would then also attempt to stop any further message delivery by making its own MQCTL call. The asynchronous consume thread, erroneously attempted to obtain the MessageListener MQCTL lock but blocked because it was held by an application thread. The application thread, holding the MessageListener MQCTL lock couldn't complete until the asynchronous consume thread completes and so both threads became deadlocked.
Problem conclusion
The MQ classes for JMS have been updated such that the asynchronous consume thread does not attempt to obtain the MessageListener MQCTL lock that was added as part of APAR IT10205. This allows the asynchronous consume thread to complete its processing which, in turn, allows application threads closing JMS MessageListeners to also complete. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v8.0 8.0.0.6 v9.0 CD 9.0.2 v9.0 LTS 9.0.0.1 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT18436
Reported component name
WMQ BASE MULTIP
Reported component ID
5724H7251
Reported release
800
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-12-15
Closed date
2016-12-20
Last modified date
2017-06-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ BASE MULTIP
Fixed component ID
5724H7251
Applicable component levels
R800 PSY
UP
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0.0.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
01 June 2017