IBM Support

IT18436: Deadlock seen in MQ Java on session.close() when async consume in use and poison message cannot be put to DLQ

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • If an application using the IBM MQ classes for JMS configures a
    JMS MessageListener to receive messages asynchronously, a
    deadlock can occur between the MQ Java client and the queue
    manager.
    
    This has been observed when the following conditions occur:
    
      a) a poison message is being passed to the asynchronous
    consume callback function in the MQ classes for JMS from the
    queue manager,
    
      b) that message cannot be requeued to either the backout
    requeue queue or the dead-letter queue,
    
      c) at the same time, a separate application thread attempts to
    close the JMS MessageConsumer, Session or Connection (or for the
    JMS 2.0 API, a JMSConsumer or JMSContext) associated with the
    MessageListener.
    
    A Javacore (thread dump) of the application will show the two
    threads that are deadlocked.  The threads below are taken from
    an MQ classes for JMS application that connects to a queue
    manager using the BINDINGS transport mode:
    
    Thread 1 - Closing a JMS Session:
    ----------------------------------
    "main" J9VMThread:0x0000000000010500,
    j9thread_t:0x000000001004F540,
    java/lang/Thread:0x00000000A00372C0, state:R, prio=5
        (java/lang/Thread getId:0x1, isDaemon:false)
        (native thread ID:0x443, native priority:0x5, native
    policy:UNKNOWN, vmstate:R, vm thread flags:0x00000001)
        (native stack address range from:0x00000FFF8B610000,
    to:0x00000FFF8BE10000, size:0x800000)
       CPU usage total: 4.614949438 secs, current
    category="Application"
       Heap bytes allocated since last GC cycle=0 (0x0)
    
    Java callstack:
      at com/ibm/mq/jmqi/local/internal/base/Native.MQCTL
    	   (Native Method)
      at com/ibm/mq/jmqi/local/LocalMQ.MQCTL(
    	   LocalMQ.java:3125)
      at com/ibm/mq/jmqi/monitoring/JmqiInterceptAdapter.
           MQCTL(JmqiInterceptAdapter.java:333)
      at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
           controlAsyncService(WMQConsumerOwnerShadow.java:163)
      at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
           stop(WMQConsumerOwnerShadow.java:463)
      at com/ibm/msg/client/wmq/internal/WMQSession.stop
           (WMQSession.java:1881)
      at com/ibm/msg/client/jms/internal/JmsSessionImpl.stop
           (JmsSessionImpl.java:2537)
      at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
           (JmsSessionImpl.java:423)
      at com/ibm/msg/client/jms/internal/JmsSessionImpl.close
           (JmsSessionImpl.java:342)
      at com/ibm/mq/jms/MQSession.close(MQSession.java:275)
    
    Thread 2 - AsyncConsume Thread:
    --------------------------------
    "Jmqi AsyncConsume Thread. tid: 40"
    J9VMThread:0x0000000000394B00, j9thread_t:0x000000001051E120,
    java/lang/Thread:0x00000000A1FA94D0, state:P, prio=5
        (java/lang/Thread getId:0x3C, isDaemon:true)
        (native thread ID:0x457, native priority:0x5, native
    policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001)
        (native stack address range from:0x00000FFF74020000,
    to:0x00000FFF740A0000, size:0x80000)
       CPU usage total: 0.815008996 secs, current
    category="Application"
    
    Parked on:
    java/util/concurrent/locks/ReentrantLock$NonfairSync@0x00000000A
    1EB4668
    Owned by: "main" (J9VMThread:0x0000000000010500,
    java/lang/Thread:0x00000000A00372C0)
       Heap bytes allocated since last GC cycle=524640 (0x80160)
    
    Java callstack:
      at sun/misc/Unsafe.park(Native Method)
      at java/util/concurrent/locks/LockSupport.park
           (LockSupport.java:198)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer.
    
    parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:846)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer.
           acquireQueued(AbstractQueuedSynchronizer.java:879)
      at java/util/concurrent/locks/AbstractQueuedSynchronizer.
           acquire(AbstractQueuedSynchronizer.java:1209)
      at java/util/concurrent/locks/ReentrantLock$NonfairSync.
           lock(ReentrantLock.java:226)
      at java/util/concurrent/locks/ReentrantLock.lock
           (ReentrantLock.java:302)
      at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
           controlAsyncService(WMQConsumerOwnerShadow.java:145)
      at com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.
           removeAsyncConsumer(WMQConsumerOwnerShadow.java:305)
      at com/ibm/msg/client/wmq/internal/WMQSession.
           removeAsyncConsumer(WMQSession.java:1739)
      at com/ibm/msg/client/wmq/internal/WMQAsyncConsumerShadow.
           poisonous(WMQAsyncConsumerShadow.java:795)
      at com/ibm/msg/client/wmq/internal/WMQAsyncConsumerShadow.
           consumer(WMQAsyncConsumerShadow.java:501)
      at com/ibm/mq/jmqi/local/internal/LocalProxyConsumer.
           jmqiConsumerMethod(LocalProxyConsumer.java:207(Compiled
    Code))
    
    
    If this scenario occurs, the MQ classes for JMS log file
    (mqjms.log.0) will contain an error entry with following
    information:
    
    com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0032:
    There is no dead letter queue defined.
    
    WebSphere MQ classes for JMS attempted to put a poisonous
    message to the dead letter queue because it could not be routed
    to the backout retry queue. This failed because there was no
    dead letter queue defined to the queue manager.
    Define a dead letter queue.
    
    Class : class com.ibm.msg.client.jms.DetailedJMSException
    Stack :
    sun.reflect.NativeConstructorAccessorImpl.newInstance0
      (NativeConstructorAccessorImpl.java)
    sun.reflect.NativeConstructorAccessorImpl.newInstance
      (NativeConstructorAccessorImpl.java)
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance
      (DelegatingConstructorAccessorImpl.java)
    java.lang.reflect.Constructor.newInstance
      (Constructor.java)
    com.ibm.msg.client.commonservices.j2se.NLSServices.
      createException(NLSServices.java)
    com.ibm.msg.client.commonservices.nls.NLSServices.
      createException(NLSServices.java:226)
    com.ibm.msg.client.wmq.internal.WMQPoison.deadletter
      (WMQPoison.java:834)
    com.ibm.msg.client.wmq.internal.WMQPoison.
      handlePoisonMessage(WMQPoison.java:483)
    com.ibm.msg.client.wmq.internal.WMQPoison.
      handlePoisonMessage(WMQPoison.java:264)
    com.ibm.msg.client.wmq.internal.WMQPoison.
      handlePoisonMessage(WMQPoison.java:237)
    com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow.
      poisonous(WMQAsyncConsumerShadow.java:749)
    com.ibm.msg.client.wmq.internal.WMQAsyncConsumerShadow.
      consumer(WMQAsyncConsumerShadow.java:501)
    com.ibm.mq.jmqi.local.internal.LocalProxyConsumer.
      jmqiConsumerMethod(LocalProxyConsumer.java:207)
    
    EXPLANATION:
    The WebSphere MQ classes for JMS connection browser caught an
    exception.
    
    
    If the JMS ConnectionFactory used by the application set the
    ASYNC_EXCEPTIONS property to the value
    WMQConstants.ASYNC_EXCEPTIONS_ALL, the the JMS ExceptionListener
    for the application will be invoked with the following
    exception:
    
    com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0032: There
    is no dead letter queue defined.
      WebSphere MQ classes for JMS attempted to put a poisonous
    message to the dead letter queue because it could not be routed
    to the backout retry queue. This failed because there was no
    dead letter queue defined to the queue manager.
      Define a dead letter queue.
    

Local fix

  • Ensure that poison messages can be requeued to either a backout
    requeue queue or the dead-letter queue.
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of the IBM MQ classes for JMS who use
    JMS MessageListener's to receive messages asynchronously and are
    using a version of the MQ classes for JMS that have the fix for
    APAR IT10205 (e.g., IBM MQ V8.0.0.5, V9.0.0.0 or V9.0.1).
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    After APAR IT10205, a new internal lock (MessageListener MQCTL
    lock) was added to the MQ classes for JMS to control the flow of
    MQCTL calls to a queue manager for JMS MessageListeners.  The
    new lock could have caused a deadlock if an application thread
    was closing either the MessageConsumer, JMSConsumer, Session,
    Connection or Context that was associated with the
    MessageListener at the same time as an asynchronous consume
    thread had been given a poison message to process that it could
    not move to the defined backout requeue queue or the queue
    manager's dead-letter queue (for example because it wasn't
    defined, was full or the application did not have permission).
    
    This was because the application thread stopping the JMS
    MessageListener would obtain the new MessageListener MQCTL lock
    and then make the MQCTL call as required.  The queue manager
    would not complete this MQCTL call until any outstanding
    messages have been acknowledged.  The asynchronous consume
    thread (the thread that delivers messages to a JMS
    MessageListener), after failing to requeue the outstanding
    poison message to either the backout requeue queue or the
    dead-letter queue, would then also attempt to stop any further
    message delivery by making its own MQCTL call.  The asynchronous
    consume thread, erroneously attempted to obtain the
    MessageListener MQCTL lock but blocked because it was held by an
    application thread.  The application thread, holding the
    MessageListener MQCTL lock couldn't complete until the
    asynchronous consume thread completes and so both threads became
    deadlocked.
    

Problem conclusion

  • The MQ classes for JMS have been updated such that the
    asynchronous consume thread does not attempt to obtain the
    MessageListener MQCTL lock that was added as part of APAR
    IT10205.  This allows the asynchronous consume thread to
    complete its processing which, in turn, allows application
    threads closing JMS MessageListeners to also complete.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v8.0       8.0.0.6
    v9.0 CD    9.0.2
    v9.0 LTS   9.0.0.1
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT18436

  • Reported component name

    WMQ BASE MULTIP

  • Reported component ID

    5724H7251

  • Reported release

    800

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    YesHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-12-15

  • Closed date

    2016-12-20

  • Last modified date

    2017-06-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WMQ BASE MULTIP

  • Fixed component ID

    5724H7251

Applicable component levels

  • R800 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0.0.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
01 June 2017