IBM Support

IV67141: WMQ-JAVA/JMS APPLICATION THREADS HANG WHEN SIMULTANEOUSLY RECEIVING AND PUTTING MESSAGES TO A QUEUE

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • While using WebSphere MQ 7.0.1.11 an application will
    intermittently become unable to pull a message from the a queue
    on the Queue Manager. There are no FDCs or errors in the log
    files. The only thing seen is an open read handle on the queue
    in question. If the application is restarted everything starts
    working.
    

Local fix

  • For the Server Connection channel which the application uses to
    communicate with the queue manager, set the CHANNEL attribute:
    
      SHARECNV=1
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    Users of the WebSphere MQ classes for Java/JMS APIs at the
    WebSphere MQ v7.0 version, including users of the WebSphere MQ
    Resource Adapter as included within WebSphere Application Server
    7.0 and 8.0.
    
    The problem does not affect later versions of WebSphere MQ.
    
    
    Platforms affected:
    MultiPlatform, AIX, HP-UX Itanium, HP-UX PA-RISC, IBM iSeries,
    Linux on Power, Linux on S390, Linux on x86, Linux on x86-64,
    Linux on zSeries, Solaris SPARC, Solaris x86-64, Windows, z/OS
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    A deadlock could be reached under a specific set of
    circumstances where multiple threads were communicating with the
    queue manager over the same TCP/IP socket.
    
    Three application threads were required to be doing the
    following:
    
    
    PUTTING THREAD
    --------------
    A thread sending a large amount of data to the queue manager,
    such that the queue manager's Operating System TCP/IP buffer was
    filled to capacity.
    
    
    GETTING THREAD
    --------------
    A thread receiving data from the queue manager.
    
    
    DISCONNECTING THREAD
    --------------------
    A thread which had requested that its MQ artifacts were
    disconnected from the queue manager.
    
    
    
    For example, if using the WebSphere MQ classes for JMS API, your
    application might have the following three method calls running
    simultaneously:
    
    THREAD 1: MessageConsumer.receive()
    
    THREAD 2: MessageProducer.send(JMSMessage)
    
    THREAD 3: Session.close()
    
    where all three threads are using a JMS Connection which is
    configured for a remote (TCP/IP) connection to the same queue
    manager, and all three threads have connected to the queue
    manager over the same TCP/IP socket, meaning that the value of
    SHARECNV on the CHANNEL is at least 3.
    
    
    If these conditions were met, then there was a small timing
    window which provided a chance that the threads could deadlock,
    a condition which would last until the TCP/IP connection to the
    queue manager was disconnected.
    
    When this happened, a Javacore taken of the JVM would show the
    following 4 thread stacks (these stack traces were taking from
    an application using the WebSphere MQ classes for JMS
    v7.0.1.12):
    
    
    
    "MQPUT thread" J9VMThread:0x00007F4A448CD700,
    j9thread_t:0x00007F4A448C9680,
    java/lang/Thread:0x00007F4A2394FF68, state:R, prio=5
           (java/lang/Thread getId:0xF, isDaemon:false)
           (native thread ID:0x4494, native priority:0x5, native
    policy:UNKNOWN)
           (native stack address range from:0x00007F4A1D96E000,
    to:0x00007F4A1D9AF000, size:0x41000)
          Java callstack:
              at java/net/SocketOutputStream.socketWrite0(Native
    Method)
              at
    java/net/SocketOutputStream.socketWrite(SocketOutputStream.java:
    103(Compiled Code))
              at
    java/net/SocketOutputStream.write(SocketOutputStream.java:147(Co
    mpiled Code))
              at
    com/ibm/mq/jmqi/remote/internal/RemoteTCPConnection.send(RemoteT
    CPConnection.java:1329(Compiled Code))
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.sendTSH(
    RemoteConnection.java:2636(Compiled Code))
              at
    com/ibm/mq/jmqi/remote/internal/RemoteHconn.sendTSH(RemoteHconn.
    java:1221(Compiled Code))
              at
    com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiPutMessageWithProp
    s(RemoteFAP.java:7877)
              at
    com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiPut(RemoteFAP.java
    :7149)
              at
    com/ibm/msg/client/wmq/internal/WMQMessageProducer$SpiIdentified
    ProducerShadow.sendInternal(WMQMessageProducer.java:857)
              at
    com/ibm/msg/client/wmq/internal/WMQMessageProducer$ProducerShado
    w.send(WMQMessageProducer.java:542)
              at
    com/ibm/msg/client/wmq/internal/WMQMessageProducer.send(WMQMessa
    geProducer.java:1223)
              at
    com/ibm/msg/client/jms/internal/JmsMessageProducerImpl.sendMessa
    ge(JmsMessageProducerImpl.java:935)
              at
    com/ibm/msg/client/jms/internal/JmsMessageProducerImpl.send_(Jms
    MessageProducerImpl.java:787)
              at
    com/ibm/msg/client/jms/internal/JmsMessageProducerImpl.send(JmsM
    essageProducerImpl.java:442)
              at
    com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:304
    )
    
    
    
    "MQGET thread" J9VMThread:0x00007F4A44637800,
    j9thread_t:0x00007F4A44612DB0,
    java/lang/Thread:0x00007F4A2393A5A0, state:CW, prio=5
           (java/lang/Thread getId:0xE, isDaemon:false)
           (native thread ID:0x4493, native priority:0x5, native
    policy:UNKNOWN)
           (native stack address range from:0x00007F4A1D9AF000,
    to:0x00007F4A1D9F0000, size:0x41000)
          Java callstack:
              at java/lang/Object.wait(Native Method)
              at java/lang/Object.wait(Object.java:196)
              at
    com/ibm/mq/jmqi/remote/internal/RemoteHconn.exchangeTSH(RemoteHc
    onn.java:1697)
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteProxyQueue.requestM
    essages(RemoteProxyQueue.java:1071)
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteProxyQueue.flushQue
    ue(RemoteProxyQueue.java:1907)
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteProxyQueue.proxyMQG
    ET(RemoteProxyQueue.java:2763)
              at
    com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiGetMessageWithReco
    n(RemoteFAP.java:6423)
              at
    com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiGetMessage(RemoteF
    AP.java:6315)
              at
    com/ibm/mq/jmqi/internal/JmqiTools.getMessage(JmqiTools.java:119
    3)
              at
    com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiGet(RemoteFAP.java
    :6280)
              at
    com/ibm/msg/client/wmq/internal/WMQConsumerShadow.getMsg(WMQCons
    umerShadow.java:1384)
              at
    com/ibm/msg/client/wmq/internal/WMQSyncConsumerShadow.receiveInt
    ernal(WMQSyncConsumerShadow.java:239)
              at
    com/ibm/msg/client/wmq/internal/WMQConsumerShadow.receive(WMQCon
    sumerShadow.java:1135)
              at
    com/ibm/msg/client/wmq/internal/WMQMessageConsumer.receive(WMQMe
    ssageConsumer.java:469)
              at
    com/ibm/msg/client/jms/internal/JmsMessageConsumerImpl.receiveIn
    boundMessage(JmsMessageConsumerImpl.java:883)
              at
    com/ibm/msg/client/jms/internal/JmsMessageConsumerImpl.receive(J
    msMessageConsumerImpl.java:546)
              at
    com/ibm/mq/jms/MQMessageConsumer.receive(MQMessageConsumer.java:
    258)
    
    
    
    "Disconnecting thread" J9VMThread:0x00007F4A448CE800,
    j9thread_t:0x00007F4A448C9B70,
    java/lang/Thread:0x00007F4A239507D8, state:CW, prio=5
           (java/lang/Thread getId:0x10, isDaemon:false)
           (native thread ID:0x4495, native priority:0x5, native
    policy:UNKNOWN)
           (native stack address range from:0x00007F4A1D92D000,
    to:0x00007F4A1D96E000, size:0x41000)
          Java callstack:
              at java/lang/Object.wait(Native Method)
              at java/lang/Object.wait(Object.java:196)
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.sendTSH(
    RemoteConnection.java:2443(Compiled Code))
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.removeHc
    onn(RemoteConnection.java:1221)
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.removeHc
    onn(RemoteConnection.java:1120)
              at
    com/ibm/mq/jmqi/remote/internal/RemoteHconn.disconnect(RemoteHco
    nn.java:2274)
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.complete
    Close(RemoteConnection.java:5357)
              at
    com/ibm/mq/jmqi/remote/internal/RemoteFAP.MQDISC(RemoteFAP.java:
    2741)
              at
    com/ibm/msg/client/wmq/internal/WMQSession.disconnect(WMQSession
    .java:760)
              at
    com/ibm/msg/client/wmq/internal/WMQSession.close(WMQSession.java
    :732)
              at
    com/ibm/msg/client/jms/internal/JmsSessionImpl.close(JmsSessionI
    mpl.java:523)
              at
    com/ibm/msg/client/jms/internal/JmsSessionImpl.close(JmsSessionI
    mpl.java:309)
              at com/ibm/mq/jms/MQSession.close(MQSession.java:195)
    
    
    
      "RcvThread:
    com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection@197659592
    [qmid=appleQM_2014-11-19_17.33.10,fap=10,channel=BIG.MSG.CHANNEL
    ,ccsid=1208,sharecnv=10,hbint=300,peer=moose.hursley.ibm.com/9.2
    0.58.191(1417),localport=54283,ssl=no]"
    J9VMThread:0x00007F4A0400E600, j9thread_t:0x00007F4A446128C0,
    java/lang/Thread:0x00007F4A1E352B38, state:B, prio=5
           (java/lang/Thread getId:0xD, isDaemon:true)
           (native thread ID:0x4492, native priority:0x5, native
    policy:UNKNOWN)
           (native stack address range from:0x00007F4A1D9F0000,
    to:0x00007F4A1DA31000, size:0x41000)
          Java callstack:
              at
    com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.getHconn
    ByConvId(RemoteConnection.java:4489(Compiled Code))
              at
    com/ibm/mq/jmqi/remote/internal/RemoteRcvThread.run(RemoteRcvThr
    ead.java:174)
              at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas
    k(WorkQueueItem.java:209)
              at
    com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
    runItem(SimpleWorkQueueItem.java:100)
              at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo
    rkQueueItem.java:224)
              at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueManager.run
    WorkQueueItem(WorkQueueManager.java:298)
              at
    com/ibm/msg/client/commonservices/j2se/workqueue/WorkQueueManage
    rImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementat
    ion.java:1220)
    

Problem conclusion

  • The locking model used by the disconnecting thread has been
    changed to not hold a lock for the time period when
    communicating with the queue manager. This permits the receiving
    thread ("RcvThread") to determine which application thread was
    the target of data sent from the queue manager, freeing up the
    "MQGET thread", and avoiding the deadlock.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v7.0       7.0.1.14
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IV67141

  • Reported component name

    WMQ LIN X86 V7

  • Reported component ID

    5724H7224

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2014-11-20

  • Closed date

    2015-04-21

  • Last modified date

    2015-04-22

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PI44248

Fix information

  • Fixed component name

    WMQ LIN X86 V7

  • Fixed component ID

    5724H7224

Applicable component levels

[{"Line of Business":{"code":"LOB36","label":"IBM Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0"}]

Document Information

Modified date:
08 March 2021