APAR status
Closed as program error.
Error description
While using WebSphere MQ 7.0.1.11 an application will intermittently become unable to pull a message from the a queue on the Queue Manager. There are no FDCs or errors in the log files. The only thing seen is an open read handle on the queue in question. If the application is restarted everything starts working.
Local fix
For the Server Connection channel which the application uses to communicate with the queue manager, set the CHANNEL attribute: SHARECNV=1
Problem summary
**************************************************************** USERS AFFECTED: Users of the WebSphere MQ classes for Java/JMS APIs at the WebSphere MQ v7.0 version, including users of the WebSphere MQ Resource Adapter as included within WebSphere Application Server 7.0 and 8.0. The problem does not affect later versions of WebSphere MQ. Platforms affected: MultiPlatform, AIX, HP-UX Itanium, HP-UX PA-RISC, IBM iSeries, Linux on Power, Linux on S390, Linux on x86, Linux on x86-64, Linux on zSeries, Solaris SPARC, Solaris x86-64, Windows, z/OS **************************************************************** PROBLEM DESCRIPTION: A deadlock could be reached under a specific set of circumstances where multiple threads were communicating with the queue manager over the same TCP/IP socket. Three application threads were required to be doing the following: PUTTING THREAD -------------- A thread sending a large amount of data to the queue manager, such that the queue manager's Operating System TCP/IP buffer was filled to capacity. GETTING THREAD -------------- A thread receiving data from the queue manager. DISCONNECTING THREAD -------------------- A thread which had requested that its MQ artifacts were disconnected from the queue manager. For example, if using the WebSphere MQ classes for JMS API, your application might have the following three method calls running simultaneously: THREAD 1: MessageConsumer.receive() THREAD 2: MessageProducer.send(JMSMessage) THREAD 3: Session.close() where all three threads are using a JMS Connection which is configured for a remote (TCP/IP) connection to the same queue manager, and all three threads have connected to the queue manager over the same TCP/IP socket, meaning that the value of SHARECNV on the CHANNEL is at least 3. If these conditions were met, then there was a small timing window which provided a chance that the threads could deadlock, a condition which would last until the TCP/IP connection to the queue manager was disconnected. When this happened, a Javacore taken of the JVM would show the following 4 thread stacks (these stack traces were taking from an application using the WebSphere MQ classes for JMS v7.0.1.12): "MQPUT thread" J9VMThread:0x00007F4A448CD700, j9thread_t:0x00007F4A448C9680, java/lang/Thread:0x00007F4A2394FF68, state:R, prio=5 (java/lang/Thread getId:0xF, isDaemon:false) (native thread ID:0x4494, native priority:0x5, native policy:UNKNOWN) (native stack address range from:0x00007F4A1D96E000, to:0x00007F4A1D9AF000, size:0x41000) Java callstack: at java/net/SocketOutputStream.socketWrite0(Native Method) at java/net/SocketOutputStream.socketWrite(SocketOutputStream.java: 103(Compiled Code)) at java/net/SocketOutputStream.write(SocketOutputStream.java:147(Co mpiled Code)) at com/ibm/mq/jmqi/remote/internal/RemoteTCPConnection.send(RemoteT CPConnection.java:1329(Compiled Code)) at com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.sendTSH( RemoteConnection.java:2636(Compiled Code)) at com/ibm/mq/jmqi/remote/internal/RemoteHconn.sendTSH(RemoteHconn. java:1221(Compiled Code)) at com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiPutMessageWithProp s(RemoteFAP.java:7877) at com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiPut(RemoteFAP.java :7149) at com/ibm/msg/client/wmq/internal/WMQMessageProducer$SpiIdentified ProducerShadow.sendInternal(WMQMessageProducer.java:857) at com/ibm/msg/client/wmq/internal/WMQMessageProducer$ProducerShado w.send(WMQMessageProducer.java:542) at com/ibm/msg/client/wmq/internal/WMQMessageProducer.send(WMQMessa geProducer.java:1223) at com/ibm/msg/client/jms/internal/JmsMessageProducerImpl.sendMessa ge(JmsMessageProducerImpl.java:935) at com/ibm/msg/client/jms/internal/JmsMessageProducerImpl.send_(Jms MessageProducerImpl.java:787) at com/ibm/msg/client/jms/internal/JmsMessageProducerImpl.send(JmsM essageProducerImpl.java:442) at com/ibm/mq/jms/MQMessageProducer.send(MQMessageProducer.java:304 ) "MQGET thread" J9VMThread:0x00007F4A44637800, j9thread_t:0x00007F4A44612DB0, java/lang/Thread:0x00007F4A2393A5A0, state:CW, prio=5 (java/lang/Thread getId:0xE, isDaemon:false) (native thread ID:0x4493, native priority:0x5, native policy:UNKNOWN) (native stack address range from:0x00007F4A1D9AF000, to:0x00007F4A1D9F0000, size:0x41000) Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:196) at com/ibm/mq/jmqi/remote/internal/RemoteHconn.exchangeTSH(RemoteHc onn.java:1697) at com/ibm/mq/jmqi/remote/internal/system/RemoteProxyQueue.requestM essages(RemoteProxyQueue.java:1071) at com/ibm/mq/jmqi/remote/internal/system/RemoteProxyQueue.flushQue ue(RemoteProxyQueue.java:1907) at com/ibm/mq/jmqi/remote/internal/system/RemoteProxyQueue.proxyMQG ET(RemoteProxyQueue.java:2763) at com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiGetMessageWithReco n(RemoteFAP.java:6423) at com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiGetMessage(RemoteF AP.java:6315) at com/ibm/mq/jmqi/internal/JmqiTools.getMessage(JmqiTools.java:119 3) at com/ibm/mq/jmqi/remote/internal/RemoteFAP.jmqiGet(RemoteFAP.java :6280) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.getMsg(WMQCons umerShadow.java:1384) at com/ibm/msg/client/wmq/internal/WMQSyncConsumerShadow.receiveInt ernal(WMQSyncConsumerShadow.java:239) at com/ibm/msg/client/wmq/internal/WMQConsumerShadow.receive(WMQCon sumerShadow.java:1135) at com/ibm/msg/client/wmq/internal/WMQMessageConsumer.receive(WMQMe ssageConsumer.java:469) at com/ibm/msg/client/jms/internal/JmsMessageConsumerImpl.receiveIn boundMessage(JmsMessageConsumerImpl.java:883) at com/ibm/msg/client/jms/internal/JmsMessageConsumerImpl.receive(J msMessageConsumerImpl.java:546) at com/ibm/mq/jms/MQMessageConsumer.receive(MQMessageConsumer.java: 258) "Disconnecting thread" J9VMThread:0x00007F4A448CE800, j9thread_t:0x00007F4A448C9B70, java/lang/Thread:0x00007F4A239507D8, state:CW, prio=5 (java/lang/Thread getId:0x10, isDaemon:false) (native thread ID:0x4495, native priority:0x5, native policy:UNKNOWN) (native stack address range from:0x00007F4A1D92D000, to:0x00007F4A1D96E000, size:0x41000) Java callstack: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:196) at com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.sendTSH( RemoteConnection.java:2443(Compiled Code)) at com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.removeHc onn(RemoteConnection.java:1221) at com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.removeHc onn(RemoteConnection.java:1120) at com/ibm/mq/jmqi/remote/internal/RemoteHconn.disconnect(RemoteHco nn.java:2274) at com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.complete Close(RemoteConnection.java:5357) at com/ibm/mq/jmqi/remote/internal/RemoteFAP.MQDISC(RemoteFAP.java: 2741) at com/ibm/msg/client/wmq/internal/WMQSession.disconnect(WMQSession .java:760) at com/ibm/msg/client/wmq/internal/WMQSession.close(WMQSession.java :732) at com/ibm/msg/client/jms/internal/JmsSessionImpl.close(JmsSessionI mpl.java:523) at com/ibm/msg/client/jms/internal/JmsSessionImpl.close(JmsSessionI mpl.java:309) at com/ibm/mq/jms/MQSession.close(MQSession.java:195) "RcvThread: com.ibm.mq.jmqi.remote.internal.RemoteTCPConnection@197659592 [qmid=appleQM_2014-11-19_17.33.10,fap=10,channel=BIG.MSG.CHANNEL ,ccsid=1208,sharecnv=10,hbint=300,peer=moose.hursley.ibm.com/9.2 0.58.191(1417),localport=54283,ssl=no]" J9VMThread:0x00007F4A0400E600, j9thread_t:0x00007F4A446128C0, java/lang/Thread:0x00007F4A1E352B38, state:B, prio=5 (java/lang/Thread getId:0xD, isDaemon:true) (native thread ID:0x4492, native priority:0x5, native policy:UNKNOWN) (native stack address range from:0x00007F4A1D9F0000, to:0x00007F4A1DA31000, size:0x41000) Java callstack: at com/ibm/mq/jmqi/remote/internal/system/RemoteConnection.getHconn ByConvId(RemoteConnection.java:4489(Compiled Code)) at com/ibm/mq/jmqi/remote/internal/RemoteRcvThread.run(RemoteRcvThr ead.java:174) at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas k(WorkQueueItem.java:209) at com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem. runItem(SimpleWorkQueueItem.java:100) at com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo rkQueueItem.java:224) at com/ibm/msg/client/commonservices/workqueue/WorkQueueManager.run WorkQueueItem(WorkQueueManager.java:298) at com/ibm/msg/client/commonservices/j2se/workqueue/WorkQueueManage rImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementat ion.java:1220)
Problem conclusion
The locking model used by the disconnecting thread has been changed to not hold a lock for the time period when communicating with the queue manager. This permits the receiving thread ("RcvThread") to determine which application thread was the target of data sent from the queue manager, freeing up the "MQGET thread", and avoiding the deadlock. --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v7.0 7.0.1.14 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IV67141
Reported component name
WMQ LIN X86 V7
Reported component ID
5724H7224
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2014-11-20
Closed date
2015-04-21
Last modified date
2015-04-22
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
PI44248
Fix information
Fixed component name
WMQ LIN X86 V7
Fixed component ID
5724H7224
Applicable component levels
[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0"}]
Document Information
Modified date:
08 March 2021