APAR status
Closed as program error.
Error description
An MQ classes for JMS application receives a notification that the queue manager to which it is connect is being quiesced via the MQ reason codes: MQRC_CONNECTION_QUIESCING (2202) and MQRC_Q_MGR_QUIESCING (2161) The application attempts to disconnect and invokes reconnect logic. However, the application is not able to reconnect and a hang in the client occurs. A Javacore of the application JVM shows the threads similar to those below, indicating this problem has occurred: "JMSCCThreadPoolWorker-##" #27 daemon prio=5 os_prio=0 tid=0x00007ff15c013800 nid=0x6b1b in Object.wait() [0x00007ff199bf9000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveAsyncTsh - locked <0x0000000790fa39e8> (a com.ibm.mq.jmqi.remote.impl.RemoteSession$AsyncTshLock) at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveTSH at com.ibm.mq.jmqi.remote.impl.RemoteSession.startConversation at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.sessio nFromEligible at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes sionFromEligibleConnection - locked <0x00000007209f7f68> (a com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification$Connec tionsLock) at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes sion at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect at com.ibm.msg.client.wmq.internal.WMQConnection.<init> at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7Pr oviderConnection at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProv iderConnection at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createCon nection at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConn ection at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection "RcvThread: com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@1544173434[qmid= CANNED_DATA,fap=13,channel=MY.SVRCONN,ccsid=500,sharecnv=10,hbin t=300,peer=localhost/127.0.0.1(1414),localport=18785,ssl=no]" #19 daemon prio=5 os_prio=0 tid=0x00007ff15c004800 nid=0x6ad6 waiting for monitor entry [0x00007ff19a2ff000] java.lang.Thread.State: BLOCKED (on object monitor) at com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSession - waiting to lock <0x00000007209f7f68> (a com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification$Connec tionsLock) at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas k at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem. runItem at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run
Local fix
Set the sharing conversations (SHARECNV) value on the server-connection channel used by the application to the value 1.
Problem summary
**************************************************************** USERS AFFECTED: This issue affects users of the: - IBM MQ V8 and V9 classes for JMS - IBM MQ V8 and V9 classes for Java - IBM MQ V8 and V9 JCA resource adapter - IBM MQ V8 and V9 OSGi bundles - IBM MQ V8 and V9 Managed File Transfer Platforms affected: MultiPlatform **************************************************************** PROBLEM DESCRIPTION: An MQ queue manager can send notifications to connected clients requesting a controlled closure of connection handles (conversations), for example when it is being quiesced. When this occurred, the Java message queueing interface (JMQI) received the notification on its "Remote Receive Thread" and would attempt to close the conversation and disassociate it from the connection it was using. If an application thread was simultaneously requesting a new conversation with the same connection details as used by the conversation being ended, then the JMQI may try and allocate the new conversation over an existing connection. This is known as multiplexing and is primarily controlled by the Sharing Conversations (SHARECNV) server/client-connection channel attribute. In the case where these two events occurred concurrently, the application thread that was attempting to start a new conversation (for example creating a new MQQueueManager in the classes for Java or a JMS Connection, JMS Session or JMS Context in the classes for JMS) would take a lock on an internal "RemoteConnectionSpecification" object. It would then send a data flow to the queue manager over the existing remote socket connection to request a new conversation be allocated and wait for the Remote Receive Thread to pass it the response as received from the queue manager. However, if the Remote Receive Thread was attempting to end a conversation, then it would request the lock for the internal RemoteConnectionSpecification object. The Remote Receive Thread in this case would not be able to obtain the lock on the internal RemoteConnectionSpecification object, because it was held by the application thread that was starting a new conversation. The application thread would only release the lock on the RemoteConnectionSpecification object when it was given the response to its conversation start request by the Remote Receive Thread, which would not occur because this thread was waiting for the lock on the RemoteConnectionSpecification object.
Problem conclusion
The logic regarding the locking of the internal RemoteConnectionSpecification object has been updated in the JMQI implementation to prevent the deadlock scenario described from occurring. If your Java application uses the Java Security Manager, after this APAR has been applied, it will be necessary to add the following additional permission to the java.security.policy file used by the application: permission java.lang.RuntimePermission "modifyThread"; Without the addition of the above RuntimePermission, the following exception may be thrown: Exception in thread "main" java.security.AccessControlException: Access denied ("java.lang.RuntimePermission" "modifyThread") at java.security.AccessController.throwACE at java.security.AccessController.checkPermissionHelper at java.security.AccessController.checkPermission at java.lang.SecurityManager.checkPermission at java.util.concurrent.ThreadPoolExecutor.checkShutdownAccess at java.util.concurrent.ThreadPoolExecutor.shutdownNow at java.util.concurrent.Executors$DelegatedExecutorService.shutdown Now at com.ibm.mq.jmqi.remote.impl.RemoteConnection$2.run at java.security.AccessController.doPrivileged at com.ibm.mq.jmqi.remote.impl.RemoteConnection.disconnect at com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSession at com.ibm.mq.jmqi.remote.impl.RemoteSession.disconnect at com.ibm.mq.jmqi.remote.impl.RemoteConnection.completeClose at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQDISC --------------------------------------------------------------- The fix is targeted for delivery in the following PTFs: Version Maintenance Level v8.0 8.0.0.9 v9.0 CD 9.0.5 v9.0 LTS 9.0.0.3 The latest available maintenance can be obtained from 'WebSphere MQ Recommended Fixes' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037 If the maintenance level is not yet available information on its planned availability can be found in 'WebSphere MQ Planned Maintenance Release Dates' http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309 ---------------------------------------------------------------
Temporary fix
Comments
APAR Information
APAR number
IT22127
Reported component name
WMQ BASE MULTIP
Reported component ID
5724H7251
Reported release
800
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2017-08-24
Closed date
2017-11-28
Last modified date
2017-12-01
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WMQ BASE MULTIP
Fixed component ID
5724H7251
Applicable component levels
R800 PSY
UP
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0.0.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
01 December 2017