IT22127: Java client hang when queue manager is quiescing as new connection attempts are made

APAR status

• Closed as program error.

Error description

• An MQ classes for JMS application receives a notification that
  the queue manager to which it is connect is being quiesced via
  the MQ reason codes:
  
    MQRC_CONNECTION_QUIESCING (2202)
  
  and
  
    MQRC_Q_MGR_QUIESCING (2161)
  
  The application attempts to disconnect and invokes reconnect
  logic.  However, the application is not able to reconnect and a
  hang in the client occurs.
  
  A Javacore of the application JVM shows the threads similar to
  those below, indicating this problem has occurred:
  
  "JMSCCThreadPoolWorker-##" #27 daemon prio=5 os_prio=0
  tid=0x00007ff15c013800 nid=0x6b1b in Object.wait()
  [0x00007ff199bf9000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
          at java.lang.Object.wait(Native Method)
          at
  com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveAsyncTsh
          - locked <0x0000000790fa39e8> (a
  com.ibm.mq.jmqi.remote.impl.RemoteSession$AsyncTshLock)
          at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveTSH
          at
  com.ibm.mq.jmqi.remote.impl.RemoteSession.startConversation
          at
  com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.sessio
  nFromEligible
          at
  com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes
  sionFromEligibleConnection
          - locked <0x00000007209f7f68> (a
  com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification$Connec
  tionsLock)
          at
  com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes
  sion
          at
  com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession
          at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect
          at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect
          at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect
          at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect
          at com.ibm.msg.client.wmq.internal.WMQConnection.<init>
          at
  com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7Pr
  oviderConnection
          at
  com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProv
  iderConnection
          at
  com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createCon
  nection
          at
  com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConn
  ection
          at
  com.ibm.mq.jms.MQConnectionFactory.createCommonConnection
          at
  com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection
  
  
  "RcvThread:
  com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@1544173434[qmid=
  CANNED_DATA,fap=13,channel=MY.SVRCONN,ccsid=500,sharecnv=10,hbin
  t=300,peer=localhost/127.0.0.1(1414),localport=18785,ssl=no]"
  #19 daemon prio=5 os_prio=0 tid=0x00007ff15c004800 nid=0x6ad6
  waiting for monitor entry [0x00007ff19a2ff000]
     java.lang.Thread.State: BLOCKED (on object monitor)
          at
  com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSession
          - waiting to lock <0x00000007209f7f68> (a
  com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification$Connec
  tionsLock)
          at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run
          at
  com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas
  k
          at
  com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.
  runItem
          at
  com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run
  

Local fix

• Set the sharing conversations (SHARECNV) value on the
  server-connection channel used by the application to the value
  1.
  

Problem summary

• ****************************************************************
  USERS AFFECTED:
  This issue affects users of the:
  
    - IBM MQ V8 and V9 classes for JMS
    - IBM MQ V8 and V9 classes for Java
    - IBM MQ V8 and V9 JCA resource adapter
    - IBM MQ V8 and V9 OSGi bundles
    - IBM MQ V8 and V9 Managed File Transfer
  
  
  Platforms affected:
  MultiPlatform
  
  ****************************************************************
  PROBLEM DESCRIPTION:
  An MQ queue manager can send notifications to connected clients
  requesting a controlled closure of connection handles
  (conversations), for example when it is being quiesced.  When
  this occurred, the Java message queueing interface (JMQI)
  received the notification on its "Remote Receive Thread" and
  would attempt to close the conversation and disassociate it from
  the connection it was using.
  
  If an application thread was simultaneously requesting a new
  conversation with the same connection details as used by the
  conversation being ended, then the JMQI may try and allocate the
  new conversation over an existing connection.  This is known as
  multiplexing and is primarily controlled by the Sharing
  Conversations (SHARECNV) server/client-connection channel
  attribute.
  
  In the case where these two events occurred concurrently, the
  application thread that was attempting to start a new
  conversation (for example creating a new MQQueueManager in the
  classes for Java or a JMS Connection, JMS Session or JMS Context
  in the classes for JMS) would take a lock on an internal
  "RemoteConnectionSpecification" object.  It would then send a
  data flow to the queue manager over the existing remote socket
  connection to request a new conversation be allocated and wait
  for the Remote Receive Thread to pass it the response as
  received from the queue manager.  However, if the Remote Receive
  Thread was attempting to end a conversation, then it would
  request the lock for the internal RemoteConnectionSpecification
  object.
  
  The Remote Receive Thread in this case would not be able to
  obtain the lock on the internal RemoteConnectionSpecification
  object, because it was held by the application thread that was
  starting a new conversation.  The application thread would only
  release the lock on the RemoteConnectionSpecification object
  when it was given the response to its conversation start request
  by the Remote Receive Thread, which would not occur because this
  thread was waiting for the lock on the
  RemoteConnectionSpecification object.
  

Problem conclusion

• The logic regarding the locking of the internal
  RemoteConnectionSpecification object has been updated in the
  JMQI implementation to prevent the deadlock scenario described
  from occurring.
  
  If your Java application uses the Java Security Manager, after
  this APAR has been applied, it will be necessary to add the
  following additional permission to the java.security.policy file
  used by the application:
  
      permission java.lang.RuntimePermission "modifyThread";
  
  Without the addition of the above RuntimePermission, the
  following exception may be thrown:
  
  Exception in thread "main" java.security.AccessControlException:
  Access denied ("java.lang.RuntimePermission" "modifyThread")
  at java.security.AccessController.throwACE
  at java.security.AccessController.checkPermissionHelper
  at java.security.AccessController.checkPermission
  at java.lang.SecurityManager.checkPermission
  at java.util.concurrent.ThreadPoolExecutor.checkShutdownAccess
  at java.util.concurrent.ThreadPoolExecutor.shutdownNow
  at
  java.util.concurrent.Executors$DelegatedExecutorService.shutdown
  Now
  at com.ibm.mq.jmqi.remote.impl.RemoteConnection$2.run
  at java.security.AccessController.doPrivileged
  at com.ibm.mq.jmqi.remote.impl.RemoteConnection.disconnect
  at com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSession
  at com.ibm.mq.jmqi.remote.impl.RemoteSession.disconnect
  at com.ibm.mq.jmqi.remote.impl.RemoteConnection.completeClose
  at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQDISC
  
  ---------------------------------------------------------------
  The fix is targeted for delivery in the following PTFs:
  
  Version    Maintenance Level
  v8.0       8.0.0.9
  v9.0 CD    9.0.5
  v9.0 LTS   9.0.0.3
  
  The latest available maintenance can be obtained from
  'WebSphere MQ Recommended Fixes'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
  
  If the maintenance level is not yet available information on
  its planned availability can be found in 'WebSphere MQ
  Planned Maintenance Release Dates'
  http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
  ---------------------------------------------------------------
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WMQ BASE MULTIP

• Fixed component ID

  5724H7251

Applicable component levels

• R800 PSY

     UP