IBM Support

IT22127: Java client hang when queue manager is quiescing as new connection attempts are made

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • An MQ classes for JMS application receives a notification that
    the queue manager to which it is connect is being quiesced via
    the MQ reason codes:
    
      MQRC_CONNECTION_QUIESCING (2202)
    
    and
    
      MQRC_Q_MGR_QUIESCING (2161)
    
    The application attempts to disconnect and invokes reconnect
    logic.  However, the application is not able to reconnect and a
    hang in the client occurs.
    
    A Javacore of the application JVM shows the threads similar to
    those below, indicating this problem has occurred:
    
    "JMSCCThreadPoolWorker-##" #27 daemon prio=5 os_prio=0
    tid=0x00007ff15c013800 nid=0x6b1b in Object.wait()
    [0x00007ff199bf9000]
       java.lang.Thread.State: TIMED_WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            at
    com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveAsyncTsh
            - locked <0x0000000790fa39e8> (a
    com.ibm.mq.jmqi.remote.impl.RemoteSession$AsyncTshLock)
            at com.ibm.mq.jmqi.remote.impl.RemoteSession.receiveTSH
            at
    com.ibm.mq.jmqi.remote.impl.RemoteSession.startConversation
            at
    com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.sessio
    nFromEligible
            at
    com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes
    sionFromEligibleConnection
            - locked <0x00000007209f7f68> (a
    com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification$Connec
    tionsLock)
            at
    com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSes
    sion
            at
    com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession
            at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect
            at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect
            at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect
            at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect
            at com.ibm.msg.client.wmq.internal.WMQConnection.<init>
            at
    com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7Pr
    oviderConnection
            at
    com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProv
    iderConnection
            at
    com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createCon
    nection
            at
    com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConn
    ection
            at
    com.ibm.mq.jms.MQConnectionFactory.createCommonConnection
            at
    com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection
    
    
    "RcvThread:
    com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection@1544173434[qmid=
    CANNED_DATA,fap=13,channel=MY.SVRCONN,ccsid=500,sharecnv=10,hbin
    t=300,peer=localhost/127.0.0.1(1414),localport=18785,ssl=no]"
    #19 daemon prio=5 os_prio=0 tid=0x00007ff15c004800 nid=0x6ad6
    waiting for monitor entry [0x00007ff19a2ff000]
       java.lang.Thread.State: BLOCKED (on object monitor)
            at
    com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSession
            - waiting to lock <0x00000007209f7f68> (a
    com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification$Connec
    tionsLock)
            at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run
            at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTas
    k
            at
    com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.
    runItem
            at
    com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run
    

Local fix

  • Set the sharing conversations (SHARECNV) value on the
    server-connection channel used by the application to the value
    1.
    

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of the:
    
      - IBM MQ V8 and V9 classes for JMS
      - IBM MQ V8 and V9 classes for Java
      - IBM MQ V8 and V9 JCA resource adapter
      - IBM MQ V8 and V9 OSGi bundles
      - IBM MQ V8 and V9 Managed File Transfer
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    An MQ queue manager can send notifications to connected clients
    requesting a controlled closure of connection handles
    (conversations), for example when it is being quiesced.  When
    this occurred, the Java message queueing interface (JMQI)
    received the notification on its "Remote Receive Thread" and
    would attempt to close the conversation and disassociate it from
    the connection it was using.
    
    If an application thread was simultaneously requesting a new
    conversation with the same connection details as used by the
    conversation being ended, then the JMQI may try and allocate the
    new conversation over an existing connection.  This is known as
    multiplexing and is primarily controlled by the Sharing
    Conversations (SHARECNV) server/client-connection channel
    attribute.
    
    In the case where these two events occurred concurrently, the
    application thread that was attempting to start a new
    conversation (for example creating a new MQQueueManager in the
    classes for Java or a JMS Connection, JMS Session or JMS Context
    in the classes for JMS) would take a lock on an internal
    "RemoteConnectionSpecification" object.  It would then send a
    data flow to the queue manager over the existing remote socket
    connection to request a new conversation be allocated and wait
    for the Remote Receive Thread to pass it the response as
    received from the queue manager.  However, if the Remote Receive
    Thread was attempting to end a conversation, then it would
    request the lock for the internal RemoteConnectionSpecification
    object.
    
    The Remote Receive Thread in this case would not be able to
    obtain the lock on the internal RemoteConnectionSpecification
    object, because it was held by the application thread that was
    starting a new conversation.  The application thread would only
    release the lock on the RemoteConnectionSpecification object
    when it was given the response to its conversation start request
    by the Remote Receive Thread, which would not occur because this
    thread was waiting for the lock on the
    RemoteConnectionSpecification object.
    

Problem conclusion

  • The logic regarding the locking of the internal
    RemoteConnectionSpecification object has been updated in the
    JMQI implementation to prevent the deadlock scenario described
    from occurring.
    
    If your Java application uses the Java Security Manager, after
    this APAR has been applied, it will be necessary to add the
    following additional permission to the java.security.policy file
    used by the application:
    
        permission java.lang.RuntimePermission "modifyThread";
    
    Without the addition of the above RuntimePermission, the
    following exception may be thrown:
    
    Exception in thread "main" java.security.AccessControlException:
    Access denied ("java.lang.RuntimePermission" "modifyThread")
    at java.security.AccessController.throwACE
    at java.security.AccessController.checkPermissionHelper
    at java.security.AccessController.checkPermission
    at java.lang.SecurityManager.checkPermission
    at java.util.concurrent.ThreadPoolExecutor.checkShutdownAccess
    at java.util.concurrent.ThreadPoolExecutor.shutdownNow
    at
    java.util.concurrent.Executors$DelegatedExecutorService.shutdown
    Now
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection$2.run
    at java.security.AccessController.doPrivileged
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.disconnect
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.removeSession
    at com.ibm.mq.jmqi.remote.impl.RemoteSession.disconnect
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.completeClose
    at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQDISC
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v8.0       8.0.0.9
    v9.0 CD    9.0.5
    v9.0 LTS   9.0.0.3
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT22127

  • Reported component name

    WMQ BASE MULTIP

  • Reported component ID

    5724H7251

  • Reported release

    800

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-08-24

  • Closed date

    2017-11-28

  • Last modified date

    2017-12-01

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WMQ BASE MULTIP

  • Fixed component ID

    5724H7251

Applicable component levels

  • R800 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSYHRD","label":"IBM MQ"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0.0.0","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
01 December 2017