IBM Support

IT19154: Deadlock in the MQ resource adapter within WebSphere AppServer closing the ServerSessionPool after a broken connection.

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A deadlock can occur in the WebSphere MQ resource adapter while
    closing the ServerSessionPool associated with an activation
    specification.  A Javacore (thread dump) of the application
    server JVM shows the following two threads in a logical
    deadlock:
    
    "WebSphere non-WLM Dispatch Thread t=00abae88"
    J9VMThread:0x000000003BA58600, j9thread_t:0x000000480908B488,
    java/lang/Thread:0x000000008E5DE4C0, state:CW, prio=5
           (native thread ID:0x3B321000, native priority:0x5, native
    policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101)
    Waiting on:
    com/ibm/mq/connector/inbound/ServerSessionImpl$IsInUseLock@0x000
    000009304F648 Owned by: <unowned>
    Java callstack:
      at java/lang/Object.wait
      at java/lang/Object.wait
      at com/ibm/mq/connector/inbound/ServerSessionImpl.close
         (entered lock:
    com/ibm/mq/connector/inbound/ServerSessionImpl$IsInUseLock@0x000
    000009304F648, entry count: 1)
      at
    com/ibm/mq/connector/inbound/ServerSessionPoolImpl.closeInternal
      at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.close
      at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv
    eryASF
      at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop
      at
    com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation
         (entered lock: java/util/HashMap@0x0000000083E0D2C0, entry
    count: 1)
      at
    com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass
    LoaderContext
      at
    com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint
      at com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl.pause
         (entered lock:
    com/ibm/ejs/j2c/mbeans/MessageEndpointMBeanImpl@0x000000008B5454
    88, entry count: 1)
      at sun/reflect/NativeMethodAccessorImpl.invoke0
      ...
    
    
    "WebSphere WLM Dispatch Thread t=00ac20b8"
    J9VMThread:0x000000003B9BA300, j9thread_t:0x000000480916C4D0,
    java/lang/Thread:0x000000008E5BBAD0, state:CW, prio=5
           (java/lang/Thread getId:0x9E, isDaemon:true)
           (native thread ID:0x3A9C4400, native priority:0x5, native
    policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000101)
    Waiting on:
    com/ibm/mq/connector/inbound/ServerSessionPoolImpl$CloseState@0x
    000000009304F130 Owned by: <unowned>
    Java callstack:
      at java/lang/Object.wait
      at java/lang/Object.wait
      at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.close
         (entered lock:
    com/ibm/mq/connector/inbound/ServerSessionPoolImpl$CloseState@0x
    000000009304F130, entry count: 2)
      at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv
    eryASF
      at com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop
      at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.onExcepti
    on
         (entered lock:
    com/ibm/ejs/container/MessageEndpointFactoryImpl@0x000000008AF39
    900, entry count: 1)
      at com/ibm/mq/connector/inbound/AbstractWorkImpl.run
      at com/ibm/mq/connector/inbound/ServerSessionImpl.start
      at
    com/ibm/ws/wmqra/zdispatcher/WASServantConnectionConsumer.delive
    rMessagesFromReference
      at
    com/ibm/ws/wmqra/zdispatcher/WASServantConnectionConsumerReposit
    ory.dispatch
      at
    com/ibm/ws/wmqra/zdispatcher/WASServantDispatchTarget.dispatch
      at com/ibm/ws/sib/z/dispatcher/WorkObjectImpl.fireWork
      at
    com/ibm/ws390/sib/dispatcher/WorkObjectProxyImpl.dispatchWorkObj
    ectImpl
      ...
    

Local fix

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of the:
    
      - WebSphere MQ V7.1 resource adapter
      - WebSphere MQ V7.5 resource adapter
      - IBM MQ V8 resource adapter
      - IBM MQ V9 resource adapter
    
      - WebSphere Application Server V8.5 MQ messaging provider
      - WebSphere Application Server V9 MQ messaging provider
    
    who use activation specifications to deliver messages
    asynchronously to message-driven bean (MDB) applications.
    
    
    Platforms affected:
    MultiPlatform
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    A deadlock within the MQ resource adapter occurred in the
    scenario when:
    
      1) an XA transaction error was thrown (for example, because of
    a broken TCP/IP connection between the application server and
    queue manager) while a ServerSession thread was attempting to
    deliver a message to a message-driven bean (MDB) application
    (see the thread named "WebSphere WLM Dispatch Thread t=00ac20b8"
    in the Problem Description),
    
    and
    
      2) another thread was already attempting to stop the message
    endpoint and close the ServerSessionPool associated with an
    activation specification (see the thread named "WebSphere
    non-WLM Dispatch Thread t=00abae88" in the Problem Description).
    
    
    This was because the thread stopping the message endpoint was
    attempting to close off ServerSessions in the activation
    specification's ServerSessionPool. While doing this, it found a
    ServerSession that was being used to try and deliver a message
    to an MDB, so it blocked waiting for that ServerSession to
    finish and be returned to the ServerSessionPool before
    continuing to close it.  However, once the thread that was
    running the ServerSession experienced a XA error during message
    delivery it also attempted to perform some clean-up, before
    starting reconnection retry processing.  This included closing
    the ServerSessionPool associated with the activation
    specification.
    
    The thread running the ServerSession determined that another
    thread was already processing a close of the ServerSessionPool
    and so blocked, waiting for that operation to complete.
    However, this thread could not complete the pool closure because
    the ServerSession that was in use had not been returned to the
    pool.  As such, neither threads could progress.
    

Problem conclusion

  • The MQ resource adapter has been updated such that when a thread
    is closing an activation specification's ServerSessionPool it
    does not wait for ServerSessions that are processing a message
    delivery failure to be returned to the pool before they are
    closed.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v7.1       7.1.0.9
    v7.5       7.5.0.9
    v8.0       8.0.0.8
    v9.0 LTS   9.0.0.2
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT19154

  • Reported component name

    WMQ WINDOWS V7

  • Reported component ID

    5724H7220

  • Reported release

    710

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    YesHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2017-02-07

  • Closed date

    2017-06-30

  • Last modified date

    2017-06-30

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WMQ WINDOWS V7

  • Fixed component ID

    5724H7220

Applicable component levels

  • R710 PSY

       UP

[{"Line of Business":{"code":"LOB36","label":"IBM Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]

Document Information

Modified date:
08 March 2021