IBM Support

PH21305: HANG IN ADJUNCT REGION WHEN DEACTIVATING MQ RESOURCE ADAPTER

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • A hang can occur when trying to stop an application containing
    an MDB bound to an MQ Activation Specification. The
    deactivation of the endpoint is blocked in the throttle in the
    Control Region Adjunct process.
    
    A javacore of the control region adjunct process shows a
    callstack of the form below:
    
    3XMTHREADINFO      "WebSphere non-WLM Dispatch Thread
    t=009ac248" J9VMThread:0x0000000039B3F100,
    omrthread_t:0x000000500A328560,
    java/lang/Thread:0x0000000086F90998, state:CW, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x87,
    isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x37EF6000, native
    priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread
    flags:0x00000101)
    3XMCPUTIME               CPU usage total: 21.335000000 secs,
    current category="Application"
    3XMTHREADBLOCK     Waiting on:
    com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083
    BDF228 Owned by: <unowned>
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0 (0x0)
    3XMTHREADINFO3           Java callstack:
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:189(Compiled Code))
    at
    com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire(ReentrantMute
    x.java:167)
       (entered lock:
    com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083
    BDF228, entry count: 2)
    at
    com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire(ReentrantMute
    x.java:73)
       (entered lock:
    com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083
    BDF228, entry count: 1)
    at
    com/ibm/mq/jmqi/remote/api/RemoteHconn.requestDispatchLock(Remot
    eHconn.java:1198)
    at
    com/ibm/mq/jmqi/remote/api/RemoteFAP.MQCTL(RemoteFAP.java:2548)
    at
    com/ibm/mq/jmqi/monitoring/JmqiInterceptAdapter.MQCTL(JmqiInterc
    eptAdapter.java:333)
    at
    com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.suspendAs
    yncService(WMQConsumerOwnerShadow.java:523)
    at
    com/ibm/msg/client/wmq/internal/WMQConnection.suspendAsyncServic
    e(WMQConnection.java:2308)
    at
    com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu
    merShadow.java:366)
    at
    com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu
    merShadow.java:319)
    at
    com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu
    merShadow.java:308)
    at
    com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti
    onBrowserShadow.close(WMQConnectionBrowser.java:488)
    at
    com/ibm/msg/client/wmq/internal/WMQConnectionBrowser.close(WMQCo
    nnectionBrowser.java:1518)
    at
    com/ibm/msg/client/jms/internal/JmsConnectionBrowserImpl.close(J
    msConnectionBrowserImpl.java:250)
    at
    com/ibm/msg/client/jms/internal/JmsConnectionBrowserImpl.close(J
    msConnectionBrowserImpl.java:204)
    at
    com/ibm/mq/jms/MQConnectionBrowser.close(MQConnectionBrowser.jav
    a:80)
    at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv
    eryASF(MessageEndpointDeployment.java:630)
    at
    com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop(Mess
    ageEndpointDeployment.java:537)
    at
    com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation(Re
    sourceAdapterImpl.java:680)
       (entered lock: java/util/HashMap@0x0000000081F4F3B0, entry
    count: 1)
    at
    com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass
    LoaderContext(ActivationSpecWrapperImpl.java:513)
    at
    com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint(Act
    ivationSpecWrapperImpl.java:433)
    at
    com/ibm/ejs/j2c/RAWrapperImpl.deactivateEndpoint(RAWrapperImpl.j
    ava:1486)
       (entered lock:
    com/ibm/ejs/j2c/RAWrapperImpl@0x0000000082206DE8, entry count:
    2)
    at
    com/ibm/ejs/j2c/RALifeCycleManagerImpl.deactivateEndPoint(RALife
    CycleManagerImpl.java:1918)
       (entered lock:
    com/ibm/ejs/j2c/RAWrapperImpl@0x0000000082206DE8, entry count:
    1)
    at
    com/ibm/ejs/container/MessageEndpointFactoryImpl.deactivateEndpo
    int(MessageEndpointFactoryImpl.java:237)
    ...
    
    The callstacks of one or more other threads show the thread
    stuck in the throttle, for example:
    
    3XMTHREADINFO      "DispatchThread:
    [com.ibm.mq.jmqi.remote.impl.RemoteSession[:/f76f99c1][connectio
    nId=C3E2D8C3D4D8E9C54040404040404040D7563E14B92C0001]]"
    J9VMThread:0x00000000346AF000, omrthread_t:0x000000500A1B7B40,
    java/lang/Thread:0x0000000087E1B968, state:CW, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x5E,
    isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x327A1800, native
    priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread
    flags:0x00000101)
    3XMCPUTIME               CPU usage total: 0.026000000 secs,
    current category="Application"
    3XMTHREADBLOCK     Waiting on:
    com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences@0
    x0000000083BDE6E0 Owned by: <unowned>
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0 (0x0)
    3XMTHREADINFO3           Java callstack:
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:189(Compiled Code))
    at
    com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences.p
    reDispatch(EndpointInflightMessageReferences.java:297)
       (entered lock:
    com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences@0
    x0000000083BDE6E0, entry count: 1)
    at
    com/ibm/ws/wmqra/zdispatcher/InflightMessageReferenceManager.pre
    Dispatch(InflightMessageReferenceManager.java:228)
    at
    com/ibm/ws/wmqra/zdispatcher/WASAdjunctMessageReferenceHandler.h
    andleMessageReference(WASAdjunctMessageReferenceHandler.java:185
    )
       (entered lock:
    com/ibm/ws/wmqra/zdispatcher/WASAdjunctMessageReferenceHandler@0
    x0000000083BDE738, entry count: 1)
    at
    com/ibm/mq/jms/MQMessageReferenceHandler.handleMessageReference(
    MQMessageReferenceHandler.java:82)
    at
    com/ibm/msg/client/jms/internal/JmsConnectionImpl$JmsProviderMes
    sageRefHandler.handleMessageReference(JmsConnectionImpl.java:195
    8)
    at
    com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti
    onBrowserShadow.consumer(WMQConnectionBrowser.java:947)
    at
    com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.driveConsumer(Remot
    eProxyQueue.java:4367)
    at
    com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.deliverMsgs(RemoteP
    roxyQueue.java:4961)
    at
    com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgsReco
    nnectable(RemoteDispatchThread.java:532)
    at
    com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgs(Rem
    oteDispatchThread.java:505)
    at
    com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.run(RemoteDispa
    tchThread.java:275)
    at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas
    k(WorkQueueItem.java:314)
    at
    com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
    runItem(SimpleWorkQueueItem.java:99)
    at
    com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo
    rkQueueItem.java:336)
    at
    com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
    le.run(WorkQueueManagerImpl.java:550)
    at java/lang/Thread.run(Thread.java:818)
    
    The problem was not reproducable when the alternate
    consumer counting mechanism for the throttle, introduced in
    APAR PI61894, was in use (specified by setting the
    com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount
    system property to true).
    

Local fix

  • WebSphere Variable
    com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount set to a value
    of 'true', as described in PI61894 could avoid this.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  WebSphere Application Server users of       *
    *                  WebSphere MQ Activation Specifications      *
    ****************************************************************
    * PROBLEM DESCRIPTION: The process to stop an application      *
    *                      may hang if it contains MDBs bound to   *
    *                      the runtime using WMQ Activation        *
    *                      Specifications.                         *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Applications containing MDBs bound to the runtime via WMQ
    Activation Specifications listen in the control region adjunct
    process which contains a throttle.  When such an application
    is stopped the WMQ resource adapter will wait for its existing
    processing to stop, this includes threads that are calling the
    MessageRefenceHandler in the control region adjunct that
    listens and throttles the inbound work.  A timing related bug
    meant that the threads stuck in the throttle could remain
    permanently stuck when stopping the listener, in turn hanging
    the stop of the listener itself.
    

Problem conclusion

  • A mechanism was added to the throttle to track the activation
    state of the Activation Specification Endpoint so ensure that
    the throttle was released correctly during endpoint
    deactivation.
    
    The fix for this APAR is targeted for inclusion in fix pack
    9.0.5.4.
    For more information, see 'Recommended Updates for WebSphere
    Application Server':
    https://www.ibm.com/support/pages/node/715553
    

Temporary fix

  • The problem was not reproducable when the alternate
    consumer counting mechanism for the throttle, introduced in
    APAR PI61894, was in use (specified by setting the
    com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount
    system property to true).
    

Comments

APAR Information

  • APAR number

    PH21305

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    900

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2020-01-22

  • Closed date

    2020-06-15

  • Last modified date

    2020-06-15

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"900","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
16 June 2020