IBM Support

PH21305: HANG IN ADJUNCT REGION WHEN DEACTIVATING MQ RESOURCE ADAPTER

Subscribe to this APAR

By subscribing, you receive periodic emails alerting you to the status of the APAR, along with a link to the fix after it becomes available. You can track this item individually or track all items by product.

Notify me when this APAR changes.

Notify me when an APAR for this component changes.

 

APAR status

  • Closed as program error.

Error description

  • A hang can occur when trying to stop an application containing
an MDB bound to an MQ Activation Specification. The
deactivation of the endpoint is blocked in the throttle in the
Control Region Adjunct process.

A javacore of the control region adjunct process shows a
callstack of the form below:

3XMTHREADINFO      "WebSphere non-WLM Dispatch Thread
t=009ac248" J9VMThread:0x0000000039B3F100,
omrthread_t:0x000000500A328560,
java/lang/Thread:0x0000000086F90998, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x87,
isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x37EF6000, native
priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread
flags:0x00000101)
3XMCPUTIME               CPU usage total: 21.335000000 secs,
current category="Application"
3XMTHREADBLOCK     Waiting on:
com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083
BDF228 Owned by: <unowned>
3XMHEAPALLOC             Heap bytes allocated since last GC
cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:189(Compiled Code))
at
com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire(ReentrantMute
x.java:167)
   (entered lock:
com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083
BDF228, entry count: 2)
at
com/ibm/mq/jmqi/remote/util/ReentrantMutex.acquire(ReentrantMute
x.java:73)
   (entered lock:
com/ibm/mq/jmqi/remote/api/RemoteHconn$DispatchLock@0x0000000083
BDF228, entry count: 1)
at
com/ibm/mq/jmqi/remote/api/RemoteHconn.requestDispatchLock(Remot
eHconn.java:1198)
at
com/ibm/mq/jmqi/remote/api/RemoteFAP.MQCTL(RemoteFAP.java:2548)
at
com/ibm/mq/jmqi/monitoring/JmqiInterceptAdapter.MQCTL(JmqiInterc
eptAdapter.java:333)
at
com/ibm/msg/client/wmq/internal/WMQConsumerOwnerShadow.suspendAs
yncService(WMQConsumerOwnerShadow.java:523)
at
com/ibm/msg/client/wmq/internal/WMQConnection.suspendAsyncServic
e(WMQConnection.java:2308)
at
com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu
merShadow.java:366)
at
com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu
merShadow.java:319)
at
com/ibm/msg/client/wmq/internal/WMQConsumerShadow.close(WMQConsu
merShadow.java:308)
at
com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti
onBrowserShadow.close(WMQConnectionBrowser.java:488)
at
com/ibm/msg/client/wmq/internal/WMQConnectionBrowser.close(WMQCo
nnectionBrowser.java:1518)
at
com/ibm/msg/client/jms/internal/JmsConnectionBrowserImpl.close(J
msConnectionBrowserImpl.java:250)
at
com/ibm/msg/client/jms/internal/JmsConnectionBrowserImpl.close(J
msConnectionBrowserImpl.java:204)
at
com/ibm/mq/jms/MQConnectionBrowser.close(MQConnectionBrowser.jav
a:80)
at
com/ibm/mq/connector/inbound/MessageEndpointDeployment.stopDeliv
eryASF(MessageEndpointDeployment.java:630)
at
com/ibm/mq/connector/inbound/MessageEndpointDeployment.stop(Mess
ageEndpointDeployment.java:537)
at
com/ibm/mq/connector/ResourceAdapterImpl.endpointDeactivation(Re
sourceAdapterImpl.java:680)
   (entered lock: java/util/HashMap@0x0000000081F4F3B0, entry
count: 1)
at
com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateUnderRAClass
LoaderContext(ActivationSpecWrapperImpl.java:513)
at
com/ibm/ejs/j2c/ActivationSpecWrapperImpl.deactivateEndPoint(Act
ivationSpecWrapperImpl.java:433)
at
com/ibm/ejs/j2c/RAWrapperImpl.deactivateEndpoint(RAWrapperImpl.j
ava:1486)
   (entered lock:
com/ibm/ejs/j2c/RAWrapperImpl@0x0000000082206DE8, entry count:
2)
at
com/ibm/ejs/j2c/RALifeCycleManagerImpl.deactivateEndPoint(RALife
CycleManagerImpl.java:1918)
   (entered lock:
com/ibm/ejs/j2c/RAWrapperImpl@0x0000000082206DE8, entry count:
1)
at
com/ibm/ejs/container/MessageEndpointFactoryImpl.deactivateEndpo
int(MessageEndpointFactoryImpl.java:237)
...

The callstacks of one or more other threads show the thread
stuck in the throttle, for example:

3XMTHREADINFO      "DispatchThread:
[com.ibm.mq.jmqi.remote.impl.RemoteSession[:/f76f99c1][connectio
nId=C3E2D8C3D4D8E9C54040404040404040D7563E14B92C0001]]"
J9VMThread:0x00000000346AF000, omrthread_t:0x000000500A1B7B40,
java/lang/Thread:0x0000000087E1B968, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x5E,
isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x327A1800, native
priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread
flags:0x00000101)
3XMCPUTIME               CPU usage total: 0.026000000 secs,
current category="Application"
3XMTHREADBLOCK     Waiting on:
com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences@0
x0000000083BDE6E0 Owned by: <unowned>
3XMHEAPALLOC             Heap bytes allocated since last GC
cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:189(Compiled Code))
at
com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences.p
reDispatch(EndpointInflightMessageReferences.java:297)
   (entered lock:
com/ibm/ws/wmqra/zdispatcher/EndpointInflightMessageReferences@0
x0000000083BDE6E0, entry count: 1)
at
com/ibm/ws/wmqra/zdispatcher/InflightMessageReferenceManager.pre
Dispatch(InflightMessageReferenceManager.java:228)
at
com/ibm/ws/wmqra/zdispatcher/WASAdjunctMessageReferenceHandler.h
andleMessageReference(WASAdjunctMessageReferenceHandler.java:185
)
   (entered lock:
com/ibm/ws/wmqra/zdispatcher/WASAdjunctMessageReferenceHandler@0
x0000000083BDE738, entry count: 1)
at
com/ibm/mq/jms/MQMessageReferenceHandler.handleMessageReference(
MQMessageReferenceHandler.java:82)
at
com/ibm/msg/client/jms/internal/JmsConnectionImpl$JmsProviderMes
sageRefHandler.handleMessageReference(JmsConnectionImpl.java:195
8)
at
com/ibm/msg/client/wmq/internal/WMQConnectionBrowser$WMQConnecti
onBrowserShadow.consumer(WMQConnectionBrowser.java:947)
at
com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.driveConsumer(Remot
eProxyQueue.java:4367)
at
com/ibm/mq/jmqi/remote/impl/RemoteProxyQueue.deliverMsgs(RemoteP
roxyQueue.java:4961)
at
com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgsReco
nnectable(RemoteDispatchThread.java:532)
at
com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.deliverMsgs(Rem
oteDispatchThread.java:505)
at
com/ibm/mq/jmqi/remote/impl/RemoteDispatchThread.run(RemoteDispa
tchThread.java:275)
at
com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.runTas
k(WorkQueueItem.java:314)
at
com/ibm/msg/client/commonservices/workqueue/SimpleWorkQueueItem.
runItem(SimpleWorkQueueItem.java:99)
at
com/ibm/msg/client/commonservices/workqueue/WorkQueueItem.run(Wo
rkQueueItem.java:336)
at
com/ibm/ws/wmqcsi/workqueue/WorkQueueManagerImpl$WorkQueueRunnab
le.run(WorkQueueManagerImpl.java:550)
at java/lang/Thread.run(Thread.java:818)

The problem was not reproducable when the alternate
consumer counting mechanism for the throttle, introduced in
APAR PI61894, was in use (specified by setting the
com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount
system property to true).

Local fix

  • WebSphere Variable
com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount set to a value
of 'true', as described in PI61894 could avoid this.

Problem summary

  • ****************************************************************
* USERS AFFECTED:  WebSphere Application Server users of       *
*                  WebSphere MQ Activation Specifications      *
****************************************************************
* PROBLEM DESCRIPTION: The process to stop an application      *
*                      may hang if it contains MDBs bound to   *
*                      the runtime using WMQ Activation        *
*                      Specifications.                         *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
Applications containing MDBs bound to the runtime via WMQ
Activation Specifications listen in the control region adjunct
process which contains a throttle.  When such an application
is stopped the WMQ resource adapter will wait for its existing
processing to stop, this includes threads that are calling the
MessageRefenceHandler in the control region adjunct that
listens and throttles the inbound work.  A timing related bug
meant that the threads stuck in the throttle could remain
permanently stuck when stopping the listener, in turn hanging
the stop of the listener itself.

Problem conclusion

  • A mechanism was added to the throttle to track the activation
state of the Activation Specification Endpoint so ensure that
the throttle was released correctly during endpoint
deactivation.

The fix for this APAR is targeted for inclusion in fix pack
9.0.5.4.
For more information, see 'Recommended Updates for WebSphere
Application Server':
https://www.ibm.com/support/pages/node/715553

Temporary fix

  • The problem was not reproducable when the alternate
consumer counting mechanism for the throttle, introduced in
APAR PI61894, was in use (specified by setting the
com.ibm.ws.wmqra.ThrottleOnActiveConsumerCount
system property to true).

Comments

  • 



APAR Information




    

  • APAR number
    PH21305
    • 

  • Reported component name
    WEBSPHERE FOR Z
    • 

  • Reported component ID
    5655I3500
    • 

  • Reported release
    900
    • 

  • Status
    CLOSED  PER
    • 

  • PE
    NoPE
    • 

  • HIPER
    NoHIPER
    • 

  • Special Attention
    NoSpecatt / Xsystem
    • 

  • Submitted date
    2020-01-22
    • 

  • Closed date
    2020-06-15
    • 

  • Last modified date
    2020-06-15
    • 





    

  • APAR is sysrouted FROM one or more of the following:
    

    • 

  • APAR is sysrouted TO one or more of the following:
    

    • 



Fix information


    

  • Fixed component name
    WEBSPHERE FOR Z
    • 

  • Fixed component ID
    5655I3500
    • 



Applicable component levels


    








      
              
                            

              

              [{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"900","Line of Business":{"code":"LOB45","label":"Automation"}}]
              

                          

          
 
        

      

    

      

        

          

            
Document Information


            

            


                        

            Modified date:
            

            16 June 2020
            

            
                      

        


        

        


      

    

  



    

  

  
    
            

          
Page Feedback