IBM Support

IT06660: WEBSPHERE APPSERVER ON Z/OS ACTIVATION SPEC DEADLOCKS WHILE BEING PAUSED

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A WebSphere MQ messaging provider activation specification in
    WebSphere Application Server for z/OS may hang while being
    paused due to a logical deadlock between two threads.
    
    In a Javacore you will see the following lock information:
    
    3LKMONOBJECT com/ibm/mq/connector/inbound/
     ServerSessionImpl$IsInUseLock@0x
     000000482E732E40/0x000000482E732E58:
    
    3LKNOTIFYQ Waiting to be notified:
    3LKWAITNOTIFY "WebSphere t=00ab8408"
     (0x0000004808CDFC00)
    2LKMONINUSE sys_mon_t:0x0000004812E59AC0 infl_mon_t:
     0x0000004812E59B30:
    
    This is followed by thread information where two threads are
    shown to be in the following state:
    
    WebSphere t=00ab8408" J9VMThread:0x0000004808CDFC00,
     Java callstack:
    at java/lang/Object.wait(Native Method)
    at java/lang/Object.wait(Object.java:167
     (Compiled Code))
    at com/ibm/mq/connector/inbound/
     ServerSessionImpl.close(ServerSessionImpl.java:299)
    at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.
     closeInternal(ServerSessionPoolImpl.java:648)
    at com/ibm/mq/connector/inbound/ServerSessionPoolImpl.
     close(ServerSessionPoolImpl.java:557)
    at com/ibm/mq/connector/inbound/MessageEndpointDeployment.
     stop(MessageEndpointDeployment.java:473)
    at com/ibm/mq/connector/ResourceAdapterImpl.
     endpointDeactivation(ResourceAdapterImpl.java:523)
    
    and
    
    "WebSphere:ORB.thread.pool t=00ac0528"
     Java callstack:
    at com/ibm/ws390/orb/ClientDelegate.
     invokeRequestCFW(Native Method)
    at com/ibm/ws390/orb/ClientDelegate.
     commonInvoke(ClientDelegate.java:1042
     (Compiled Code))
    at com/ibm/ws390/orb/ClientDelegate.
     invoke(ClientDelegate.java:889)
    at org/omg/CORBA/portable/ObjectImpl.
     _invoke(ObjectImpl.java:484)
    at com/ibm/ws390/management/connector/corba/
     _CorbaConnectorStub.invoke
     (_CorbaConnectorStub.java:634)
    at com/ibm/ws390/management/connector/corba/
     CorbaConnectorClient.invoke
     (CorbaConnectorClient.java:903)
    at com/ibm/ws390/management/InterProcessAdminService.
     invoke(InterProcessAdminService.java:371)
    at com/ibm/ws/management/AdminServiceImpl$1.
     run(AdminServiceImpl.java:1432)
    at com/ibm/ws/security/util/AccessController.
     doPrivileged(AccessController.java:118
     (Compiled Code))
    at com/ibm/ws/management/AdminServiceImpl.
     invoke(AdminServiceImpl.java:1224)
    at com/ibm/ws/wmqra/utils/EndpointPauser$1.run
     (EndpointPauser.java:99)
    at com/ibm/ws/sib/security/auth/AuthUtilsImpl$5.
     run(AuthUtilsImpl.java:1454)
    at com/ibm/ws/security/auth/ContextManagerImpl.
     runAs(ContextManagerImpl.java:5365)
    at com/ibm/ws/security/auth/ContextManagerImpl.
     runAsSystem(ContextManagerImpl.java:5453)
    at com/ibm/ws/sib/security/auth/AuthUtilsImpl.
     runAsSystem(AuthUtilsImpl.java:1449)
    at com/ibm/ws/wmqra/utils/EndpointPauser.
     pauseJ2CMessageEndpoint(EndpointPauser.java:96)
    at com/ibm/ws/wmqra/helper/WASRuntimeHelper.
     deliveryFailed(WASRuntimeHelper.java:326)
    at com/ibm/mq/connector/inbound/WorkImpl.
     callDeliveryFailed(WorkImpl.java:511)
    
    
    The Job Log might also report the threads as hung, as per
    WebSphere Application Server hung thread detection processing.
    For example:
    
    BBOO0221W: WSVR0605W: Thread "WebSphere WLM Dispatch Thread
    t=00ac0528" (00000011) has been active for 706600 milliseconds
    and may be hung. There is/are 1 thread(s) in total in the server
    that may be hung.
    

Local fix

Problem summary

  • ****************************************************************
    USERS AFFECTED:
    This issue affects users of:
    
    - The WebSphere Application Server for z/OS V7 WebSphere MQ
    messaging provider.
    - The WebSphere Application Server for z/OS V8 WebSphere MQ
    messaging provider.
    - The WebSphere Application Server for z/OS V8.5 WebSphere MQ
    messaging provider.
    
    who are using an activation specification to drive a
    Message-Driven Bean (MDB) application which consumes messages
    from WebSphere MQ.
    
    
    Platforms affected:
    z/OS
    
    ****************************************************************
    PROBLEM DESCRIPTION:
    The default configuration of an activation specification defined
    against the WebSphere MQ messaging provider within WebSphere
    Application Server is to stop the messaging endpoint on the
    first failure to deliver a message. This behaviour is
    configurable using the properties:
    
    - Stop endpoint if message delivery fails
    
    and
    
    - Number of sequential delivery failures before suspending
    endpoint
    
    
    Using the default configuration, if a Message-Driven Bean (MDB)
    application rolls a message back because it could not be
    processed, WebSphere Application Server will attempt to stop the
    activation specification. When this is requested, the activation
    specification waits for all current activity (included queued up
    work) to complete, either successfully or due to an error,
    before stopping.
    
    It was observed that when the activation specification on
    WebSphere Application Server for z/OS attempted to pause, it
    ended up hanging because of two Java threads were in a logical
    deadlock processing the stop request.
    
    The scenario was as follows.
    
    A thread that was running a Server Session and had driven the
    MDB application with a message requested the endpoint to stop.
    For WebSphere Application Server for z/OS, this thread was not
    necessarily the thread that performed the actual stop of the
    endpoint and the subsequent closure of the Server Session Pool
    associated with the activation specification. In this case, the
    thread associated with the Server Session waited for the
    deactivation of the endpoint to complete, while the Server
    Session was still marked as "in use". However, the thread that
    was processing the endpoint deactivation waited for all Server
    Sessions from the activation specification's Server Session Pool
    to complete their processing and to be returned to the pool.
    
    This resulted in the thread stopping the endpoint waiting for a
    Server Session to be returned to the pool, which was busy
    waiting for the processing of stopping the endpoint to complete.
    

Problem conclusion

  • The WebSphere MQ resource adapter code has been changed to
    ensure when WebSphere Application Server for z/OS requests a
    stop of the messaging endpoint (activation specification), the
    thread processing the stop request does not wait for the Server
    Session that requested the stop to complete before pausing.
    
    ---------------------------------------------------------------
    The fix is targeted for delivery in the following PTFs:
    
    Version    Maintenance Level
    v7.0       7.0.1.14
    v7.1       7.1.0.7
    v7.5       7.5.0.6
    v8.0       8.0.0.3
    
    The latest available maintenance can be obtained from
    'WebSphere MQ Recommended Fixes'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006037
    
    If the maintenance level is not yet available information on
    its planned availability can be found in 'WebSphere MQ
    Planned Maintenance Release Dates'
    http://www-1.ibm.com/support/docview.wss?rs=171&uid=swg27006309
    ---------------------------------------------------------------
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT06660

  • Reported component name

    WMQ WINDOWS V7

  • Reported component ID

    5724H7220

  • Reported release

    710

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2015-01-26

  • Closed date

    2015-03-30

  • Last modified date

    2015-03-30

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

    PI35752

Fix information

  • Fixed component name

    WMQ WINDOWS V7

  • Fixed component ID

    5724H7220

Applicable component levels

  • R710 PSY

       UP

[{"Line of Business":{"code":"LOB36","label":"IBM Automation"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFKSJ","label":"WebSphere MQ"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1"}]

Document Information

Modified date:
09 March 2021