IBM Support

PM02852: HIGH NUMBER OF INVALIDATION REQUESTS CAN CAUSE A DEADLOCK IN THE SERVER.

A fix is available

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • When a high number of invalidation requests are being processed
    in a multi-Servant environment, a deadlock can occur which
    causes the server to become unresponsive and hang.
    .
    The customer was running a Portal server with 5 Servants. Portal
    pages were being accessed which also cause invalidations to flow
    .
    
    When an item is invalidated in the cache, the invalidation
    message is sent from the originating SR to the CR, which proxies
    it to ALL the SRs (excluding the origin SR) synchronously.
    If there are 5 servants, the flow is ... SR1 --> CR --> (SR2,
    SR3, SR4, SR5). If the call from one of the target SRs does not
    return, the original SR1 call is also hung.
    .
    ORB threads are used in the SR to send invalidations to the CR,
    as well as receive the proxied calls from other servants.
    If there is a lot of activity, the ORB thread used by the CR to
    proxy the invalidations to the other SRs can seem to be hung
    because the SR does not return. This can happen, if the
    DRSServantProxy call gets stuck in the SR waiting on locks
    owned by the other ORB threads sending invalidation requests to
    the other servants.
    .
    The CR might show the majority of its threads trying to proxy
    invalidation requests to the other servants:
    com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest
    Native Method
    com/ibm/ws390/orb/ClientDelegate.invoke
    (Compiled Code)
    org/omg/CORBA/portable/ObjectImpl._invoke
    ObjectImpl.java(Inlined Compiled Code)
    com/ibm/ws/drs/ws390/proxy/servant/_DRSServantProxyStub.updateEn
    tryProp             (Compiled Code)
    com/ibm/ws/drs/ws390/DRSServantProxyWrapper.updateEntryProp
    DRSServantProxyWrapper.java(Inlined Compiled Code)
    com/ibm/ws/cache/drs/ws390/DynacacheDRSControllerDataXfer.update
    EntryProp           DynacacheDRSControllerDataXfer.java(Compiled
    Code)
    com/ibm/ws/drs/ws390/proxy/controller/DRSControllerProxyHelper.u
    pdateEntryProp      DRSControllerProxyHelper.java(Compiled Code)
    com/ibm/ws/drs/ws390/proxy/controller/DRSControllerProxyImpl.upd
    ateEntryProp        DRSControllerProxyImpl.java(Inlined Compiled
    Code)
    com/ibm/ws/drs/ws390/proxy/controller/_DRSControllerProxyImpl_Ti
    e.updateEntryProp   _DRSControllerProxyImpl_Tie.java(Compiled
    Code)
    com/ibm/ws/drs/ws390/proxy/controller/_DRSControllerProxyImpl_Ti
    e._invoke           _DRSControllerProxyImpl_Tie.java(Compiled
    Code)
    sun/reflect/GeneratedMethodAccessor8.invoke
    (Compiled Code)
    sun/reflect/DelegatingMethodAccessorImpl.invoke
    DelegatingMethodAccessorImpl.java(Compiled Code)
    java/lang/reflect/Method.invoke
    Method.java(Compiled Code)
    com/ibm/ws390/orb/RIRProtectedProxyHandler.invoke
    (Compiled Code)
    $Proxy0._invoke
    (Compiled Code)
    com/ibm/ws390/orb/CommonBridge.invoke
    (Compiled Code)
    com/ibm/ws390/orb/ORBEJSBridge.invoke
    ORBEJSBridge.java(Inlined Compiled Code)
    com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo
    keParmSetter
    .
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  IBM WebSphere Application Server V6.1       *
    *                  users of dynacache with multiple servant    *
    *                  regions.                                    *
    ****************************************************************
    * PROBLEM DESCRIPTION: A deadlock occurs and the server        *
    *                      hangs when a dynacache user requests    *
    *                      a high number of invalidations and      *
    *                      there are multiple servant regions.     *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Data Replication Services (DRS) uses proxy calls to flow
    updates between the control region (CR) and servant regions
    (SR).  These proxy calls run on Object Request Broker (ORB)
    threads.  The SRs call back into the CR and a deadlock occurs
    because all the ORB threads are in use and a thread is not
    available to handle a proxied call from the SR.
    

Problem conclusion

  • Dynacache has been modified to batch invalidations on the SR
    and immediately return so the ORB thread is freed up.  The
    invalidations are then processed on a daemon thread at
    periodic intervals.
    
    APAR PM02952 is currently targeted for inclusion in Service
    Level (Fix Pack) 6.1.0.31 of WebSphere Application Server V6.1.
    
    Please refer to URL:
    //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
    for Fix Pack availability.
    

Temporary fix

Comments

APAR Information

  • APAR number

    PM02852

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    610

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2009-12-04

  • Closed date

    2010-01-20

  • Last modified date

    2010-06-03

  • APAR is sysrouted FROM one or more of the following:

    PM02163

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R610 PSY UK56378

       UP10/05/08 P F005

Fix is available

  • Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
10 February 2022