A fix is available
APAR status
Closed as program error.
Error description
When a high number of invalidation requests are being processed in a multi-Servant environment, a deadlock can occur which causes the server to become unresponsive and hang. . The customer was running a Portal server with 5 Servants. Portal pages were being accessed which also cause invalidations to flow . When an item is invalidated in the cache, the invalidation message is sent from the originating SR to the CR, which proxies it to ALL the SRs (excluding the origin SR) synchronously. If there are 5 servants, the flow is ... SR1 --> CR --> (SR2, SR3, SR4, SR5). If the call from one of the target SRs does not return, the original SR1 call is also hung. . ORB threads are used in the SR to send invalidations to the CR, as well as receive the proxied calls from other servants. If there is a lot of activity, the ORB thread used by the CR to proxy the invalidations to the other SRs can seem to be hung because the SR does not return. This can happen, if the DRSServantProxy call gets stuck in the SR waiting on locks owned by the other ORB threads sending invalidation requests to the other servants. . The CR might show the majority of its threads trying to proxy invalidation requests to the other servants: com/ibm/ws390/orb/ClientDelegate.jorbInvokeRequest Native Method com/ibm/ws390/orb/ClientDelegate.invoke (Compiled Code) org/omg/CORBA/portable/ObjectImpl._invoke ObjectImpl.java(Inlined Compiled Code) com/ibm/ws/drs/ws390/proxy/servant/_DRSServantProxyStub.updateEn tryProp (Compiled Code) com/ibm/ws/drs/ws390/DRSServantProxyWrapper.updateEntryProp DRSServantProxyWrapper.java(Inlined Compiled Code) com/ibm/ws/cache/drs/ws390/DynacacheDRSControllerDataXfer.update EntryProp DynacacheDRSControllerDataXfer.java(Compiled Code) com/ibm/ws/drs/ws390/proxy/controller/DRSControllerProxyHelper.u pdateEntryProp DRSControllerProxyHelper.java(Compiled Code) com/ibm/ws/drs/ws390/proxy/controller/DRSControllerProxyImpl.upd ateEntryProp DRSControllerProxyImpl.java(Inlined Compiled Code) com/ibm/ws/drs/ws390/proxy/controller/_DRSControllerProxyImpl_Ti e.updateEntryProp _DRSControllerProxyImpl_Tie.java(Compiled Code) com/ibm/ws/drs/ws390/proxy/controller/_DRSControllerProxyImpl_Ti e._invoke _DRSControllerProxyImpl_Tie.java(Compiled Code) sun/reflect/GeneratedMethodAccessor8.invoke (Compiled Code) sun/reflect/DelegatingMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java(Compiled Code) java/lang/reflect/Method.invoke Method.java(Compiled Code) com/ibm/ws390/orb/RIRProtectedProxyHandler.invoke (Compiled Code) $Proxy0._invoke (Compiled Code) com/ibm/ws390/orb/CommonBridge.invoke (Compiled Code) com/ibm/ws390/orb/ORBEJSBridge.invoke ORBEJSBridge.java(Inlined Compiled Code) com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke.ORBEJSBridgeInvo keParmSetter .
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: IBM WebSphere Application Server V6.1 * * users of dynacache with multiple servant * * regions. * **************************************************************** * PROBLEM DESCRIPTION: A deadlock occurs and the server * * hangs when a dynacache user requests * * a high number of invalidations and * * there are multiple servant regions. * **************************************************************** * RECOMMENDATION: * **************************************************************** Data Replication Services (DRS) uses proxy calls to flow updates between the control region (CR) and servant regions (SR). These proxy calls run on Object Request Broker (ORB) threads. The SRs call back into the CR and a deadlock occurs because all the ORB threads are in use and a thread is not available to handle a proxied call from the SR.
Problem conclusion
Dynacache has been modified to batch invalidations on the SR and immediately return so the ORB thread is freed up. The invalidations are then processed on a daemon thread at periodic intervals. APAR PM02952 is currently targeted for inclusion in Service Level (Fix Pack) 6.1.0.31 of WebSphere Application Server V6.1. Please refer to URL: //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack availability.
Temporary fix
Comments
APAR Information
APAR number
PM02852
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
610
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2009-12-04
Closed date
2010-01-20
Last modified date
2010-06-03
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R610 PSY UK56378
UP10/05/08 P F005
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"6.1","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
10 February 2022