A fix is available
APAR status
Closed as program error.
Error description
An app was updated/redeployed without quiescing the server. A deadlock was encountered which ended up causing the servant to timeout with ABENDSEC3 SEC3 RSN04130004. This can occur under the following circumstances A) the server has been configured to initialize all EJBs at application startup time or The server has a startup bean deployed in it or the server has an MDB deployed in it B) an app deployment is in progress C) inbound remote EJB requests are coming in to the server D) an EJB being invoked just so happens to overlap the EJB wrapper for an EJB just initializing Javacore from the failure contains: 1LKDEADLOCK Deadlock detected !!! NULL --------------------- NULL 2LKDEADLOCKTHR Thread "WebSphere:ORB.thread.pool t=007bd3e8" (0x5B3DB100) 3LKDEADLOCKWTR is waiting for: 4LKDEADLOCKMON sys_mon_t:0x572541C0 infl_mon_t: 0x57254200: 4LKDEADLOCKOBJ com/ibm/ejs/csi/J2EENameImpl@53711A30/53711A3C: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "WebSphere t=007bb718" (0x5B414E00) 3LKDEADLOCKWTR which is waiting for: 4LKDEADLOCKMON sys_mon_t:0x57703C4C infl_mon_t: 0x57703C8C: 4LKDEADLOCKOBJ com/ibm/ejs/util/cache/WrapperBucket@36A5B778/36A5B784: 3LKDEADLOCKOWN which is owned by: 2LKDEADLOCKTHR Thread "WebSphere:ORB.thread.pool t=007bd3e8" (0x5B3DB100) Call stack for "WebSphere:ORB.thread.pool t=007bd3e8" ====================================================== "WebSphere:ORB.thread.pool t=007bd3e8" (TID:0x5B3DB100, sys_thread_t:0x5B3DBAF0, state:B, native ID:0x3B5E9000) prio=5 at com/ibm/ejs/container/HomeRecord.getHomeAndInitialize( HomeRecord.java:467) at com/ibm/ejs/container/EJSContainer.getHomeWrapperCommon( EJSContainer.java:1273) at com/ibm/ejs/container/EJSContainer.getHomeInstance( EJSContainer.java:1182) at com/ibm/ejs/container/EJSContainer.startBean( EJSContainer.java:1168) at com/ibm/ws/runtime/component/EJBContainerImpl.startBean( EJBContainerImpl.java:3484) at com/ibm/ws/runtime/component/EJBContainerImpl. startAndBindHome(EJBContainerImpl.java:4761) at com/ibm/ws/runtime/component/EJBContainerImpl. initializeDeferredEJB(EJBContainerImpl.java:4546) at com/ibm/ejs/container/HomeOfHomes.getHome( HomeOfHomes.java:345) at com/ibm/ejs/container/HomeOfHomes.internalCreateWrapper( HomeOfHomes.java:481) at com/ibm/ejs/container/EJSContainer.createWrapper( EJSContainer.java:4396(Compiled Code)) at com/ibm/ejs/container/WrapperManager.faultOnKey( WrapperManager.java:528(Compiled Code)) at com/ibm/ejs/util/cache/Cache.findAndFault(Cache.java:497( Compiled Code)) at com/ibm/ejs/container/WrapperManager.keyToObject( WrapperManager.java:481(Compiled Code)) at com/ibm/ejs/oa/EJSOAImpl.keyToObject(EJSOAImpl.java:553( Compiled Code)) at com/ibm/ejs/oa/EJSRootOAImpl.keyToObject( EJSRootOAImpl.java:271(Compiled Code)) at com/ibm/ws390/orb/CommonBridge.invoke(Bytecode PC:777(Compiled Code)) at com/ibm/ws390/orb/ORBEJSBridge.invoke(Bytecode PC:6( Compiled Code)) at sun/reflect/GeneratedMethodAccessor234.invoke(Bytecode PC:24( Compiled Code)) at sun/reflect/DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43(Compiled Code)) at java/lang/reflect/Method.invoke(Method.java:618( Compiled Code)) at com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke. ORBEJSBridgeInvokeParmSetter( ORBEJSBridgeInvoke.java:159(Compiled Code)) at com/ibm/ws390/orb/CommonBridge.nativeRunApplicationThread( Native Method) at com/ibm/ws390/orb/CommonBridge.runApplicationThread( Bytecode PC:7) at com/ibm/ws/util/ThreadPool$ZOSWorker.run( ThreadPool.java:1697) Call stack for "WebSphere t=007bb718" ===================================== "WebSphere t=007bb718" (TID:0x5B414E00, sys_thread_t:0x5B412730, state:B, native ID:0x3B629D00) prio=5 at com/ibm/ejs/util/cache/Cache.findAndFault(Cache.java:463( Compiled Code)) at com/ibm/ejs/container/WrapperManager.getWrapper( WrapperManager.java:277(Compiled Code)) at com/ibm/ejs/container/EJSHome.getWrapper(EJSHome.java:776) at com/ibm/ejs/container/EJSContainer.loadBeanMetaData( EJSContainer.java:1697(Compiled Code)) at com/ibm/ejs/container/HomeOfHomes.loadBeanMetaData( HomeOfHomes.java:641) at com/ibm/ejs/container/HomeRecord.getHomeAndInitialize( HomeRecord.java:479) at com/ibm/ejs/container/EJSContainer.getHomeWrapperCommon( EJSContainer.java:1273) at com/ibm/ejs/container/EJSContainer.getHomeInstance( EJSContainer.java:1182) at com/ibm/ejs/container/EJSContainer.startBean( EJSContainer.java:1168) at com/ibm/ws/runtime/component/EJBContainerImpl.startBean( EJBContainerImpl.java:3484) at com/ibm/ws/runtime/component/EJBContainerImpl.install( EJBContainerImpl.java:2888(Compiled Code)) at com/ibm/ws/runtime/component/EJBContainerImpl.start( EJBContainerImpl.java:3730) at com/ibm/ws/runtime/component/ApplicationMgrImpl.start( ApplicationMgrImpl.java:1250(Compiled Code)) at com/ibm/ws/runtime/component/DeployedApplicationImpl. fireDeployedObjectStart(DeployedApplicationImpl.java:1140) at com/ibm/ws/runtime/component/DeployedModuleImpl.start( DeployedModuleImpl.java:572(Compiled Code)) at com/ibm/ws/runtime/component/DeployedApplicationImpl.start( DeployedApplicationImpl.java:819(Compiled Code)) at com/ibm/ws/runtime/component/ApplicationMgrImpl. startApplication( ApplicationMgrImpl.java:921(Compiled Code)) at com/ibm/ws/runtime/component/ApplicationMgrImpl$1.run( ApplicationMgrImpl.java:1437) at com/ibm/ws/security/auth/ContextManagerImpl.runAs( ContextManagerImpl.java:4158(Compiled Code)) at com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem( ContextManagerImpl.java:4255(Compiled Code)) at com/ibm/ws/security/core/SecurityContext.runAsSystem( SecurityContext.java:245) at com/ibm/ws/runtime/component/ApplicationMgrImpl. startApplication( ApplicationMgrImpl.java:1442(Compiled Code)) at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:79(Compiled Code)) at sun/reflect/DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43(Compiled Code)) at java/lang/reflect/Method.invoke(Method.java:618( Compiled Code)) at sun/reflect/misc/Trampoline.invoke(MethodUtil.java:68) at sun/reflect/GeneratedMethodAccessor238.invoke(Bytecode PC:40) at sun/reflect/DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43(Compiled Code)) at java/lang/reflect/Method.invoke(Method.java:618( Compiled Code)) at sun/reflect/misc/MethodUtil.invoke(MethodUtil.java:271) at javax/management/modelmbean/RequiredModelMBean.invokeMethod( RequiredModelMBean.java:1092) at javax/management/modelmbean/RequiredModelMBean.invoke( RequiredModelMBean.java:973) at com/sun/jmx/mbeanserver/DynamicMetaDataImpl.invoke( DynamicMetaDataImpl.java:231) at com/sun/jmx/mbeanserver/MetaDataImpl.invoke( MetaDataImpl.java:238) at com/sun/jmx/interceptor/DefaultMBeanServerInterceptor.invoke( DefaultMBeanServerInterceptor.java:833) at com/sun/jmx/mbeanserver/JmxMBeanServer.invoke( JmxMBeanServer.java:802) at com/ibm/ws390/management/connector/corba/CorbaConnectorImpl. invoke(CorbaConnectorImpl.java:705) at com/ibm/ws390/management/connector/corba/ _CorbaConnectorImplBase. _invoke(_CorbaConnectorImplBase.java:406) at com/ibm/ws390/orb/CommonBridge.invoke( Bytecode PC:1337(Compiled Code)) at com/ibm/ws390/orb/ORBEJSBridge.invoke(Bytecode PC:6( Compiled Code)) at sun/reflect/GeneratedMethodAccessor234.invoke( Bytecode PC:24(Compiled Code)) at sun/reflect/DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43(Compiled Code)) at java/lang/reflect/Method.invoke(Method.java:618( Compiled Code)) at com/ibm/ws390/orb/parameters/ORBEJSBridgeInvoke. ORBEJSBridgeInvokeParmSetter( ORBEJSBridgeInvoke.java:159(Compiled Code))
Local fix
quiesce workload before attempting to deploy a new version of the application.
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server with Enterprise Java Beans (EJBs) * **************************************************************** * PROBLEM DESCRIPTION: Application start with EJBs can * * deadlock if the first request to an * * EJB is made simultaneously. * **************************************************************** * RECOMMENDATION: * **************************************************************** Remote EJB request processing acquires a hash table bucket lock; if the remote request is the first call to that EJB, then an application-wide lock is acquired. During application start of non-deferred EJBs, such as MDBs or startup beans, the application-wide lock is acquired and then the hash table bucket lock is acquired. If these two events happen concurrently, a deadlock occurs.
Problem conclusion
The application start code path was changed to acquire the hash table bucket lock before the application-wide lock, which prevents the deadlock. APAR PM38277 is currently targeted for inclusion in Service Level (Fix Pack) 7.0.0.19 of WebSphere Application Server V7.0. Please refer to URL: //www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack availability.
Temporary fix
Comments
APAR Information
APAR number
PM38277
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-05-03
Closed date
2011-05-10
Last modified date
2011-10-04
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R700 PSY UK71297
UP11/09/10 P F109
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
10 February 2022