Fixes are available
8.5.5.9: WebSphere Application Server V8.5.5 Fix Pack 9
8.5.5.10: WebSphere Application Server V8.5.5 Fix Pack 10
8.5.5.11: WebSphere Application Server V8.5.5 Fix Pack 11
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
APAR status
Closed as program error.
Error description
The server startup can hang and the javacore reports a deadlock like below : Deadlock detected !!! --------------------- Thread "P=476746:O=0:CT" (0x00000000C0000100) is waiting for: sys_mon_t:0x000001001D0AE088 infl_mon_t: 0x000001001D0AE0D0: com/ibm/ejs/j2c/RAWrapperImpl@0x000000005176CF80 which is owned by: Thread "HAManager.thread.pool : 1" (0x00000000C33A2800) which is waiting for: sys_mon_t:0x0000010012F33558 infl_mon_t: 0x0000010012F335A0: java/util/Hashtable@0x00000000438BDF40 which is owned by: Thread "P=476746:O=0:CT" (0x00000000C0000100)
Local fix
set com.ibm.ejs.j2c.J2CServerListener.deferEndpointActivation = false Please notice that setting this property to false may lead to antoher problem that the adapters may not start propertly.
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server * **************************************************************** * PROBLEM DESCRIPTION: Server startup can be affected due to * * synchroniztion issue in J2C leading * * to deadlock * **************************************************************** * RECOMMENDATION: * **************************************************************** There is a possibility of a deadlock situation during application server startup due to synchronization issues in the J2C component. Capturing javacores (thread dumps) can help to identify the deadlock like below : Deadlock detected !!! --------------------- Thread "P=476746:O=0:CT" (0x00000000C0000100) is waiting for: sys_mon_t:0x000001001D0AE088 infl_mon_t: 0x000001001D0AE0D0: com/ibm/ejs/j2c/RAWrapperImpl@0x000000005176CF80 which is owned by: Thread "HAManager.thread.pool : 1" (0x00000000C33A2800) which is waiting for: sys_mon_t:0x0000010012F33558 infl_mon_t: 0x0000010012F335A0: java/util/Hashtable@0x00000000438BDF40 which is owned by: Thread "P=476746:O=0:CT" (0x00000000C0000100) "P=476746:O=0:CT" J9VMThread:0x00000000C0000100, j9thread_t:0x000001001013A540, java/lang/Thread:0x0000000040AB68C0, state:B, prio=5 (java/lang/Thread getId:0x1, isDaemon:false) (native thread ID:0x2260057, native priority:0x5, native policy:UNKNOWN) CPU usage total: 52.556231000 secs, user: 42.696168000 secs, system: 9.860063000 secs Blocked on: com/ibm/ejs/j2c/RAWrapperImpl@0x000000005176CF80 Owned by: "HAManager.thread.pool : 1" (J9VMThread:0x00000000C33A2800, java/lang/Thread:0x0000000046183D40) Heap bytes allocated since last GC cycle=0 (0x0) Java callstack: at com/ibm/ejs/j2c/J2CServerListener.resumeInboundRAs(J2CServerList ener.java:216) (entered lock: java/util/Hashtable@0x00000000438BDF40, entry count: 1) at com/ibm/ejs/j2c/J2CServerListener.propertyChange(J2CServerListen er.java:150) (entered lock: java/lang/Object@0x0000000043227388, entry count: 1) at java/beans/PropertyChangeSupport.firePropertyChange(PropertyChan geSupport.java:267(Compiled Code)) at java/beans/PropertyChangeSupport.firePropertyChange(PropertyChan geSupport.java:275(Compiled Code)) at java/beans/PropertyChangeSupport.firePropertyChange(PropertyChan geSupport.java:59) at com/ibm/wsspi/runtime/component/WsComponentImpl.setState(WsCompo nentImpl.java:562) at com/ibm/ws/runtime/component/ComponentImpl.setState(ComponentImp l.java:144) at com/ibm/ws/runtime/component/ComponentImpl.setStartState(Compone ntImpl.java:164) at com/ibm/ws/runtime/component/ServerImpl.start(ServerImpl.java:53 0) at com/ibm/ws/runtime/WsServerImpl.bootServerContainer(WsServerImpl .java:311) (entered lock: com/ibm/ws/runtime/component/ServerImpl@0x0000000041D06808, entry count: 1) at com/ibm/ws/runtime/WsServerImpl.start(WsServerImpl.java:224) at com/ibm/ws/runtime/WsServerImpl.main(WsServerImpl.java:697) at com/ibm/ws/runtime/WsServer.main(WsServer.java:59) at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessor Impl.java:60) at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:37(Compiled Code)) at java/lang/reflect/Method.invoke(Method.java:611(Compiled Code)) at com/ibm/wsspi/bootstrap/WSLauncher.launchMain(WSLauncher.java:23 4) at com/ibm/wsspi/bootstrap/WSLauncher.main(WSLauncher.java:96) at com/ibm/wsspi/bootstrap/WSLauncher.run(WSLauncher.java:77) at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessor Impl.java:60) at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:37) at java/lang/reflect/Method.invoke(Method.java:611) at org/eclipse/equinox/internal/app/EclipseAppContainer.callMethodW ithException(EclipseAppContainer.java:587) at org/eclipse/equinox/internal/app/EclipseAppHandle.run(EclipseApp Handle.java:198) at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.run Application(EclipseAppLauncher.java:110) at org/eclipse/core/runtime/internal/adaptor/EclipseAppLauncher.sta rt(EclipseAppLauncher.java:79) at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStart er.java:369) at org/eclipse/core/runtime/adaptor/EclipseStarter.run(EclipseStart er.java:179) at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessor Impl.java:60) at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:37) at java/lang/reflect/Method.invoke(Method.java:611) at org/eclipse/core/launcher/Main.invokeFramework(Main.java:340) at org/eclipse/core/launcher/Main.basicRun(Main.java:282) at org/eclipse/core/launcher/Main.run(Main.java:981) at com/ibm/wsspi/bootstrap/WSPreLauncher.launchEclipse(WSPreLaunche r.java:398) at com/ibm/wsspi/bootstrap/WSPreLauncher.main(WSPreLauncher.java:16 1) HAManager.thread.pool : 1" J9VMThread:0x00000000C33A2800, j9thread_t:0x00000100180CABC0, java/lang/Thread:0x0000000046183D40, state:B, prio= (java/lang/Thread getId:0x89, isDaemon:true) (native thread ID:0x3F10091, native priority:0x5, native policy:UNKNOWN) CPU usage total: 6.453545000 secs, user: 3.696306000 secs, system: 2.757239000 secs locked on: java/util/Hashtable@0x00000000438BDF40 Owned by: "P=476746:O=0:CT" (J9VMThread:0x00000000C0000100, java/lang/Thread:0x0000000040A Heap bytes allocated since last GC cycle=0 (0x0) Java callstack: at com/ibm/ejs/j2c/RALifeCycleManagerImpl.getResourceAdapterInstanc e(RALifeCycleManagerImpl.java:2353) at com/ibm/ejs/j2c/ConnectionFactoryBuilderServerImpl.createMCFandP M(ConnectionFactoryBuilderServerImpl.java:570) at com/ibm/ejs/j2c/ConnectionFactoryBuilderServerImpl.processObject Instance(ConnectionFactoryBuilderServerImpl.java:1067) at com/ibm/ejs/j2c/ServerFunction.processObjectInstance(ServerFunct ion.java:1517) at com/ibm/ejs/j2c/ConnectionFactoryBuilderImpl.getObjectInstance(C onnectionFactoryBuilderImpl.java:659) at org/apache/aries/jndi/ObjectFactoryHelper.getObjectInstanceUsing ObjectFactoryBuilders(ObjectFactoryHelper.java:351) at org/apache/aries/jndi/ObjectFactoryHelper.getObjectInstance(Obje ctFactoryHelper.java:109) at org/apache/aries/jndi/OSGiObjectFactoryBuilder.getObjectInstance (OSGiObjectFactoryBuilder.java:57) at javax/naming/spi/NamingManager.getObjectInstance(NamingManager.j ava:349) at com/ibm/ws/naming/util/Helpers.processSerializedObjectForLookupE xt(Helpers.java:993) at com/ibm/ws/naming/util/Helpers.processSerializedObjectForLookup( Helpers.java:796) at com/ibm/ws/naming/jndicos/CNContextImpl.processBoundObjectForLoo kup(CNContextImpl.java:2877) at com/ibm/ws/naming/jndicos/CNContextImpl.processResolveResults(CN ContextImpl.java:3974) at com/ibm/ws/naming/jndicos/CNContextImpl.doLookup(CNContextImpl.j ava:1876) at com/ibm/ws/naming/jndicos/CNContextImpl.doLookup(CNContextImpl.j ava:1777(Compiled Code)) at com/ibm/ws/naming/jndicos/CNContextImpl.lookupExt(CNContextImpl. java:1434(Compiled Code)) at com/ibm/ws/naming/jndicos/CNContextImpl.lookup(CNContextImpl.jav a:616) at com/ibm/ws/naming/util/WsnInitCtx.lookup(WsnInitCtx.java:165) at com/ibm/ws/naming/util/WsnInitCtx.lookup(WsnInitCtx.java:179) at org/apache/aries/jndi/DelegateContext.lookup(DelegateContext.jav a:161) at javax/naming/InitialContext.lookup(InitialContext.java:436) at com/ibm/j2ca/jdbc/inbound/JDBCInboundDBHandlerWithXid.createDBCo nnection(JDBCInboundDBHandlerWithXid.java:170) at com/ibm/j2ca/jdbc/inbound/JDBCInboundDBHandlerWithXid.access$000 (JDBCInboundDBHandlerWithXid.java:49) at com/ibm/j2ca/jdbc/inbound/JDBCInboundDBHandlerWithXid$1.run(JDBC InboundDBHandlerWithXid.java:127) at java/security/AccessController.doPrivileged(AccessController.jav a:327(Compiled Code)) at com/ibm/j2ca/jdbc/inbound/JDBCInboundDBHandlerWithXid.privileged CreateDBConnection(JDBCInboundDBHandlerWithXid.java:123) at com/ibm/j2ca/jdbc/inbound/JDBCInboundDBHandlerWithXid.<init>(JDB CInboundDBHandlerWithXid.java:92) at com/ibm/j2ca/jdbc/inbound/JDBCEventStoreWithXid.<init>(JDBCEvent StoreWithXid.java:111) at com/ibm/j2ca/jdbc/JDBCResourceAdapter.createEventStore(JDBCResou rceAdapter.java:197) at com/ibm/j2ca/base/WBIResourceAdapter.createEventManager(WBIResou rceAdapter.java:458) at com/ibm/j2ca/base/WBIResourceAdapter$1.createEventManager(WBIRes ourceAdapter.java:203) at com/ibm/j2ca/base/internal/EndpointManager.startPolling(Endpoint Manager.java:291) (entered lock: java/lang/Object@0x000000005176CE80, entry count: 2) at com/ibm/j2ca/base/internal/EndpointManager.addEndpoint(EndpointM anager.java:169) (entered lock: java/lang/Object@0x000000005176CE80, entry count: 1) at com/ibm/j2ca/base/WBIResourceAdapter.endpointActivation(WBIResou rceAdapter.java:328) (entered lock: com/ibm/j2ca/base/internal/EndpointManager@0x000000005176CE90, entry count: 1) at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.activateUnderRAClassLo aderContext(ActivationSpecWrapperImpl.java:646) at com/ibm/ejs/j2c/ActivationSpecWrapperImpl.reActivateEndpoint(Act ivationSpecWrapperImpl.java:571) at com/ibm/ejs/j2c/RAWrapperImpl.resumeRAInbound(RAWrapperImpl.java :1578) (entered lock: com/ibm/ejs/j2c/RAWrapperImpl@0x000000005176CF80, entry count: 2) at com/ibm/ejs/j2c/RAWrapperImpl.memberIsActivated(RAWrapperImpl.ja va:2756) (entered lock: com/ibm/ejs/j2c/RAWrapperImpl@0x000000005176CF80, entry count: 1) at com/ibm/ws/hamanager/impl/HAGroupImpl.doActivateMember(HAGroupIm pl.java:713) at com/ibm/ws/hamanager/impl/HAGroupImpl$HAGroupUserCallback.doCall back(HAGroupImpl.java:1376) at com/ibm/ws/hamanager/impl/Worker.run(Worker.java:64) at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1864)
Problem conclusion
The J2C component has been updated to resolve the syncronization issue to prevent the deadlock. The fix for this APAR is currently targeted for inclusion in fix pack 8.5.5.9. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PI49357
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2015-09-25
Closed date
2016-01-07
Last modified date
2016-01-07
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R850 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
28 April 2022