Fixes are available
16.0.0.4: WebSphere Application Server Liberty 16.0.0.4
17.0.0.1: WebSphere Application Server Liberty 17.0.0.1
17.0.0.2: WebSphere Application Server Liberty 17.0.0.2
17.0.0.3: WebSphere Application Server Liberty 17.0.0.3
17.0.0.4: WebSphere Application Server Liberty 17.0.0.4
18.0.0.1: WebSphere Application Server Liberty 18.0.0.1
18.0.0.2: WebSphere Application Server Liberty 18.0.0.2
18.0.0.3: WebSphere Application Server Liberty 18.0.0.3
18.0.0.4: WebSphere Application Server Liberty 18.0.0.4
19.0.0.1: WebSphere Application Server Liberty 19.0.0.1
19.0.0.2: WebSphere Application Server Liberty 19.0.0.2
19.0.0.3: WebSphere Application Server Liberty 19.0.0.3
19.0.0.4: WebSphere Application Server Liberty 19.0.0.4
19.0.0.5: WebSphere Application Server Liberty 19.0.0.5
19.0.0.6: WebSphere Application Server Liberty 19.0.0.6
19.0.0.7: WebSphere Application Server Liberty 19.0.0.7
19.0.0.8: WebSphere Application Server Liberty 19.0.0.8
19.0.0.9: WebSphere Application Server Liberty 19.0.0.9
19.0.0.10: WebSphere Application Server Liberty 19.0.0.10
19.0.0.11: WebSphere Application Server Liberty 19.0.0.11
19.0.0.12: WebSphere Application Server Liberty 19.0.0.12
20.0.0.1: WebSphere Application Server Liberty 20.0.0.1
20.0.0.2: WebSphere Application Server Liberty 20.0.0.2
20.0.0.3: WebSphere Application Server Liberty 20.0.0.3
20.0.0.4: WebSphere Application Server Liberty 20.0.0.4
20.0.0.5: WebSphere Application Server Liberty 20.0.0.5
20.0.0.6: WebSphere Application Server Liberty 20.0.0.6
20.0.0.7: WebSphere Application Server Liberty 20.0.0.7
20.0.0.8: WebSphere Application Server Liberty 20.0.0.8
20.0.0.9: WebSphere Application Server Liberty 20.0.0.9
20.0.0.10: WebSphere Application Server Liberty 20.0.0.10
20.0.0.11: WebSphere Application Server Liberty 20.0.0.11
20.0.0.12: WebSphere Application Server Liberty 20.0.0.12
21.0.0.3: WebSphere Application Server Liberty 21.0.0.3
21.0.0.4: WebSphere Application Server Liberty 21.0.0.4
21.0.0.5: WebSphere Application Server Liberty 21.0.0.5
21.0.0.6: WebSphere Application Server Liberty 21.0.0.6
21.0.0.7: WebSphere Application Server Liberty 21.0.0.7
21.0.0.8: WebSphere Application Server Liberty 21.0.0.8
21.0.0.9: WebSphere Application Server Liberty 21.0.0.9
21.0.0.1: WebSphere Application Server Liberty 21.0.0.1
21.0.0.2: WebSphere Application Server Liberty 21.0.0.2
21.0.0.10: WebSphere Application Server Liberty 21.0.0.10
21.0.0.11: WebSphere Application Server Liberty 21.0.0.11
21.0.0.12: WebSphere Application Server Liberty 21.0.0.12
22.0.0.1: WebSphere Application Server Liberty 22.0.0.1
22.0.0.2: WebSphere Application Server Liberty 22.0.0.2
22.0.0.3: WebSphere Application Server Liberty 22.0.0.3
22.0.0.4: WebSphere Application Server Liberty 22.0.0.4
APAR status
Closed as program error.
Error description
A deadlock can occur in WebSphere Application Server Liberty starting a WAB. This issue should only happen in cases where a WAB bundle is installed dynamically by code included in a Liberty feature. This can result in a deadlock when resolving the gateway bundles used by the applications within Liberty.
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server Liberty * **************************************************************** * PROBLEM DESCRIPTION: Starting a Web Application Bundle (WAB) * * can result in a deadlock sometimes when * * the WAB is installed and started * * dynamically * **************************************************************** * RECOMMENDATION: * **************************************************************** A potential deadlock can be hit in Liberty during a WAB startup where a ThreadContextClassloader is created (ClassLoadingServiceImpl.createTCCL) and the same thread then tries to retrieve the ThreadContextClassloader (ClassLoadingServiceImpl.createThreadContextClassLoader calling into CanonicalStore.retrieveOrCreate). Looking at a javacore you will see a stack similar to the following: 3XMTHREADINFO "RUN_SERVICE_Thread-859" J9VMThread:0x00000053C7841800, j9thread_t:0x0000005433961BC0, java/lang/Thread:0x000000517366F438, state:P, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x521, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x1C282E00, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001) 3XMCPUTIME CPU usage total: 26.106000000 secs, current category="Application" 3XMTHREADBLOCK Parked on: java/util/concurrent/CountDownLatch$Sync@0x000000518B346078 Owned by: <unknown> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at sun/misc/Unsafe.park(Native Method) 4XESTACKTRACE at java/util/concurrent/locks/LockSupport.park(LockSupport.java:186 ) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndChe ckInterrupt(AbstractQueuedSynchronizer.java:847) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireS haredInterruptibly(AbstractQueuedSynchronizer.java:1008) 4XESTACKTRACE at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireSha redInterruptibly(AbstractQueuedSynchronizer.java:1315) 4XESTACKTRACE at java/util/concurrent/CountDownLatch.await(CountDownLatch.java:24 2) 4XESTACKTRACE at com/ibm/ws/classloading/internal/util/CanonicalStore$FutureRef.g et(CanonicalStore.java:123) 4XESTACKTRACE at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:95(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:75(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:453(Compile d Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:80(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain er.java:818(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM odule(WebModuleHandlerImpl.java:166) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC ontainer(WABInstaller.java:408) 5XESTACKTRACE (entered lock: com/ibm/ws/app/manager/wab/internal/WABGroup@0x000000518E9A1388, entry count: 1) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja va:373) 5XESTACKTRACE (entered lock: com/ibm/ws/app/manager/wab/internal/WAB$AddRemoveLock@0x00000051 8B13DC68, entry count: 2) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48 2(Compiled Code)) 5XESTACKTRACE (entered lock: java/util/concurrent/atomic/AtomicReference@0x000000518B13DC78, entry count: 3) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57 (Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:469(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:1(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke d.java:256(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/AbstractTracked.track(AbstractTracked.java :229(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.bundleChanged(Bundle Tracker.java:444(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/framework/BundleContextImpl.dispatchEv ent(BundleContextImpl.java:903(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/EventManager.dispatchEvent(E ventManager.java:230(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/eventmgr/ListenerQueue.dispatchEventS ynchronous(ListenerQueue.java:148(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEventPrivileged(EquinoxEventPublisher.java:213(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEvent(EquinoxEventPublisher.java:120(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEvent(EquinoxEventPublisher.java:112(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxContainerAdaptor.publ ishModuleEvent(EquinoxContainerAdaptor.java:156(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/container/Module.publishEvent(Module.java:461(C ompiled Code)) 4XESTACKTRACE at org/eclipse/osgi/container/Module.doStart(Module.java:563(Compil ed Code)) 4XESTACKTRACE at org/eclipse/osgi/container/Module.start(Module.java:434(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/util/SecureAction.start(SecureAction. java:454(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/container/ModuleContainer.applyDelta(ModuleCont ainer.java:717(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/container/ModuleContainer.resolveAndApply(Modul eContainer.java:491(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain er.java:437(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain er.java:427(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/container/Module.start(Module.java:411(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB undle.java:383(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/GatewayBundleFactory.start(Gate wayBundleFactory.java:136) 4XESTACKTRACE at com/ibm/ws/classloading/internal/GatewayBundleFactory.createGate wayBundleClassLoader(GatewayBundleFactory.java:93) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT CCL(ClassLoadingServiceImpl.java:494) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.access$ 200(ClassLoadingServiceImpl.java:84(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat eInstance(ClassLoadingServiceImpl.java:456(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat eInstance(ClassLoadingServiceImpl.java:453(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:85(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:75(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:453(Compile d Code)) 4XESTACKTRACE at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:80(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain er.java:818(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM odule(WebModuleHandlerImpl.java:166) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC ontainer(WABInstaller.java:408) 5XESTACKTRACE (entered lock: com/ibm/ws/app/manager/wab/internal/WABGroup@0x0000005111734958, entry count: 1) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja va:373) 5XESTACKTRACE (entered lock: com/ibm/ws/app/manager/wab/internal/WAB$AddRemoveLock@0x00000051 8B13DC68, entry count: 1) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48 2(Compiled Code)) 5XESTACKTRACE (entered lock: java/util/concurrent/atomic/AtomicReference@0x000000518B13DC78, entry count: 2) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57 (Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:469(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:1(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke d.java:256(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/AbstractTracked.trackInitial(AbstractTrack ed.java:183(Compiled Code)) 4XESTACKTRACE at org/osgi/util/tracker/BundleTracker.open(BundleTracker.java:156( Compiled Code)) 4XESTACKTRACE at com/ibm/ws/app/manager/wab/internal/WAB$1.run(WAB.java:132) 5XESTACKTRACE (entered lock: java/util/concurrent/atomic/AtomicReference@0x000000518B13DC78, entry count: 1) 4XESTACKTRACE at java/util/concurrent/Executors$RunnableAdapter.call(Executors.ja va:522(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/FutureTask.run(FutureTask.java:277(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExec utor.java:1153(Compiled Code)) 4XESTACKTRACE at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe cutor.java:628) 4XESTACKTRACE at com/ibm/cics/wlp/threading/CICSThread.run(CICSThread.java:243) 4XESTACKTRACE at com/ibm/cics/wlp/threading/CICSPooledThreadFactory.joinAsThreadI nternal(CICSPooledThreadFactory.java:408) 4XESTACKTRACE at com/ibm/cics/wlp/threading/CICSPooledThreadFactory.joinAsThread( CICSPooledThreadFactory.java:334) 4XESTACKTRACE at com/ibm/cics/server/internal/ThreadJoiner.main(ThreadJoiner.java :103) 4XESTACKTRACE at sun/reflect/GeneratedMethodAccessor14.invoke(Bytecode PC:24) 4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:55(Compiled Code)) 4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:508(Compiled Code)) 4XESTACKTRACE at com/ibm/cics/server/Wrapper.call_main(Wrapper.java:750) 4XESTACKTRACE at com/ibm/cics/server/Wrapper.callOSGiClass(Wrapper.java:2700) 4XESTACKTRACE at com/ibm/cics/server/Wrapper.invokeJvmServerOSGiClass(Wrapper.jav a:2572) 4XESTACKTRACE at com/ibm/cics/server/Wrapper.jvmServerOSGiEntry(Wrapper.java:2502 ) 4XESTACKTRACE at com/ibm/cics/osgi/impl/Controller.runService(Controller.java:176 3) 4XESTACKTRACE at com/ibm/cics/osgi/impl/Controller.acceptRequest(Controller.java: 338) 4XESTACKTRACE at sun/reflect/GeneratedMethodAccessor11.invoke(Bytecode PC:130) 4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:55(Compiled Code)) 4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:508(Compiled Code)) 4XESTACKTRACE at com/ibm/cics/router/Router.route(Router.java:1207) 3XMTHREADINFO3 No native callstack available on this platform
Problem conclusion
The property osgi.module.auto.start.on.resolve=false is now set by default to stop the automatic starting of OSGi modules when they are resolved. The fix for this APAR is currently targeted for inclusion in fix pack 16.0.0.4. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
In versions 16.0.0.2 and 16.0.0.3 you can set osgi.module.auto.start.on.resolve=false in your bootstrap.properties file.
Comments
APAR Information
APAR number
PI70104
Reported component name
LIBERTY PROFILE
Reported component ID
5724J0814
Reported release
855
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-10-04
Closed date
2016-10-05
Last modified date
2016-10-05
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
LIBERTY PROFILE
Fixed component ID
5724J0814
Applicable component levels
R855 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"855","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
03 May 2022