IBM Support

PI70104: Starting a Web Application Bundle (WAB) can result in a deadlock sometimes when the WAB is installed and started dynamically

Fixes are available

16.0.0.4: WebSphere Application Server Liberty 16.0.0.4
17.0.0.1: WebSphere Application Server Liberty 17.0.0.1
17.0.0.2: WebSphere Application Server Liberty 17.0.0.2
17.0.0.3: WebSphere Application Server Liberty 17.0.0.3
17.0.0.4: WebSphere Application Server Liberty 17.0.0.4
18.0.0.1: WebSphere Application Server Liberty 18.0.0.1
18.0.0.2: WebSphere Application Server Liberty 18.0.0.2
18.0.0.3: WebSphere Application Server Liberty 18.0.0.3
18.0.0.4: WebSphere Application Server Liberty 18.0.0.4
19.0.0.1: WebSphere Application Server Liberty 19.0.0.1
19.0.0.2: WebSphere Application Server Liberty 19.0.0.2
19.0.0.3: WebSphere Application Server Liberty 19.0.0.3
19.0.0.4: WebSphere Application Server Liberty 19.0.0.4
19.0.0.5: WebSphere Application Server Liberty 19.0.0.5
19.0.0.6: WebSphere Application Server Liberty 19.0.0.6
19.0.0.7: WebSphere Application Server Liberty 19.0.0.7
19.0.0.8: WebSphere Application Server Liberty 19.0.0.8
19.0.0.9: WebSphere Application Server Liberty 19.0.0.9
19.0.0.10: WebSphere Application Server Liberty 19.0.0.10
19.0.0.11: WebSphere Application Server Liberty 19.0.0.11
19.0.0.12: WebSphere Application Server Liberty 19.0.0.12
20.0.0.1: WebSphere Application Server Liberty 20.0.0.1
20.0.0.2: WebSphere Application Server Liberty 20.0.0.2
20.0.0.3: WebSphere Application Server Liberty 20.0.0.3
20.0.0.4: WebSphere Application Server Liberty 20.0.0.4
20.0.0.5: WebSphere Application Server Liberty 20.0.0.5
20.0.0.6: WebSphere Application Server Liberty 20.0.0.6
20.0.0.7: WebSphere Application Server Liberty 20.0.0.7
20.0.0.8: WebSphere Application Server Liberty 20.0.0.8
20.0.0.9: WebSphere Application Server Liberty 20.0.0.9
20.0.0.10: WebSphere Application Server Liberty 20.0.0.10
20.0.0.11: WebSphere Application Server Liberty 20.0.0.11
20.0.0.12: WebSphere Application Server Liberty 20.0.0.12

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A deadlock can occur in WebSphere Application Server Liberty
    starting a WAB.  This issue should only happen in cases
    where a WAB bundle is installed dynamically by code included
    in a Liberty feature.  This can result in a deadlock when
    resolving the gateway bundles used by the applications
    within Liberty.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server Liberty                              *
    ****************************************************************
    * PROBLEM DESCRIPTION: Starting a Web Application Bundle (WAB) *
    *                      can result in a deadlock sometimes when *
    *                      the WAB is installed and started        *
    *                      dynamically                             *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    A potential deadlock can be hit in Liberty during a WAB startup
    where a ThreadContextClassloader is created
    (ClassLoadingServiceImpl.createTCCL) and the same thread then
    tries to retrieve the ThreadContextClassloader
    (ClassLoadingServiceImpl.createThreadContextClassLoader calling
    into CanonicalStore.retrieveOrCreate).
    
    Looking at a javacore you will see a stack similar to the
    following:
    
    3XMTHREADINFO      "RUN_SERVICE_Thread-859"
    J9VMThread:0x00000053C7841800, j9thread_t:0x0000005433961BC0,
    java/lang/Thread:0x000000517366F438, state:P, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x521,
    isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x1C282E00, native
    priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread
    flags:0x00020001)
    3XMCPUTIME               CPU usage total: 26.106000000 secs,
    current category="Application"
    3XMTHREADBLOCK     Parked on:
    java/util/concurrent/CountDownLatch$Sync@0x000000518B346078
    Owned by: <unknown>
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=0 (0x0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at sun/misc/Unsafe.park(Native
    Method)
    4XESTACKTRACE                at
    java/util/concurrent/locks/LockSupport.park(LockSupport.java:186
    )
    4XESTACKTRACE                at
    java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndChe
    ckInterrupt(AbstractQueuedSynchronizer.java:847)
    4XESTACKTRACE                at
    java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireS
    haredInterruptibly(AbstractQueuedSynchronizer.java:1008)
    4XESTACKTRACE                at
    java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireSha
    redInterruptibly(AbstractQueuedSynchronizer.java:1315)
    4XESTACKTRACE                at
    java/util/concurrent/CountDownLatch.await(CountDownLatch.java:24
    2)
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/util/CanonicalStore$FutureRef.g
    et(CanonicalStore.java:123)
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:95(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:75(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:453(Compile
    d Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:80(Compiled
    Code))
    4XESTACKTRACE                at
    com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain
    er.java:818(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM
    odule(WebModuleHandlerImpl.java:166)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC
    ontainer(WABInstaller.java:408)
    5XESTACKTRACE                   (entered lock:
    com/ibm/ws/app/manager/wab/internal/WABGroup@0x000000518E9A1388,
    entry count: 1)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja
    va:373)
    5XESTACKTRACE                   (entered lock:
    com/ibm/ws/app/manager/wab/internal/WAB$AddRemoveLock@0x00000051
    8B13DC68, entry count: 2)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48
    2(Compiled Code))
    5XESTACKTRACE                   (entered lock:
    java/util/concurrent/atomic/AtomicReference@0x000000518B13DC78,
    entry count: 3)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57
    (Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:469(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:1(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke
    d.java:256(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/AbstractTracked.track(AbstractTracked.java
    :229(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/BundleTracker$Tracked.bundleChanged(Bundle
    Tracker.java:444(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/internal/framework/BundleContextImpl.dispatchEv
    ent(BundleContextImpl.java:903(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/framework/eventmgr/EventManager.dispatchEvent(E
    ventManager.java:230(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/framework/eventmgr/ListenerQueue.dispatchEventS
    ynchronous(ListenerQueue.java:148(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEventPrivileged(EquinoxEventPublisher.java:213(Compiled
    Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEvent(EquinoxEventPublisher.java:120(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEvent(EquinoxEventPublisher.java:112(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/internal/framework/EquinoxContainerAdaptor.publ
    ishModuleEvent(EquinoxContainerAdaptor.java:156(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/Module.publishEvent(Module.java:461(C
    ompiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/Module.doStart(Module.java:563(Compil
    ed Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/Module.start(Module.java:434(Compiled
    Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/framework/util/SecureAction.start(SecureAction.
    java:454(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/ModuleContainer.applyDelta(ModuleCont
    ainer.java:717(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/ModuleContainer.resolveAndApply(Modul
    eContainer.java:491(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain
    er.java:437(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain
    er.java:427(Compiled Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/container/Module.start(Module.java:411(Compiled
    Code))
    4XESTACKTRACE                at
    org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB
    undle.java:383(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/GatewayBundleFactory.start(Gate
    wayBundleFactory.java:136)
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/GatewayBundleFactory.createGate
    wayBundleClassLoader(GatewayBundleFactory.java:93)
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    CCL(ClassLoadingServiceImpl.java:494)
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.access$
    200(ClassLoadingServiceImpl.java:84(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat
    eInstance(ClassLoadingServiceImpl.java:456(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat
    eInstance(ClassLoadingServiceImpl.java:453(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:85(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:75(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:453(Compile
    d Code))
    4XESTACKTRACE                at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:80(Compiled
    Code))
    4XESTACKTRACE                at
    com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain
    er.java:818(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM
    odule(WebModuleHandlerImpl.java:166)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC
    ontainer(WABInstaller.java:408)
    5XESTACKTRACE                   (entered lock:
    com/ibm/ws/app/manager/wab/internal/WABGroup@0x0000005111734958,
    entry count: 1)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja
    va:373)
    5XESTACKTRACE                   (entered lock:
    com/ibm/ws/app/manager/wab/internal/WAB$AddRemoveLock@0x00000051
    8B13DC68, entry count: 1)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48
    2(Compiled Code))
    5XESTACKTRACE                   (entered lock:
    java/util/concurrent/atomic/AtomicReference@0x000000518B13DC78,
    entry count: 2)
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57
    (Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:469(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:1(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke
    d.java:256(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/AbstractTracked.trackInitial(AbstractTrack
    ed.java:183(Compiled Code))
    4XESTACKTRACE                at
    org/osgi/util/tracker/BundleTracker.open(BundleTracker.java:156(
    Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/app/manager/wab/internal/WAB$1.run(WAB.java:132)
    5XESTACKTRACE                   (entered lock:
    java/util/concurrent/atomic/AtomicReference@0x000000518B13DC78,
    entry count: 1)
    4XESTACKTRACE                at
    java/util/concurrent/Executors$RunnableAdapter.call(Executors.ja
    va:522(Compiled Code))
    4XESTACKTRACE                at
    java/util/concurrent/FutureTask.run(FutureTask.java:277(Compiled
    Code))
    4XESTACKTRACE                at
    java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExec
    utor.java:1153(Compiled Code))
    4XESTACKTRACE                at
    java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe
    cutor.java:628)
    4XESTACKTRACE                at
    com/ibm/cics/wlp/threading/CICSThread.run(CICSThread.java:243)
    4XESTACKTRACE                at
    com/ibm/cics/wlp/threading/CICSPooledThreadFactory.joinAsThreadI
    nternal(CICSPooledThreadFactory.java:408)
    4XESTACKTRACE                at
    com/ibm/cics/wlp/threading/CICSPooledThreadFactory.joinAsThread(
    CICSPooledThreadFactory.java:334)
    4XESTACKTRACE                at
    com/ibm/cics/server/internal/ThreadJoiner.main(ThreadJoiner.java
    :103)
    4XESTACKTRACE                at
    sun/reflect/GeneratedMethodAccessor14.invoke(Bytecode PC:24)
    4XESTACKTRACE                at
    sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
    AccessorImpl.java:55(Compiled Code))
    4XESTACKTRACE                at
    java/lang/reflect/Method.invoke(Method.java:508(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/cics/server/Wrapper.call_main(Wrapper.java:750)
    4XESTACKTRACE                at
    com/ibm/cics/server/Wrapper.callOSGiClass(Wrapper.java:2700)
    4XESTACKTRACE                at
    com/ibm/cics/server/Wrapper.invokeJvmServerOSGiClass(Wrapper.jav
    a:2572)
    4XESTACKTRACE                at
    com/ibm/cics/server/Wrapper.jvmServerOSGiEntry(Wrapper.java:2502
    )
    4XESTACKTRACE                at
    com/ibm/cics/osgi/impl/Controller.runService(Controller.java:176
    3)
    4XESTACKTRACE                at
    com/ibm/cics/osgi/impl/Controller.acceptRequest(Controller.java:
    338)
    4XESTACKTRACE                at
    sun/reflect/GeneratedMethodAccessor11.invoke(Bytecode PC:130)
    4XESTACKTRACE                at
    sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
    AccessorImpl.java:55(Compiled Code))
    4XESTACKTRACE                at
    java/lang/reflect/Method.invoke(Method.java:508(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/cics/router/Router.route(Router.java:1207)
    3XMTHREADINFO3           No native callstack available on this
    platform
    

Problem conclusion

  • The property osgi.module.auto.start.on.resolve=false is now set
    by default to stop the automatic starting of OSGi modules when
    they are resolved.
    
    The fix for this APAR is currently targeted for inclusion in fix
    pack 16.0.0.4.  Please refer to the Recommended Updates page for
    delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

  • In versions 16.0.0.2 and 16.0.0.3 you can set
    osgi.module.auto.start.on.resolve=false in your
    bootstrap.properties file.
    

Comments

APAR Information

  • APAR number

    PI70104

  • Reported component name

    LIBERTY PROFILE

  • Reported component ID

    5724J0814

  • Reported release

    855

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-10-04

  • Closed date

    2016-10-05

  • Last modified date

    2016-10-05

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    LIBERTY PROFILE

  • Fixed component ID

    5724J0814

Applicable component levels

  • R855 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"855","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
18 October 2021