IBM Support

PI53419: LIBERTY SERVER Z/OS: DEADLOCK ADDING WABS TO WEBCONTAINER

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Customer is running liberty server embedded in a CICS
    environment on z/OS and reported some CICS tasks that did
    not
    complete.
    £ADESTADS.TASK157.ADHT] State:(BLOCKED)  LockName:
    £java.util.
    concurrent.atomic.AtomicReference@3b588b9d] LockOwner:
    £Default
    Executor-thread-24]
    Analysis of a console dump showed a deadlock:
    
    com/ibm/ws/classloading/internal/util/CanonicalStore@0x
    48ee8c9a80
    owner thread id: 0x240ee000 name: Default
    Executor-thread-1109
    waiting thread id: 0x24043800 name: Default
    Executor-thread-1079
    ..
    java/util/concurrent/atomic/AtomicReference@0x4a6e548aa0
    
    owner thread id: 0x24043800 name: Default
    Executor-thread-1079
    waiting thread id: 0x240ee000 name: Default
    Executor-thread-1109
    --
    Review of the logic on the threads showed that the out
    of order lock obtains occur when WAB1 is being added to
    the webcontainer on one thread and WAB2 is being
    started on the other thread. The Deadlock can occur if
    WAB2 needs to add the bundle for WAB1 during OSGi
    Bundle Resolution.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server Liberty                              *
    ****************************************************************
    * PROBLEM DESCRIPTION: Deadlock adding WABs to the             *
    *                      webcontainer                            *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    A deadlock can occur when starting a WebSphere Liberty
    application server with multiple WABs.  The deadlock can be
    reported in a thread dump as such:
    com/ibm/ws/classloading/internal/util/CanonicalStore@0x48ee8c7a8
    0
    owner thread id: 0x240ee000 name: Default Executor-thread-1139
    waiting thread id: 0x24043800 name: Default Executor-thread-
    1379
    ..
    java/util/concurrent/atomic/AtomicReference@0x4a6e545aa0
    
    owner thread id: 0x24043800 name: Default Executor-thread-1379
    waiting thread id: 0x240ee000 name: Default Executor-thread-
    1139
    
    One callstack will look like this:
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:46
    4(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57
    (Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:469(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:1(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke
    d.java:256(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.track(AbstractTracked.java
    :229(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.bundleChanged(Bundle
    Tracker.java:444(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/BundleContextImpl.dispatchEv
    ent(BundleContextImpl.java:902(Compiled Code))
                    at
    org/eclipse/osgi/framework/eventmgr/EventManager.dispatchEvent(E
    ventManager.java:230(Compiled Code))
                    at
    org/eclipse/osgi/framework/eventmgr/ListenerQueue.dispatchEventS
    ynchronous(ListenerQueue.java:148(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEventPrivileged(EquinoxEventPublisher.java:165(Compiled
    Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEvent(EquinoxEventPublisher.java:75(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEvent(EquinoxEventPublisher.java:67(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxContainerAdaptor.publ
    ishModuleEvent(EquinoxContainerAdaptor.java:102(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.applyDelta(ModuleCont
    ainer.java:691(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.resolveAndApply(Modul
    eContainer.java:491(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain
    er.java:437(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain
    er.java:427(Compiled Code))
                    at
    org/eclipse/osgi/container/Module.start(Module.java:416(Compiled
    Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB
    undle.java:393(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB
    undle.java:412(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/GatewayBundleFactory.start(Gate
    wayBundleFactory.java:135(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/GatewayBundleFactory.createGate
    wayBundleClassLoader(GatewayBundleFactory.java:92(Compiled
    Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    CCL(ClassLoadingServiceImpl.java:464(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.access$
    200(ClassLoadingServiceImpl.java:82(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat
    eInstance(ClassLoadingServiceImpl.java:436(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat
    eInstance(ClassLoadingServiceImpl.java:433(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:85(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:75(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:433(Compile
    d Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:78(Compiled
    Code))
                    at
    com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain
    er.java:798(Compiled Code))
                    at
    com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM
    odule(WebModuleHandlerImpl.java:165(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC
    ontainer(WABInstaller.java:377(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja
    va:373(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48
    0(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57
    (Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:469(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:1(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke
    d.java:256(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.trackInitial(AbstractTrack
    ed.java:183(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker.open(BundleTracker.java:156(
    Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB$1.run(WAB.java:132(Compi
    led Code))
                    at
    java/util/concurrent/Executors$RunnableAdapter.call(Executors.ja
    va:483(Compiled Code))
                    at
    java/util/concurrent/FutureTask.run(FutureTask.java:274(Compiled
    Code))
                    at
    java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExec
    utor.java:1157)
                    at
    java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe
    cutor.java:627)
                    at java/lang/Thread.run(Thread.java:798)
    
    while the other (and possibly many others) will look like this:
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:46
    4(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57
    (Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:469(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:1(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke
    d.java:256(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.track(AbstractTracked.java
    :229(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.bundleChanged(Bundle
    Tracker.java:444(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/BundleContextImpl.dispatchEv
    ent(BundleContextImpl.java:902(Compiled Code))
                    at
    org/eclipse/osgi/framework/eventmgr/EventManager.dispatchEvent(E
    ventManager.java:230(Compiled Code))
                    at
    org/eclipse/osgi/framework/eventmgr/ListenerQueue.dispatchEventS
    ynchronous(ListenerQueue.java:148(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEventPrivileged(EquinoxEventPublisher.java:165(Compiled
    Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEvent(EquinoxEventPublisher.java:75(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis
    hBundleEvent(EquinoxEventPublisher.java:67(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxContainerAdaptor.publ
    ishModuleEvent(EquinoxContainerAdaptor.java:102(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.applyDelta(ModuleCont
    ainer.java:691(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.resolveAndApply(Modul
    eContainer.java:491(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain
    er.java:437(Compiled Code))
                    at
    org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain
    er.java:427(Compiled Code))
                    at
    org/eclipse/osgi/container/Module.start(Module.java:416(Compiled
    Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB
    undle.java:393(Compiled Code))
                    at
    org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB
    undle.java:412(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/GatewayBundleFactory.start(Gate
    wayBundleFactory.java:135(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/GatewayBundleFactory.createGate
    wayBundleClassLoader(GatewayBundleFactory.java:92(Compiled
    Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    CCL(ClassLoadingServiceImpl.java:464(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.access$
    200(ClassLoadingServiceImpl.java:82(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat
    eInstance(ClassLoadingServiceImpl.java:436(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat
    eInstance(ClassLoadingServiceImpl.java:433(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:85(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC
    reate(CanonicalStore.java:75(Compiled Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:433(Compile
    d Code))
                    at
    com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT
    hreadContextClassLoader(ClassLoadingServiceImpl.java:78(Compiled
    Code))
                    at
    com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain
    er.java:798(Compiled Code))
                    at
    com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM
    odule(WebModuleHandlerImpl.java:165(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC
    ontainer(WABInstaller.java:377(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja
    va:373(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48
    0(Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57
    (Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:469(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun
    dleTracker.java:1(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke
    d.java:256(Compiled Code))
                    at
    org/osgi/util/tracker/AbstractTracked.trackInitial(AbstractTrack
    ed.java:183(Compiled Code))
                    at
    org/osgi/util/tracker/BundleTracker.open(BundleTracker.java:156(
    Compiled Code))
                    at
    com/ibm/ws/app/manager/wab/internal/WAB$1.run(WAB.java:132(Compi
    led Code))
                    at
    java/util/concurrent/Executors$RunnableAdapter.call(Executors.ja
    va:483(Compiled Code))
                    at
    java/util/concurrent/FutureTask.run(FutureTask.java:274(Compiled
    Code))
                    at
    java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExec
    utor.java:1157)
                    at
    java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe
    cutor.java:627)
                    at java/lang/Thread.run(Thread.java:798)
    

Problem conclusion

  • The fix for this APAR is to improve the locking mechanism used
    in the WAB code to avoid this deadlock.
    
    The fix for this APAR is currently targeted for inclusion in fix
    pack 16.0.0.2.  Please refer to the Recommended Updates page for
    delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

  • 
    

Comments

  • 
    

APAR Information

  • APAR number

    PI53419

  • Reported component name

    LIBERTY PROF -

  • Reported component ID

    5655W6514

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2015-12-02

  • Closed date

    2016-05-02

  • Last modified date

    2016-06-13

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    LIBERTY PROF -

  • Fixed component ID

    5655W6514

Applicable component levels

[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Platform":[{"code":"PF054","label":"z\/OS"}],"Version":"850"}]

Document Information

Modified date:
28 March 2021