IBM Support

PI71240: SIB MESSAGE DEADLOCK AFTER SERVANT IS RESTARED DUE TO ABENDEC3

Fixes are available

9.0.0.3: WebSphere Application Server traditional V9.0 Fix Pack 3
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
9.0.0.4: WebSphere Application Server traditional V9.0 Fix Pack 4
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
9.0.0.5: WebSphere Application Server traditional V9.0 Fix Pack 5
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
9.0.0.6: WebSphere Application Server traditional V9.0 Fix Pack 6
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
9.0.0.7: WebSphere Application Server traditional V9.0 Fix Pack 7
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
9.0.0.8: WebSphere Application Server traditional V9.0 Fix Pack 8
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
9.0.0.9: WebSphere Application Server traditional V9.0 Fix Pack 9
9.0.0.10: WebSphere Application Server traditional V9.0 Fix Pack 10
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
9.0.0.11: WebSphere Application Server traditional V9.0 Fix Pack 11
9.0.5.0: WebSphere Application Server traditional Version 9.0.5 Refresh Pack
9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
9.0.5.4: WebSphere Application Server traditional Version 9.0.5 Fix Pack 4
9.0.5.5: WebSphere Application Server traditional Version 9.0.5 Fix Pack 5
9.0.5.6: WebSphere Application Server traditional Version 9.0.5 Fix Pack 6

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A hang issue happened causing the servant region to abend with
    an EC3 and cause a new servant region to restart (normal
    behavior after an EC3 timeout). After the problem that was
    causing the hang was solved, the servant region didn't
    processing any work.  This servant runs an application  which
    is an MDB that consumes messages from a SIB queue. This SIB
    queue is hosted in a message engine that was running in this
    same application server (in the
    ADJUNCT  region).
    
    Javacores show a deadlock:
    
    Deadlock detected !!!
    ---------------------
    Thread "JFAP Proxy Event Queue Listener" (0x00000000264FB400)
      is waiting for:
    
    java/util/concurrent/locks/ReentrantLock$NonfairSync@0x000000008
    2FE8358
      which is owned by:
    Thread "Consumer w0sr01.000-PDC_BX : 221746"
    (0x00000000266A1100)
      which is waiting for:
        sys_mon_t:0x00000048402FE098 infl_mon_t: 0x00000048402FE110:
    
    com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration@0x00000
    00082FE8
    3F8
      which is owned by:
    Thread "JFAP Proxy Event Queue Listener" (0x00000000264FB400)
    
    These two threads have this traceback:
    
    "JFAP Proxy Event Queue Listener" J9VMThread:0x00000000264FB400,
    j9thread_t:0x0000004809C2C170,
    java/lang/Thread:0x0000000082DA70E8,
    state:P, prio=5
    (java/lang/Thread getId:0x8B, isDaemon:true)
    (native thread ID:0x21DD5800, native priority:0x5, native
    policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001)
    CPU usage total: 4538.684000000 secs
    Parked on:
    java/util/concurrent/locks/ReentrantLock$NonfairSync@0x000000008
    2FE8358
    Owned by: "Consumer w0sr01.000-PDC_BX : 221746"
    (J9VMThread:0x00000000266A1100,
    java/lang/Thread:0x00000000841D2618)
    Heap bytes allocated since last GC cycle=0 (0x0)
    Java callstack:
      at sun/misc/Unsafe.park(Native Method)
      at java/util/concurrent/locks/LockSupport.
    park(LockSupport.java:182(Compiled Code))
      at java/util/concurrent/locks/AbstractQueuedSynchronizer.
    parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822(Compil
    ed Code))
      atjava/util/concurrent/locks/AbstractQueuedSynchronizer.
    acquireQueued(AbstractQueuedSynchronizer.java:853(Compiled
    Code))
      at java/util/concurrent/locks/AbstractQueuedSynchronizer.
    acquire(AbstractQueuedSynchronizer.java:1189(Compiled Code))
      at java/util/concurrent/locks/ReentrantLock$NonfairSync.
    lock(ReentrantLock.java:197(Compiled Code))
      at java/util/concurrent/locks/ReentrantLock.
    lock(ReentrantLock.java:273(Compiled Code))
      at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
    put(JSLocalConsumerPoint.java:753(Compiled Code))
      at com/ibm/ws/sib/processor/impl/ConsumerDispatcher.
    giveMessageToConsumer(ConsumerDispatcher.java:855(Compiled
    Code))
    <truncated stack trace >
    
    "Consumer w0sr01.000-PDC_BX : 221746"
    J9VMThread:0x00000000266A1100,
    j9thread_t:0x0000004809D90550,
    java/lang/Thread:0x00000000841D2618,
    state:B, prio=5
    java/lang/Thread getId:0x71F6C, isDaemon:true)
    (native thread ID:0x21D4F800, native priority:0x5, native
    policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201)
    CPU usage total: 3.470000000 secs
    Blocked on:
    com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration@0x00000
    00082FE8
    3F8 Owned by: "JFAP Proxy Event Queue Listener"
    (J9VMThread:0x00000000264FB400,
    java/lang/Thread:0x0000000082DA70E8)
    Heap bytes allocated since last GC cycle=0 (0x0)
    Java callstack:
      at
    com/ibm/ws/sib/processor/impl/AbstractLockedMessageEnumeration.
    addNewMessage(AbstractLockedMessageEnumeration.java:470(Compiled
    Code))
      at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
    lockMessages(JSLocalConsumerPoint.java:2966(Compiled Code))
      at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
    processQueuedMsgs(JSL ocalConsumerPoint.java:2652(Compiled
    Code))
      at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint.
    runAsynchConsumer(JSLocalConsumerPoint.java:3318(Compiled Code))
        (entered lock: com/ibm/ws/sib/processor/impl/
    AsynchConsumer@0x0000000082FE83D8, entry count: 1)
      at
    com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint$AsynchThread
    .run(JSLocalConsumerPoint.java:3637(Compiled Code))
      at com/ibm/ws/util/ThreadPool$Worker.
    run(ThreadPool.java:1881(CompiledCode))
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V8.0, V8.5, and V9.0                 *
    ****************************************************************
    * PROBLEM DESCRIPTION: Deadlock in                             *
    *                      com/ibm/ws/sib/processor/impl/JSLockedM *
    *                      es                                      *
    *                      sageEnumeration                         *
    ****************************************************************
    * RECOMMENDATION:  Very rare, if seen apply the fix and        *
    *                  restart                                     *
    *                  the appserver.                              *
    ****************************************************************
    Deadlock in
    com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration
    A dump needs to be taken and the thread stacks analyzed to
    determne of the deadlock is the same.
    

Problem conclusion

Temporary fix

Comments

APAR Information

  • APAR number

    PI71240

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-10-25

  • Closed date

    2017-01-10

  • Last modified date

    2017-01-10

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R850 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
19 October 2021