Fixes are available
9.0.0.3: WebSphere Application Server traditional V9.0 Fix Pack 3
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
9.0.0.4: WebSphere Application Server traditional V9.0 Fix Pack 4
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
9.0.0.5: WebSphere Application Server traditional V9.0 Fix Pack 5
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
9.0.0.6: WebSphere Application Server traditional V9.0 Fix Pack 6
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
9.0.0.7: WebSphere Application Server traditional V9.0 Fix Pack 7
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
9.0.0.8: WebSphere Application Server traditional V9.0 Fix Pack 8
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
9.0.0.9: WebSphere Application Server traditional V9.0 Fix Pack 9
9.0.0.10: WebSphere Application Server traditional V9.0 Fix Pack 10
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
9.0.0.11: WebSphere Application Server traditional V9.0 Fix Pack 11
9.0.5.0: WebSphere Application Server traditional Version 9.0.5 Refresh Pack
9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
9.0.5.4: WebSphere Application Server traditional Version 9.0.5 Fix Pack 4
9.0.5.5: WebSphere Application Server traditional Version 9.0.5 Fix Pack 5
WebSphere Application Server traditional 9.0.5.6
9.0.5.7: WebSphere Application Server traditional Version 9.0.5 Fix Pack 7
9.0.5.8: WebSphere Application Server traditional Version 9.0.5.8
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
9.0.5.9: WebSphere Application Server traditional Version 9.0.5.9
9.0.5.10: WebSphere Application Server traditional Version 9.0.5.10
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
9.0.5.11: WebSphere Application Server traditional Version 9.0.5.11
APAR status
Closed as program error.
Error description
A hang issue happened causing the servant region to abend with an EC3 and cause a new servant region to restart (normal behavior after an EC3 timeout). After the problem that was causing the hang was solved, the servant region didn't processing any work. This servant runs an application which is an MDB that consumes messages from a SIB queue. This SIB queue is hosted in a message engine that was running in this same application server (in the ADJUNCT region). Javacores show a deadlock: Deadlock detected !!! --------------------- Thread "JFAP Proxy Event Queue Listener" (0x00000000264FB400) is waiting for: java/util/concurrent/locks/ReentrantLock$NonfairSync@0x000000008 2FE8358 which is owned by: Thread "Consumer w0sr01.000-PDC_BX : 221746" (0x00000000266A1100) which is waiting for: sys_mon_t:0x00000048402FE098 infl_mon_t: 0x00000048402FE110: com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration@0x00000 00082FE8 3F8 which is owned by: Thread "JFAP Proxy Event Queue Listener" (0x00000000264FB400) These two threads have this traceback: "JFAP Proxy Event Queue Listener" J9VMThread:0x00000000264FB400, j9thread_t:0x0000004809C2C170, java/lang/Thread:0x0000000082DA70E8, state:P, prio=5 (java/lang/Thread getId:0x8B, isDaemon:true) (native thread ID:0x21DD5800, native priority:0x5, native policy:UNKNOWN, vmstate:P, vm thread flags:0x00020001) CPU usage total: 4538.684000000 secs Parked on: java/util/concurrent/locks/ReentrantLock$NonfairSync@0x000000008 2FE8358 Owned by: "Consumer w0sr01.000-PDC_BX : 221746" (J9VMThread:0x00000000266A1100, java/lang/Thread:0x00000000841D2618) Heap bytes allocated since last GC cycle=0 (0x0) Java callstack: at sun/misc/Unsafe.park(Native Method) at java/util/concurrent/locks/LockSupport. park(LockSupport.java:182(Compiled Code)) at java/util/concurrent/locks/AbstractQueuedSynchronizer. parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822(Compil ed Code)) atjava/util/concurrent/locks/AbstractQueuedSynchronizer. acquireQueued(AbstractQueuedSynchronizer.java:853(Compiled Code)) at java/util/concurrent/locks/AbstractQueuedSynchronizer. acquire(AbstractQueuedSynchronizer.java:1189(Compiled Code)) at java/util/concurrent/locks/ReentrantLock$NonfairSync. lock(ReentrantLock.java:197(Compiled Code)) at java/util/concurrent/locks/ReentrantLock. lock(ReentrantLock.java:273(Compiled Code)) at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint. put(JSLocalConsumerPoint.java:753(Compiled Code)) at com/ibm/ws/sib/processor/impl/ConsumerDispatcher. giveMessageToConsumer(ConsumerDispatcher.java:855(Compiled Code)) <truncated stack trace > "Consumer w0sr01.000-PDC_BX : 221746" J9VMThread:0x00000000266A1100, j9thread_t:0x0000004809D90550, java/lang/Thread:0x00000000841D2618, state:B, prio=5 java/lang/Thread getId:0x71F6C, isDaemon:true) (native thread ID:0x21D4F800, native priority:0x5, native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201) CPU usage total: 3.470000000 secs Blocked on: com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration@0x00000 00082FE8 3F8 Owned by: "JFAP Proxy Event Queue Listener" (J9VMThread:0x00000000264FB400, java/lang/Thread:0x0000000082DA70E8) Heap bytes allocated since last GC cycle=0 (0x0) Java callstack: at com/ibm/ws/sib/processor/impl/AbstractLockedMessageEnumeration. addNewMessage(AbstractLockedMessageEnumeration.java:470(Compiled Code)) at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint. lockMessages(JSLocalConsumerPoint.java:2966(Compiled Code)) at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint. processQueuedMsgs(JSL ocalConsumerPoint.java:2652(Compiled Code)) at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint. runAsynchConsumer(JSLocalConsumerPoint.java:3318(Compiled Code)) (entered lock: com/ibm/ws/sib/processor/impl/ AsynchConsumer@0x0000000082FE83D8, entry count: 1) at com/ibm/ws/sib/processor/impl/JSLocalConsumerPoint$AsynchThread .run(JSLocalConsumerPoint.java:3637(Compiled Code)) at com/ibm/ws/util/ThreadPool$Worker. run(ThreadPool.java:1881(CompiledCode))
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server V8.0, V8.5, and V9.0 * **************************************************************** * PROBLEM DESCRIPTION: Deadlock in * * com/ibm/ws/sib/processor/impl/JSLockedM * * es * * sageEnumeration * **************************************************************** * RECOMMENDATION: Very rare, if seen apply the fix and * * restart * * the appserver. * **************************************************************** Deadlock in com/ibm/ws/sib/processor/impl/JSLockedMessageEnumeration A dump needs to be taken and the thread stacks analyzed to determne of the deadlock is the same.
Problem conclusion
The fix is to correctly apply the locking hierarchy in jsLocalConsumerPoint.CleanupBifurcatedConsumer(). APAR PI71240 is currently targeted for inclusion in Fix Packs 8.0.0.13, 8.5.5.12 and 9.0.0.3. of WebSphere Application Server. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980 In addition, please refer to URL: http://www.ibm.com/support/docview.wss?rs=404&uid=swg27006970 for Fix Pack PTF information.
Temporary fix
Comments
APAR Information
APAR number
PI71240
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-10-25
Closed date
2017-01-10
Last modified date
2017-01-10
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R850 PSY
UP
Document Information
Modified date:
04 May 2022