STUCK threads encountered with Content Engine running on Weblogic using Jrockit and DB2.

Symptom

Performance will decrease due to the stuck threads and the system will eventually hang.

To confirm the issue, use the Weblogic admin server console to "Dump Thread Stacks". If there are stuck threads matching either of the signatures below then the issue is confirmed.

Signature 1
"[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" id=95 idx=0x160 tid=828 prio=1 alive, in native, blocked, daemon
-- Blocked trying to get lock: com/ibm/db2/jcc/t4/c@0x02953980[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Locks.java:1208)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1341)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1258)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2455)[optimized]
at com/ibm/db2/jcc/t4/ac.rollback(ac.java:1013)
at com/ibm/db2/jcc/t4/ac.b(ac.java:1039)
at com/ibm/db2/jcc/t4/ac.rollback(ac.java:1016)
^-- Holding lock: com/ibm/db2/jcc/t4/c@0x00920D10[thin lock]
at weblogic/jdbc/jta/DataSource.rollback(DataSource.java:1019)
at weblogic/transaction/internal/XAServerResourceInfo.rollback(XAServerResourceInfo.java:1401)
....
"[STUCK] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" id=119 idx=0x1c0 tid=4536 prio=1 alive, in native, blocked, daemon
-- Blocked trying to get lock: com/ibm/db2/jcc/t4/c@0x00920D10[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Locks.java:1208)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1341)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1258)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2455)[optimized]
at com/ibm/db2/jcc/t4/ac.rollback(ac.java:1013)
at com/ibm/db2/jcc/t4/ac.b(ac.java:1039)
at com/ibm/db2/jcc/t4/ac.rollback(ac.java:1016)
^-- Holding lock: com/ibm/db2/jcc/t4/c@0x02953980[thin lock]
at weblogic/jdbc/jta/DataSource.rollback(DataSource.java:1019)
at weblogic/transaction/internal/XAServerResourceInfo.rollback(XAServerResourceInfo.java:1401)

Signature 2
"[STUCK] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1776 idx=0x280 tid=4832 prio=1 alive, in native, waiting, daemon
-- Waiting for notification on: netscape/ldap/LDAPSearchListener@0x04982BA8[fat lock]
at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)
at jrockit/vm/Locks.wait(Locks.java:2250)[inlined]
at java/lang/Object.wait(Object.java:474)[optimized]
at netscape/ldap/LDAPMessageQueue.waitForMessage()V(Unknown Source)
^-- Lock released while waiting: netscape/ldap/LDAPSearchListener@0x04982BA8[recursive]
at netscape/ldap/LDAPMessageQueue.waitFirstMessage()V(Unknown Source)
^-- Lock released while waiting: netscape/ldap/LDAPSearchListener@0x04982BA8[fat lock]
at netscape/ldap/LDAPConnection.sendRequest(Lnetscape/ldap/client/opers/JDAPProtocolOp;Lnetscape/ldap/LDAPMessageQueue;Lnetscape/ldap/LDAPConstraints;)V(Unknown Source)
^-- Holding lock: netscape/ldap/LDAPConnection@0x04982C60[thin lock]
at netscape/ldap/LDAPConnection.search(Ljava/lang/String;ILjava/lang/String;[Ljava/lang/String;ZLnetscape/ldap/LDAPSearchConstraints;)Lnetscape/ldap/LDAPSearchResults;(Unknown Source)
at weblogic/security/providers/authentication/LDAPAtnDelegate.listMemberGroups(LDAPAtnDelegate.java:2122)