Direct link to fix
APAR status
Closed as program error.
Error description
Concurrent requests for updating the same user attribute for the same user can cause a deadlock situation. In this situation, no one can update or read the user attributes of the affected users. Typically, users will retry this operation. If they do, more and more threads hang until the system becomes unresponsive. You can see the following stack trace in the log: ThreadMonitor W WSVR0605W: Thread "WebContainer : 234" (00003c38) has been active for 666057 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung. at com.lombardisoftware.servlet.util.ProfileCache.getCacheData(Prof ileCache.java:126) at com.lombardisoftware.servlet.util.ProfileCache.getProfile(Profil eCache.java:98) at com.lombardisoftware.server.ejb.security.SecurityCore.getUserAtt ributes(SecurityCore.java:1200) at sun.reflect.GeneratedMethodAccessor188.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod AccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:508) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingRef lection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.inv okeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.pro ceed(ReflectiveMethodInvocation.java:157) at com.lombardisoftware.utility.spring.TransactionInterceptor$2.cal l(TransactionInterceptor.java:70) at com.lombardisoftware.utility.spring.ProgrammaticTransactionSuppo rt.executeInExistingTransaction(ProgrammaticTransactionSupport.j ava:798) at com.lombardisoftware.utility.spring.ProgrammaticTransactionSuppo rt.execute(ProgrammaticTransactionSupport.java:420) at com.lombardisoftware.utility.spring.TransactionInterceptor.invok e(TransactionInterceptor.java:68) at org.springframework.aop.framework.ReflectiveMethodInvocation.pro ceed(ReflectiveMethodInvocation.java:179) at com.lombardisoftware.utility.spring.CoreEntryInterceptor.invoke( CoreEntryInterceptor.java:44) at com.lombardisoftware.utility.spring.PSCoreEntryInterceptor.invok e(PSCoreEntryInterceptor.java:14) at org.springframework.aop.framework.ReflectiveMethodInvocation.pro ceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkD ynamicAopProxy.java:207) at com.sun.proxy.$Proxy72.getUserAttributes(Unknown Source) at com.lombardisoftware.server.ejb.security.SecurityBean$39.execute (SecurityBean.java:559) at com.lombardisoftware.server.ejb.security.SecurityBean$39.execute (SecurityBean.java:557) at com.lombardisoftware.server.ejb.EjbServerUtils.handleEjbCall(Ejb ServerUtils.java:94) at com.lombardisoftware.server.ejb.security.SecurityBean.handleEjbC all(SecurityBean.java:50) at com.lombardisoftware.server.ejb.security.SecurityBean.getUserAtt ributes(SecurityBean.java:557) at com.lombardisoftware.server.ejb.security.EJSRemoteStatelessSecur ity_287acf5b.getUserAttributes(Unknown Source) at com.lombardisoftware.server.ejb.security._SecurityInterface_Stub .getUserAttributes(_SecurityInterface_Stub.java:1831) at com.lombardisoftware.client.delegate.SecurityDelegateDefault$33. run(SecurityDelegateDefault.java:835) at java.security.AccessController.doPrivileged(AccessController.jav a:696) at com.lombardisoftware.client.delegate.common.WebsphereDelegateHel per.doAsCurrentSubjectContextSensitive(WebsphereDelegateHelper.j ava:225) at com.lombardisoftware.client.delegate.SecurityDelegateDefault.get UserAttributes(SecurityDelegateDefault.java:832) at com.ibm.bpm.rest.jaxb.mapping.OrgDataMappingUtils.mapUser(OrgDat aMappingUtils.java:206) at com.ibm.bpm.rest.jaxb.mapping.OrgDataMappingUtils.mapUser(OrgDat aMappingUtils.java:149) The following symptoms in the java processes and the database are visible, if the hang situation occurs: 1) Java cores a) thread with lock in method setUserAttributes() (e.g. WebContainer : 62) 3XMTHREADINFO "WebContainer : 62" ... 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at java/net/SocketInputStream.socketRead0(Native Method) 4XESTACKTRACE at java/net/SocketInputStream.socketRead(SocketInputStream.java:127 (Compiled Code)) 4XESTACKTRACE at java/net/SocketInputStream.read(SocketInputStream.java:182(Compi led Code)) 4XESTACKTRACE at java/net/SocketInputStream.read(SocketInputStream.java:152(Compi led Code)) 4XESTACKTRACE at com/ibm/db2/jcc/t4/y.b(y.java:224(Compiled Code))... 4XESTACKTRACE at com/lombardisoftware/userorg/UserOrgModule.setUserAttribute(User OrgModule.java:858) 4XESTACKTRACE at com/lombardisoftware/server/ejb/security/SecurityCore.setUserAtt ributes(SecurityCore.java:1277(Compiled Code)) 5XESTACKTRACE (entered lock: java/lang/String@0x00000001003CF918, entry count: 1) b) many other threads in getUserAttributes() blocked by the first thread (e.g. WebContainer : 62) 3XMTHREADINFO "WebContainer : 2" J9VMThread:0x000000000877F200, omrthread_t:0x000000001A1FB6F8, java/lang/Thread:0x00000001204767A8, state:B, prio=5 ... 3XMTHREADBLOCK Blocked on: java/lang/String@0x00000001003CF918 Owned by: "WebContainer : 62" (J9VMThread:0x000000000B19DF00,java/lang/Thread:0x00000001EA9D85 18) 3XMHEAPALLOC Heap bytes allocated since last GCcycle=0 (0x0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at com/lombardisoftware/servlet/util/ProfileCache.getCacheData(Prof ileCache.java:126(Compiled Code)) 4XESTACKTRACE at com/lombardisoftware/servlet/util/ProfileCache.getProfile(Profil eCache.java:98(Compiled Code)) 4XESTACKTRACE at com/lombardisoftware/server/ejb/security/SecurityCore.getUserAtt ributes(SecurityCore.java:1200(Compiled Code)) 4XESTACKTRACE at sun/reflect/GeneratedMethodAccessor373.invoke(BytecodePC:42(Comp iled Code)) 2) Database snapshots a) Application not commiting statement (e.g. 46539) 0x0000021824CF6360 46539 [000-46539] 1 4492 UOW-Waiting 0 0 371 1 10.148.50.36.52109.180702072351 1 140849 N C N Application : Address : 0x0000021824CF6360 AppHandl [nod-index] : 46539 [000-46539] ... Application Status : UOW-Waiting Application Name : db2jcc_application ... UOW start time : 2018-07-02-11.55.26.140406 UOW stop time : Last executed statements : Package cache ID : 0x0000017300000001 Anchor ID : 371 Statement UID : 1 SQL Type : Dynamic Statement Type : DML, Select (blockable) Statement : select GROUP_ID from LSW_USR_GRP_MEM_XREF where group_id in (?) and user_id=? union select x.CONTAINER_GROUP_ID as GROUP_ID from LSW_GRP_GRP_MEM_EXPLODED_XREF x where x.container_group_id in (?) and exists (select 1 from LSW_USR_GRP_MEM_XREF m where m.GROUP_ID=x.GROUP_ID and m.USER_ID=?) b) Other application handles waiting on the lock by first application (e.g. 46539) AppHandl [nod-index] TranHdl Lockname Type Mode Conv Sts CoorEDU AppName AuthID AppID 46539 [000-46539] 584 05004002D80000000000000052 RowLock ..X G 4492 db2jcc_a C011579 10.148.50.36.52109.180702072351 46540 [000-46540] 657 05004002D80000000000000052 RowLock ..X W 6940 db2jcc_a C011579 10.148.50.36.52112.180702072353 50786 [000-50786] 884 05004002D80000000000000052 RowLock ..X W 15260 db2jcc_a C011579 10.148.50.35.64534.180702092656 48523 [000-48523] 505 05004002D80000000000000052 RowLock ..X W 4060 db2jcc_a C011579 10.148.50.37.52851.180702081947
Local fix
Problem summary
This issue occurs because there are two distinct locks in one code path. When a user attribute is concurrently updated or read, the thread that already holds a Java-level lock gets an additional database lock. While this code path briefly releases the Java-level lock, a different thread might obtain this lock before the first thread completes its database interaction. If the second thread also interacts with the database to update this exact user attribute of this exact user, the two threads block each other infinitely.
Problem conclusion
A fix that avoids releasing the Java-level lock before the database interaction completes is available for IBM BPM V8.6 CF2018.03 and will be included in upcoming releases of Business Automation Workflow.
Temporary fix
Comments
APAR Information
APAR number
JR59724
Reported component name
BPM
Reported component ID
5737A5700
Reported release
860
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2018-07-04
Closed date
2018-11-21
Last modified date
2018-11-21
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
BPM
Fixed component ID
5737A5700
Applicable component levels
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSFPJS","label":"IBM Business Process Manager"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.6.0.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
14 September 2022