PM37247: DEADLOCK CAUSES SERVER HANG LOCKHIERARCHY.OBTAINLOCK TRANSACTIONCOORDINATORIMPL.ROLLBACK

Fixes are available

APAR status

Closed as program error.

Error description

During high volume of transaction work the Control Region may go
into a deadlocked state and become unresponsive. No new work can
be executed and the server will not respond to MODIFY or STOP
commands. A Cancel will need to be done to restart the server.

This problem is more likely to happen in setups where there is
a high volume of transactions and there are multiple servers
that are joined transitionally. The servers in the middle  could
potentially get numerous requests from upstream and downstream
servers and enter a deadlock state by exhausting all worker
threads in Control Region.

An SVCDUMP will show many Control Region worker threads
(ACR_ExecutionThread::RemoveAndProcessWork) with the following
Java stack traces at the top:

--
TCB 8ba1e0 caa 4a6d2fecf0 pthread_id 3614580000000049 WebSphere
t=008ba1e0 J9VMThread 3e31a300 (daemon)

Dsa          Entry      Offset     Function
---          -----      ------     --------
4a6d0f6360   2a926db0   00000000
bbo3pse(int*,int*,char*,char*,char*)
4a6d0f66a0   2a7d8c98   00000536   ORB_Request::pauseThread()
4a6d0f6960   2a741910   000006ac
ORB_Request::comm_outbound_ctl_local_request()
4a6d0f6be0   2a740480   00000d1e
ORB_Request::comm_outbound_request()
4a6d0f6e20   2a793160   00000c0e
Java_com_ibm_ws390_orb_ClientDelegate_invokeRequestCFW
4a6d0f7b20   2c3e8140   d3c17ec0   RUNCALLINMETHOD
4a6d0f7f60   2c420268   00000040   callVirtualObjectMethodA
4a6d0f8060   2a5815f8   00001230   ORBEJSBridge::invoke(void*)
4a6d0f9060   2a705988   00000ea8
threadDispatch(BOSS_Object_Key*,Internal_CORBA_Request&,ORB_Requ
est*)
4a6d0f9ca0   2a707ba8   00000a56
ACR_ExecutionThread::ProcessInboundRequest(acrwObj*,ThreadCleanU
p*,BOSS_Object_Key&,Interna...

Java stack:

-----------

Method
------
com/ibm/ws390/orb/ClientDelegate.invokeRequestCFW
com/ibm/ws390/orb/ClientDelegate.commonInvoke
com/ibm/ws390/orb/ClientDelegate.invoke
org/omg/CORBA/portable/ObjectImpl._invoke
com/ibm/ws390/tx/syncpoint/_SyncpointManagerStub.beforeCompletio
n
com/ibm/ws390/tx/TransactionSynchronizer$1.run
com/ibm/ws/security/auth/ContextManagerImpl.runAs
com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem
com/ibm/ws/security/auth/ContextManagerImpl.runAsSystemOrSpecifi
ed
com/ibm/ws/security/core/SecurityContext.runAsSystemOrSpecified
com/ibm/ws/tx/util/WASTMHelper$2.run
com/ibm/ws/security/util/AccessController.doPrivileged
com/ibm/ws/tx/util/WASTMHelper.runAsSystemOrSpecified
com/ibm/tx/util/TMHelper.runAsSystemOrSpecified
com/ibm/ws390/tx/TransactionSynchronizer.before_completion
com/ibm/ws390/tx/TransactionResourceImpl.prepare
com/ibm/ws390/tx/TransactionResource.prepare
org/omg/CosTransactions/_ResourceImplBase._invoke




TCB 8ba400 caa 4a756fecf0 pthread_id 36148b000000004a WebSphere
t=008ba400 J9VMThread 3e316200 (daemon)

Dsa          Entry      Offset     Function
---          -----      ------     --------
4a754f6360   2a926db0   00000000
bbo3pse(int*,int*,char*,char*,char*)
4a754f66a0   2a7d8c98   00000536   ORB_Request::pauseThread()
4a754f6960   2a741910   000006ac
ORB_Request::comm_outbound_ctl_local_request()
4a754f6be0   2a740480   00000d1e
ORB_Request::comm_outbound_request()
4a754f6e20   2a793160   00000c0e
Java_com_ibm_ws390_orb_ClientDelegate_invokeRequestCFW

Java stack:
-----------
com/ibm/ws390/orb/ClientDelegate.invokeRequestCFW
com/ibm/ws390/orb/ClientDelegate.commonInvoke
com/ibm/ws390/orb/ClientDelegate.invoke
org/omg/CORBA/portable/ObjectImpl._invoke
com/ibm/ws390/tx/alarm/_CorbaTimeoutStub.setRollbackOnly
com/ibm/ws390/tx/TransactionalUnitOfWork$1.run
com/ibm/ws/security/auth/ContextManagerImpl.runAs
com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem
com/ibm/ws/security/auth/ContextManagerImpl.runAsSystemOrSpecifi
ed
com/ibm/ws/security/core/SecurityContext.runAsSystemOrSpecified
com/ibm/ws/tx/util/WASTMHelper$2.run
com/ibm/ws/security/util/AccessController.doPrivileged
com/ibm/ws/tx/util/WASTMHelper.runAsSystemOrSpecified
com/ibm/tx/util/TMHelper.runAsSystemOrSpecified
com/ibm/ws390/tx/TransactionalUnitOfWork.rollbackOnly
com/ibm/ws390/tx/TransactionCoordinatorImpl.rollback_only
com/ibm/ws390/tx/TransactionCoordinatorImpl.rollback_only
com/ibm/ws390/tx/TransactionCoordinator.rollback_only
org/omg/CosTransactions/_CoordinatorImplBase._invoke



TCB 8b74f8 caa 4ad86fecf0 pthread_id 3615570000000056 WebSphere
t=008b74f8 J9VMThread 3e323600 (daemon)

Dsa          Entry      Offset     Function
---          -----      ------     --------
4ad84f7200   29eed358   00002014   CEEOPCW
4ad84f75e0   2a206dc0   0000014e   pthread_cond_wait
4ad84f7760   2c490bb0   00000606   monitor_wait_original
4ad84f78e0   2c494648   0000008c
j9thread_monitor_wait_interruptable
4ad84f79e0   2d0abd58   000004a6
jclCallMonitorWaitInterruptable
4ad84f7b20   2c3e8140   00c47cce   RUNCALLINMETHOD
4ad84f7f60   2c420268   00000040   callVirtualObjectMethodA
4ad84f8060   2a5815f8   00001230   ORBEJSBridge::invoke(void*)
4ad84f9060   2a705988   00000ea8
threadDispatch(BOSS_Object_Key*,Internal_CORBA_Request&,ORB_Requ
est*)
4ad84f9ca0   2a707ba8   00000a56
ACR_ExecutionThread::ProcessInboundRequest(acrwObj*,ThreadCleanU
p*,BOSS_Object_Key&,Interna...

Java stack:
-----------
java/lang/Object.wait
java/lang/Object.wait
com/ibm/ws/util/lock/WriterPriorityReadersWriterLock.startWritin
g
com/ibm/ws390/tx/LockHierarchy.obtainLock
com/ibm/ws390/tx/ReaderWriterLock.obtainWrite
com/ibm/ws390/tx/TransactionCoordinator.rollback_only
org/omg/CosTransactions/_CoordinatorImplBase._invoke
com/ibm/ws390/orb/CommonBridge.invoke
--

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server V7.0 and V8.0                        *
****************************************************************
* PROBLEM DESCRIPTION: The Application Server becomes          *
*                      unresponsive after experiencing         *
*                      workload failures that tie              *
*                      all controller regions threads.         *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
During high volume distributed transaction workload involving
multiple servers, the application server becomes unresponsive
because the high volume transactional workload encounters
multiple failures in the servant region while performing
phase one commit processing that trigger callbacks to the
controller region for transaction failure handling.

In this particular case, the callbacks to the controller
region do not make any progress because all of the threads
in the controller region are busy trying to execute
other phase-one commit work requests coming from upstream
servers and rollback-only requests coming from downstream
servers. All of these requests in the controller region are
also held from making progress because they are waiting
for the internal requests that are already executing in the
servant region (waiting for the callbacks to finish) to
complete.

At the time the server becomes unresponsive the following
code path can be seen on multiple controller region
threads' stack trace up to the number of configured threads.

Thread x:
...
com/ibm/tx/util/TMHelper.runAsSystemOrSpecified
com/ibm/ws390/tx/TransactionSynchronizer.before_completion
com/ibm/ws390/tx/TransactionResourceImpl.prepare
com/ibm/ws390/tx/TransactionResource.prepare
org/omg/CosTransactions/_ResourceImplBase._invoke

Thread y:
...
com/ibm/tx/util/TMHelper.runAsSystemOrSpecified
com/ibm/ws390/tx/TransactionalUnitOfWork.rollbackOnly
com/ibm/ws390/tx/TransactionCoordinatorImpl.rollback_only
com/ibm/ws390/tx/TransactionCoordinatorImpl.rollback_only
com/ibm/ws390/tx/TransactionCoordinator.rollback_only
org/omg/CosTransactions/_CoordinatorImplBase._invoke

Problem conclusion

Code was added to prevent the callbacks under the
circumstances described to mitigate thread deadlock.

APAR PM37247 is currently targeted for inclusion in
Service Level (Fix Pack) 7.0.0.19 and 8.0.0.1 of WebSphere
Application Server

Please refer to URL:
//www.ibm.com/support/docview.wss?rs=404&uid=swg27006970
for Fix Pack availability.

Temporary fix

Comments

APAR Information

APAR number
PM37247
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2011-04-18
Closed date
2011-05-31
Last modified date
2011-10-04

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500

Applicable component levels

R700 PSY UK71297
UP11/09/10 P F109

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
27 October 2021

Tips

PM37247: DEADLOCK CAUSES SERVER HANG LOCKHIERARCHY.OBTAINLOCK TRANSACTIONCOORDINATORIMPL.ROLLBACK_ONLY TRANSACTIONSYNCHRONIZER

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R700 PSY UK71297

Fix is available

Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.

Document Information

Share your feedback

Need support?