Topic
  • 2 replies
  • Latest Post - ‏2009-12-04T01:16:17Z by k.below
jrees
jrees
26 Posts

Pinned topic Deadlocks on BPEDB during high load with long running processes

‏2009-11-30T19:05:45Z |
WebSphere Application Server 6.1.0.19
WebSphere Process Server 6.1.2.2
AIX 6
DB2 9.5 for AIX hosting databases on a separate box
We have deployed a cluster using the golden topology. We have three physical servers each running a single node.
We have quite a few long running processes, and when our server becomes loaded down, we begin to see deadlocks on our BPEDB
We feel we have sufficient capacity to handle the transactions, and need to figure out tuning or configuration changes we can make to eliminate the deadlocks.

Currently, our work around is to run only one server in our cluster. However, this is not satisfactory as it is not permitting us to meet our intended capacity

Looking at the log, we see the deadlock, and then we see the rollback affected by further deadlocks. This issue then seems to affect other currently running processes as well, leading to a failure of all the active transactions. The failed events begin to build, and our only recourse seems to be to stop any new transactions from arriving and enlisting the BPC explorer to terminate these transactions.

Can anyone offer any advice on how to tune to eliminate these deadlocks?
Can usage of the BPC explorer during peak times contribute to the problems?

Example of the sequence (more detailed log attached)

11/19/09 20:38:46:279 GMT 00000132 ExceptionUtil E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "bpc_onMessage" on bean "BeanId(XXXXCommonModuleApp#XXXXCommonModuleEJB.jar#component.MilestoneManagerProcess, null)". Exception data: com.ibm.bpe.database.TomSQLException: com.ibm.db2.jcc.b.SqlException: DB2 SQL error: SQLCODE: -911, SQLSTATE: 40001, SQLERRMC: 2
11/19/09 20:38:46:320 GMT 00000132 ExceptionUtil E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "transactionRequiredActivitySessionSupports" on bean "BeanId(XXXXCommonModuleApp#XXXXCommonModuleEJB.jar#Module, null)". Exception data: com.ibm.db2.jcc.b.SqlException: ibmdb2jcc1034211669 Application must execute a rollback. The unit of work has already been rolled back in the database but other resource managers involved in this unit of work might not. To ensure integrity of this application, all SQL requests will be rejected until the application issues a rollback.

SCA context:
Interaction: onInvokeResponse,executeMilestonePost {XXXXCommonModule}MilestoneManagerProcess.component#MilestonePostInterface => {XXXXCommonModule}MilestonePostProcess.component
Handler: com/ibm/bpe/framework/sca/ProcessImplementationHandler (com.ibm.bpe.framework.sca.ProcessImplementationHandler,com/ibm/wsspi/sca/extensions/ImplementationGroup2)

Wrapped by:
com.ibm.ws.sca.uow.UOWRolledBackException: com.ibm.bpe.database.TomSQLException: caused by: com.ibm.bpe.database.TomSQLException
11/19/09 20:38:46:458 GMT 00000132 Core E CWSCA2001E: The SCA message-driven bean returned the following exception: "com.ibm.websphere.sca.ServiceRuntimeException: com.ibm.bpe.database.TomSQLException: caused by: com.ibm.bpe.database.TomSQLException".
11/19/09 20:38:46:466 GMT 00000132 ExceptionUtil E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "onMessage" on bean "BeanId(XXXXCommonModuleApp#XXXXCommonModuleEJB.jar#ServiceSIBusMessageBean, null)". Exception data: com.ibm.db2.jcc.b.SqlException: ibmdb2jcc1034211669 Application must execute a rollback. The unit of work has already been rolled back in the database but other resource managers involved in this unit of work might not. To ensure integrity of this application, all SQL requests will be rejected until the application issues a rollback.

SCA context:
Interaction: onInvokeResponse,executeMilestonePost {XXXXCommonModule}MilestoneManagerProcess.component#MilestonePostInterface => {XXXXCommonModule}MilestonePostProcess.component
Handler: com/ibm/bpe/framework/sca/ProcessImplementationHandler (com.ibm.bpe.framework.sca.ProcessImplementationHandler,com/ibm/wsspi/sca/extensions/ImplementationGroup2)
11/19/09 20:38:46:486 GMT 00000132 WSRdbXaResour E DSRA0304E: XAException occurred. XAException contents and details are:
The DB2 Error message is : ibmdb2jcct4204111392 Error executing XAResource.end(). Server returned XA_RBDEADLOCK.
The DB2 Error code is : -4203
The DB2 SQLState is : null
.
11/19/09 20:38:46:494 GMT 00000132 WSRdbXaResour E DSRA0302E: XAException occurred. Error code is: XAER_NOTA (-4). Exception is: XAER_NOTA : ibmdb2jcct4204111392 Error executing XAResource.end(). Server returned XA_RBDEADLOCK. : ibmdb2jcct4204111392 Error executing XAResource.rollback(). Server returned XAER_NOTA.
11/19/09 20:38:46:497 GMT 00000132 XATransaction E J2CA0027E: An exception occurred while invoking rollback on an XA Resource Adapter from dataSource jdbc/BPEDB, within transaction ID {XidImpl: formatId(57415344), gtrid_length(36), bqual_length(54), data(000001250e2b0c300000000100002ed81f06b72012e5bbedde4d7b342b26c25b288abc77000001250e2b0c300000000100002ed81f06b72012e5bbedde4d7b342b26c25b288abc77000000010000000000000000000000000002)}: com.ibm.db2.jcc.b.cj: XAER_NOTA : ibmdb2jcct4204111392 Error executing XAResource.end(). Server returned XA_RBDEADLOCK. : ibmdb2jcct4204111392 Error executing XAResource.rollback(). Server returned XAER_NOTA.
  • ramkoduri
    ramkoduri
    1 Post

    Re: Deadlocks on BPEDB during high load with long running processes

    ‏2009-12-03T22:08:38Z  
    We are also faceing the same issue from 3 months can any one please help out
  • k.below
    k.below
    14 Posts

    Re: Deadlocks on BPEDB during high load with long running processes

    ‏2009-12-04T01:16:17Z  
    I guess that the initial question would be ... what tuning has been performed so far?

    I have found the process choreographer tuning information provided in the info center to be an excellent place to start.

    Here's a link to the 6.1.2 information
    http://publib.boulder.ibm.com/infocenter/dmndhelp/v6r1mx/index.jsp?topic=/com.ibm.websphere.bpc.612.doc/doc/bpc/t5tuning.html