Topic
1 reply Latest Post - ‏2013-01-05T12:19:05Z by mmalc
SystemAdmin
SystemAdmin
289 Posts
ACCEPTED ANSWER

Pinned topic Messages failing, Error in AppTarget Logs

‏2012-12-27T22:55:43Z |
Hi All,

Please help.

I am pretty new to WebSphere ESB as well as IBM technologies. Yesterday suddenly some of the messaged related to certain applications started failing and then after an hour everything started working smoothly. I am not sure what happened. When I am checking AppTarget logs I am able to see below error:

0000002b LocalTransact E J2CA0024E: Method commit, within transaction branch ID <null> of resource pool SIB$SibRaConnectionFactory$DefaultPrincipalMapping$SCA_Auth_Alias, caught javax.resource.ResourceException: CWSIV0406E: An internal error occurred. The exception com.ibm.wsspi.sib.core.exception.SIRollbackException: CWSIC8007E: An exception was caught from the remote server with Probe Id 3-009-0007. Exception: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65. was thrown while the container was attempting to commit a local transaction.
at com.ibm.ws.sib.ra.impl.SibRaManagedConnection$SibRaLocalTransaction.commit(SibRaManagedConnection.java:1195)
at com.ibm.ejs.j2c.LocalTransactionWrapper.commit(LocalTransactionWrapper.java:263)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:942)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:830)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.end(LocalTranCoordImpl.java:1424)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.endLocalTx(UOWNativeWASStrategyImpl.java:1676)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionImportExport(UOWNativeWASStrategyImpl.java:447)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:168)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler$2.run(TargetSecurityContextHandler.java:165)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5365)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSpecified(ContextManagerImpl.java:5433)
at com.ibm.ws.sca.internal.managed.security.SecurityContextManager.runAs(SecurityContextManager.java:568)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler.processMessage(TargetSecurityContextHandler.java:161)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionLocalActivitySessionFalse(UOWNativeWASStrategyImpl.java:1319)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.deliverAsyncAtCall(UOWNativeWASStrategyImpl.java:86)
at com.ibm.ws.sca.uow.handler.DeliverAsyncAtUOWNativeHandler.processMessage(DeliverAsyncAtUOWNativeHandler.java:72)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.SCAInvocationAction.invokeAsyncOneWay(SCAInvocationAction.java:497)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeByStyle(RetryControl.java:373)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeWithRetry(RetryControl.java:245)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.ServiceInvocation.complete(ServiceInvocation.java:125)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.FlowActionFactoryImpl.create(FlowActionFactoryImpl.java:732)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.invokeFlowAction(ServiceInvokeMediation.java:663)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.mediate(ServiceInvokeMediation.java:576)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:739)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.SIBXJavaMediationPrimitive.invoke(SIBXJavaMediationPrimitive.java:108)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.Input.invoke(Input.java:228)
at com.ibm.ws.sibx.scax.mediation.engine.RequestFlow.invokeFlow(RequestFlow.java:135)
at com.ibm.ws.sibx.scax.mediation.engine.MediationFlow.invokeRequestFlow(MediationFlow.java:168)
at com.ibm.wsspi.sibx.mediation.flow.ejb.MediationFlowBean.invokeRequestFlow(MediationFlowBean.java:292)
at com.ibm.ws.sibx.scax.mediation.component.SIBXMediationFlowComponentImpl.invokeRequestFlow(SIBXMediationFlowComponentImpl.java:135)
at com.ibm.ws.sibx.scax.runtime.handler.MFCImplementationHandler.processMessage(MFCImplementationHandler.java:228)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:774)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:999)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:854)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:214)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:785)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessageWithSCAContext(HTTPExportHandler.java:1204)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessage(HTTPExportHandler.java:1180)
at com.ibm.ws.http.export.handler.HTTPExportHandler.processRequest(HTTPExportHandler.java:391)
at com.ibm.ws.http.export.handler.HTTPExportHandler.doPost(HTTPExportHandler.java:233)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1049)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:643)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)
Caused by: com.ibm.wsspi.sib.core.exception.SIRollbackException: CWSIC8007E: An exception was caught from the remote server with Probe Id 3-009-0007. Exception: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
at com.ibm.ws.sib.comms.common.CommsByteBuffer.parseSingleException(CommsByteBuffer.java:1850)
at com.ibm.ws.sib.comms.common.CommsByteBuffer.getException(CommsByteBuffer.java:1263)
at com.ibm.ws.sib.comms.common.JFAPCommunicator.checkFor_SIRollbackException(JFAPCommunicator.java:1440)
at com.ibm.ws.sib.comms.client.LocalTransactionProxy.commit(LocalTransactionProxy.java:156)
at com.ibm.ws.sib.comms.client.OptimizedUncoordinatedTransactionProxy.commit(OptimizedUncoordinatedTransactionProxy.java:113)
at com.ibm.ws.sib.ra.impl.SibRaManagedConnection$SibRaLocalTransaction.commit(SibRaManagedConnection.java:1183)
... 86 more
Caused by: com.ibm.websphere.sib.exception.SIErrorException: CWSIC8002E: An internal error occurred. An unknown or unexpected exception was thrown by the core API: exception com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
at com.ibm.ws.sib.comms.common.CommsByteBuffer.parseSingleException(CommsByteBuffer.java:1884)
... 91 more
Caused by: com.ibm.websphere.sib.exception.SIErrorException: CWSIC8002E: An internal error occurred. An unknown or unexpected exception was thrown by the core API: exception com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
... 92 more
and threw a XAException.

Can someone please help and let me know what might be the issue.

Thanks,
Hetal
Updated on 2013-01-05T12:19:05Z at 2013-01-05T12:19:05Z by mmalc
  • mmalc
    mmalc
    74 Posts
    ACCEPTED ANSWER

    Re: Messages failing, Error in AppTarget Logs

    ‏2013-01-05T12:19:05Z  in response to SystemAdmin
    Based on the DB2 error codes, it looks like you may be running out of space in your transaction logs

    http://pic.dhe.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=%2Fcom.ibm.db2.luw.messages.sql.doc%2Fdoc%2Fmsql00964c.html

    http://pic.dhe.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=%2Fcom.ibm.db2.luw.messages.doc%2Fdoc%2Frdb2stt.html

    Before going into detailed db2 investigation, I'd first check that you haven't just run out of disk space on the filesystem.