Topic
  • 5 replies
  • Latest Post - ‏2013-01-03T09:45:57Z by zhangyu
SystemAdmin
SystemAdmin
3556 Posts

Pinned topic Messages failed during certain period of time

‏2012-12-27T22:29:49Z |
Hi All,

Please help.

I am pretty new to WebSphere ESB as well as IBM technologies. Yesterday suddenly some of the messaged related to certain applications started failing and then after an hour everything started working smoothly. I am not sure what happened. When I am checking AppTarget logs I am able to see below error:

0000002b LocalTransact E J2CA0024E: Method commit, within transaction branch ID <null> of resource pool SIB$SibRaConnectionFactory$DefaultPrincipalMapping$SCA_Auth_Alias, caught javax.resource.ResourceException: CWSIV0406E: An internal error occurred. The exception com.ibm.wsspi.sib.core.exception.SIRollbackException: CWSIC8007E: An exception was caught from the remote server with Probe Id 3-009-0007. Exception: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65. was thrown while the container was attempting to commit a local transaction.
at com.ibm.ws.sib.ra.impl.SibRaManagedConnection$SibRaLocalTransaction.commit(SibRaManagedConnection.java:1195)
at com.ibm.ejs.j2c.LocalTransactionWrapper.commit(LocalTransactionWrapper.java:263)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:942)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:830)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.end(LocalTranCoordImpl.java:1424)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.endLocalTx(UOWNativeWASStrategyImpl.java:1676)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionImportExport(UOWNativeWASStrategyImpl.java:447)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:168)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler$2.run(TargetSecurityContextHandler.java:165)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5365)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSpecified(ContextManagerImpl.java:5433)
at com.ibm.ws.sca.internal.managed.security.SecurityContextManager.runAs(SecurityContextManager.java:568)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler.processMessage(TargetSecurityContextHandler.java:161)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionLocalActivitySessionFalse(UOWNativeWASStrategyImpl.java:1319)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.deliverAsyncAtCall(UOWNativeWASStrategyImpl.java:86)
at com.ibm.ws.sca.uow.handler.DeliverAsyncAtUOWNativeHandler.processMessage(DeliverAsyncAtUOWNativeHandler.java:72)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.SCAInvocationAction.invokeAsyncOneWay(SCAInvocationAction.java:497)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeByStyle(RetryControl.java:373)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeWithRetry(RetryControl.java:245)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.ServiceInvocation.complete(ServiceInvocation.java:125)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.FlowActionFactoryImpl.create(FlowActionFactoryImpl.java:732)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.invokeFlowAction(ServiceInvokeMediation.java:663)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.mediate(ServiceInvokeMediation.java:576)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:739)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.SIBXJavaMediationPrimitive.invoke(SIBXJavaMediationPrimitive.java:108)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.Input.invoke(Input.java:228)
at com.ibm.ws.sibx.scax.mediation.engine.RequestFlow.invokeFlow(RequestFlow.java:135)
at com.ibm.ws.sibx.scax.mediation.engine.MediationFlow.invokeRequestFlow(MediationFlow.java:168)
at com.ibm.wsspi.sibx.mediation.flow.ejb.MediationFlowBean.invokeRequestFlow(MediationFlowBean.java:292)
at com.ibm.ws.sibx.scax.mediation.component.SIBXMediationFlowComponentImpl.invokeRequestFlow(SIBXMediationFlowComponentImpl.java:135)
at com.ibm.ws.sibx.scax.runtime.handler.MFCImplementationHandler.processMessage(MFCImplementationHandler.java:228)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:774)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:999)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:854)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:214)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:785)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessageWithSCAContext(HTTPExportHandler.java:1204)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessage(HTTPExportHandler.java:1180)
at com.ibm.ws.http.export.handler.HTTPExportHandler.processRequest(HTTPExportHandler.java:391)
at com.ibm.ws.http.export.handler.HTTPExportHandler.doPost(HTTPExportHandler.java:233)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1049)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:643)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)
Caused by: com.ibm.wsspi.sib.core.exception.SIRollbackException: CWSIC8007E: An exception was caught from the remote server with Probe Id 3-009-0007. Exception: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
at com.ibm.ws.sib.comms.common.CommsByteBuffer.parseSingleException(CommsByteBuffer.java:1850)
at com.ibm.ws.sib.comms.common.CommsByteBuffer.getException(CommsByteBuffer.java:1263)
at com.ibm.ws.sib.comms.common.JFAPCommunicator.checkFor_SIRollbackException(JFAPCommunicator.java:1440)
at com.ibm.ws.sib.comms.client.LocalTransactionProxy.commit(LocalTransactionProxy.java:156)
at com.ibm.ws.sib.comms.client.OptimizedUncoordinatedTransactionProxy.commit(OptimizedUncoordinatedTransactionProxy.java:113)
at com.ibm.ws.sib.ra.impl.SibRaManagedConnection$SibRaLocalTransaction.commit(SibRaManagedConnection.java:1183)
... 86 more
Caused by: com.ibm.websphere.sib.exception.SIErrorException: CWSIC8002E: An internal error occurred. An unknown or unexpected exception was thrown by the core API: exception com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
at com.ibm.ws.sib.comms.common.CommsByteBuffer.parseSingleException(CommsByteBuffer.java:1884)
... 91 more
Caused by: com.ibm.websphere.sib.exception.SIErrorException: CWSIC8002E: An internal error occurred. An unknown or unexpected exception was thrown by the core API: exception com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
... 92 more
and threw a XAException.

Can someone please help and let me know what might be the issue.

Thanks,
Hetal
  • gas
    gas
    36 Posts

    Re: Messages failed during certain period of time

    ‏2012-12-28T09:18:11Z  
    Hi,

    This error:
    DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011
    suggest that transaction log is full. Check these links for some hints and possible solutions:

    http://www-01.ibm.com/support/docview.wss?uid=swg21377796
    http://publib.boulder.ibm.com/infocenter/tivihelp/v5r1/index.jsp?topic=%2Fcom.ibm.itim.infocenter.doc%2Fex460_pdg79.htm

    Gas
  • SystemAdmin
    SystemAdmin
    3556 Posts

    Re: Messages failed during certain period of time

    ‏2013-01-02T18:40:16Z  
    • gas
    • ‏2012-12-28T09:18:11Z
    Hi,

    This error:
    DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011
    suggest that transaction log is full. Check these links for some hints and possible solutions:

    http://www-01.ibm.com/support/docview.wss?uid=swg21377796
    http://publib.boulder.ibm.com/infocenter/tivihelp/v5r1/index.jsp?topic=%2Fcom.ibm.itim.infocenter.doc%2Fex460_pdg79.htm

    Gas
    Hello Gas,

    Thanks for the reply.... but looks like it will not help me :-(

    1st link : says I need to have this error : SQL0964C: The transaction log for the database is full.

    But if you will see my logs I am not getting that error.

    2nd link : is for Tivoli Identity Manager Express, where as I am working on WebSphere ESB so I think this will not help.

    Any other ideas?

    Last night also same issue happen,,, when I restarted the ESB server then it started working fine. But restarting server everytime is not the option for me. Please let me know what to do.

    Thanks,
    Hetal
  • gas
    gas
    36 Posts

    Re: Messages failed during certain period of time

    ‏2013-01-02T19:24:36Z  
    Hello Gas,

    Thanks for the reply.... but looks like it will not help me :-(

    1st link : says I need to have this error : SQL0964C: The transaction log for the database is full.

    But if you will see my logs I am not getting that error.

    2nd link : is for Tivoli Identity Manager Express, where as I am working on WebSphere ESB so I think this will not help.

    Any other ideas?

    Last night also same issue happen,,, when I restarted the ESB server then it started working fine. But restarting server everytime is not the option for me. Please let me know what to do.

    Thanks,
    Hetal
    Did you try it?

    It is exactly same error code as in your stack trace.
    Contact your DB2 admin and ask him if transaction log is not full, when it happens.
    Did you check DB logs??

    Did you make these changes??

    db2 update db cfg for yourDatabase using logsecond
    Specify a value of logsecond plus 12 for x.

    It doesnt matter if it is Identity manager or WESB since error is related to the database...
  • SystemAdmin
    SystemAdmin
    3556 Posts

    Re: Messages failed during certain period of time

    ‏2013-01-02T19:45:53Z  
    • gas
    • ‏2013-01-02T19:24:36Z
    Did you try it?

    It is exactly same error code as in your stack trace.
    Contact your DB2 admin and ask him if transaction log is not full, when it happens.
    Did you check DB logs??

    Did you make these changes??

    db2 update db cfg for yourDatabase using logsecond
    Specify a value of logsecond plus 12 for x.

    It doesnt matter if it is Identity manager or WESB since error is related to the database...
    Hi Gas,

    I didn't try it yet but got the statistics for the DB2 server.

    LOG_USED_MEG LOG_SPACE_FREE_MEG


    5 61

    also log file size 1024 # of primary log files 13 (before they are reused)

    So does it mean that still this DB might be the issue.

    As i said in my previous answer, last night when this happened I restarted the WESB server and then it started working fine till now. When we had this issue previously, after an hour it started working automatically, so I am not able to understand that if this issue is with the log file then why does it start working after restart or automatically?

    Please clarify and if still required I will try to work with DBA to change the values.

    Thanks,
    Hetal
  • zhangyu
    zhangyu
    179 Posts

    Re: Messages failed during certain period of time

    ‏2013-01-03T09:45:57Z  
    Hi Gas,

    I didn't try it yet but got the statistics for the DB2 server.

    LOG_USED_MEG LOG_SPACE_FREE_MEG


    5 61

    also log file size 1024 # of primary log files 13 (before they are reused)

    So does it mean that still this DB might be the issue.

    As i said in my previous answer, last night when this happened I restarted the WESB server and then it started working fine till now. When we had this issue previously, after an hour it started working automatically, so I am not able to understand that if this issue is with the log file then why does it start working after restart or automatically?

    Please clarify and if still required I will try to work with DBA to change the values.

    Thanks,
    Hetal
    from the error stack, i can see this problem happens on messaging engine database and as you said, the problem was gone after you restart webs server. based on the two evidences, I'm pretty sure you are facing a known issue:
    http://www-01.ibm.com/support/docview.wss?uid=swg21452323

    the solution is documented here as well:

    http://www.ibm.com/developerworks/websphere/techjournal/0807_webcon/0807_webcon.html

    just disable SIBOWNER table from db2 health monitor.