Topic
5 replies Latest Post - ‏2013-01-03T09:45:57Z by zhangyu
SystemAdmin
SystemAdmin
3556 Posts
ACCEPTED ANSWER

Pinned topic Messages failed during certain period of time

‏2012-12-27T22:29:49Z |
Hi All,

Please help.

I am pretty new to WebSphere ESB as well as IBM technologies. Yesterday suddenly some of the messaged related to certain applications started failing and then after an hour everything started working smoothly. I am not sure what happened. When I am checking AppTarget logs I am able to see below error:

0000002b LocalTransact E J2CA0024E: Method commit, within transaction branch ID <null> of resource pool SIB$SibRaConnectionFactory$DefaultPrincipalMapping$SCA_Auth_Alias, caught javax.resource.ResourceException: CWSIV0406E: An internal error occurred. The exception com.ibm.wsspi.sib.core.exception.SIRollbackException: CWSIC8007E: An exception was caught from the remote server with Probe Id 3-009-0007. Exception: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65. was thrown while the container was attempting to commit a local transaction.
at com.ibm.ws.sib.ra.impl.SibRaManagedConnection$SibRaLocalTransaction.commit(SibRaManagedConnection.java:1195)
at com.ibm.ejs.j2c.LocalTransactionWrapper.commit(LocalTransactionWrapper.java:263)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:942)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:830)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.end(LocalTranCoordImpl.java:1424)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.endLocalTx(UOWNativeWASStrategyImpl.java:1676)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionImportExport(UOWNativeWASStrategyImpl.java:447)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:168)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler$2.run(TargetSecurityContextHandler.java:165)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5365)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSpecified(ContextManagerImpl.java:5433)
at com.ibm.ws.sca.internal.managed.security.SecurityContextManager.runAs(SecurityContextManager.java:568)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler.processMessage(TargetSecurityContextHandler.java:161)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionLocalActivitySessionFalse(UOWNativeWASStrategyImpl.java:1319)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.deliverAsyncAtCall(UOWNativeWASStrategyImpl.java:86)
at com.ibm.ws.sca.uow.handler.DeliverAsyncAtUOWNativeHandler.processMessage(DeliverAsyncAtUOWNativeHandler.java:72)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.SCAInvocationAction.invokeAsyncOneWay(SCAInvocationAction.java:497)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeByStyle(RetryControl.java:373)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeWithRetry(RetryControl.java:245)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.ServiceInvocation.complete(ServiceInvocation.java:125)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.FlowActionFactoryImpl.create(FlowActionFactoryImpl.java:732)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.invokeFlowAction(ServiceInvokeMediation.java:663)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.mediate(ServiceInvokeMediation.java:576)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:739)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.SIBXJavaMediationPrimitive.invoke(SIBXJavaMediationPrimitive.java:108)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.Input.invoke(Input.java:228)
at com.ibm.ws.sibx.scax.mediation.engine.RequestFlow.invokeFlow(RequestFlow.java:135)
at com.ibm.ws.sibx.scax.mediation.engine.MediationFlow.invokeRequestFlow(MediationFlow.java:168)
at com.ibm.wsspi.sibx.mediation.flow.ejb.MediationFlowBean.invokeRequestFlow(MediationFlowBean.java:292)
at com.ibm.ws.sibx.scax.mediation.component.SIBXMediationFlowComponentImpl.invokeRequestFlow(SIBXMediationFlowComponentImpl.java:135)
at com.ibm.ws.sibx.scax.runtime.handler.MFCImplementationHandler.processMessage(MFCImplementationHandler.java:228)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:774)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:999)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:854)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:214)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:785)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessageWithSCAContext(HTTPExportHandler.java:1204)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessage(HTTPExportHandler.java:1180)
at com.ibm.ws.http.export.handler.HTTPExportHandler.processRequest(HTTPExportHandler.java:391)
at com.ibm.ws.http.export.handler.HTTPExportHandler.doPost(HTTPExportHandler.java:233)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1049)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:643)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)
Caused by: com.ibm.wsspi.sib.core.exception.SIRollbackException: CWSIC8007E: An exception was caught from the remote server with Probe Id 3-009-0007. Exception: CWSIS1002E: An unexpected exception was caught during transaction completion. Exception: com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
at com.ibm.ws.sib.comms.common.CommsByteBuffer.parseSingleException(CommsByteBuffer.java:1850)
at com.ibm.ws.sib.comms.common.CommsByteBuffer.getException(CommsByteBuffer.java:1263)
at com.ibm.ws.sib.comms.common.JFAPCommunicator.checkFor_SIRollbackException(JFAPCommunicator.java:1440)
at com.ibm.ws.sib.comms.client.LocalTransactionProxy.commit(LocalTransactionProxy.java:156)
at com.ibm.ws.sib.comms.client.OptimizedUncoordinatedTransactionProxy.commit(OptimizedUncoordinatedTransactionProxy.java:113)
at com.ibm.ws.sib.ra.impl.SibRaManagedConnection$SibRaLocalTransaction.commit(SibRaManagedConnection.java:1183)
... 86 more
Caused by: com.ibm.websphere.sib.exception.SIErrorException: CWSIC8002E: An internal error occurred. An unknown or unexpected exception was thrown by the core API: exception com.ibm.ws.sib.msgstore.PersistenceException: CWSIS1501E: The data source has produced an unexpected exception: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
at com.ibm.ws.sib.comms.common.CommsByteBuffer.parseSingleException(CommsByteBuffer.java:1884)
... 91 more
Caused by: com.ibm.websphere.sib.exception.SIErrorException: CWSIC8002E: An internal error occurred. An unknown or unexpected exception was thrown by the core API: exception com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
... 92 more
and threw a XAException.

Can someone please help and let me know what might be the issue.

Thanks,
Hetal
Updated on 2013-01-03T09:45:57Z at 2013-01-03T09:45:57Z by zhangyu
  • gas
    gas
    36 Posts
    ACCEPTED ANSWER

    Re: Messages failed during certain period of time

    ‏2012-12-28T09:18:11Z  in response to SystemAdmin
    Hi,

    This error:
    DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011
    suggest that transaction log is full. Check these links for some hints and possible solutions:

    http://www-01.ibm.com/support/docview.wss?uid=swg21377796
    http://publib.boulder.ibm.com/infocenter/tivihelp/v5r1/index.jsp?topic=%2Fcom.ibm.itim.infocenter.doc%2Fex460_pdg79.htm

    Gas
    • SystemAdmin
      SystemAdmin
      3556 Posts
      ACCEPTED ANSWER

      Re: Messages failed during certain period of time

      ‏2013-01-02T18:40:16Z  in response to gas
      Hello Gas,

      Thanks for the reply.... but looks like it will not help me :-(

      1st link : says I need to have this error : SQL0964C: The transaction log for the database is full.

      But if you will see my logs I am not getting that error.

      2nd link : is for Tivoli Identity Manager Express, where as I am working on WebSphere ESB so I think this will not help.

      Any other ideas?

      Last night also same issue happen,,, when I restarted the ESB server then it started working fine. But restarting server everytime is not the option for me. Please let me know what to do.

      Thanks,
      Hetal
      • gas
        gas
        36 Posts
        ACCEPTED ANSWER

        Re: Messages failed during certain period of time

        ‏2013-01-02T19:24:36Z  in response to SystemAdmin
        Did you try it?

        It is exactly same error code as in your stack trace.
        Contact your DB2 admin and ask him if transaction log is not full, when it happens.
        Did you check DB logs??

        Did you make these changes??

        db2 update db cfg for yourDatabase using logsecond
        Specify a value of logsecond plus 12 for x.

        It doesnt matter if it is Identity manager or WESB since error is related to the database...
        • SystemAdmin
          SystemAdmin
          3556 Posts
          ACCEPTED ANSWER

          Re: Messages failed during certain period of time

          ‏2013-01-02T19:45:53Z  in response to gas
          Hi Gas,

          I didn't try it yet but got the statistics for the DB2 server.

          LOG_USED_MEG LOG_SPACE_FREE_MEG


          5 61

          also log file size 1024 # of primary log files 13 (before they are reused)

          So does it mean that still this DB might be the issue.

          As i said in my previous answer, last night when this happened I restarted the WESB server and then it started working fine till now. When we had this issue previously, after an hour it started working automatically, so I am not able to understand that if this issue is with the log file then why does it start working after restart or automatically?

          Please clarify and if still required I will try to work with DBA to change the values.

          Thanks,
          Hetal
          • zhangyu
            zhangyu
            179 Posts
            ACCEPTED ANSWER

            Re: Messages failed during certain period of time

            ‏2013-01-03T09:45:57Z  in response to SystemAdmin
            from the error stack, i can see this problem happens on messaging engine database and as you said, the problem was gone after you restart webs server. based on the two evidences, I'm pretty sure you are facing a known issue:
            http://www-01.ibm.com/support/docview.wss?uid=swg21452323

            the solution is documented here as well:

            http://www.ibm.com/developerworks/websphere/techjournal/0807_webcon/0807_webcon.html

            just disable SIBOWNER table from db2 health monitor.