Topic
No replies
SystemAdmin
SystemAdmin
289 Posts
ACCEPTED ANSWER

Pinned topic Hanging threads in ESB

‏2013-03-27T21:36:19Z |
Hi Team,

We had an issue in our landscape ESB is communicating with Oracle. In this case Oracle had recompilation of the packages and because of that ESB was not getting the response back (success or fail response for the request) from Oracle. This made the threads to wait for infinite time (not sure where to set up the time out for this)and finally we ran out of the threads.

Any idea that in this type of cases, how can i set up a timeout for the threads so that this will not wait for infinite time and give timeout and free up the thread?

This is the error that I was able to see at the time of issue:

3/23/13 4:19:23:739 PDT 0000001a ThreadMonitor W WSVR0605W: Thread "WebContainer : 13" (00000ad8) has been active for 769485 milliseconds and may be hung. There is/are 5 thread(s) in total in the server that may be hung.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:167)
at com.ibm.ws.sib.jfapchannel.impl.ExchangeReceiveListener.waitToComplete(ExchangeReceiveListener.java:227)
at com.ibm.ws.sib.jfapchannel.impl.ConversationImpl.exchange(ConversationImpl.java:988)
at com.ibm.ws.sib.comms.common.JFAPCommunicator.jfapExchange(JFAPCommunicator.java:386)
at com.ibm.ws.sib.comms.client.LocalTransactionProxy.commit(LocalTransactionProxy.java:147)
at com.ibm.ws.sib.comms.client.OptimizedUncoordinatedTransactionProxy.commit(OptimizedUncoordinatedTransactionProxy.java:113)
at com.ibm.ws.sib.ra.impl.SibRaManagedConnection$SibRaLocalTransaction.commit(SibRaManagedConnection.java:1183)
at com.ibm.ejs.j2c.LocalTransactionWrapper.commit(LocalTransactionWrapper.java:263)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:942)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.complete(LocalTranCoordImpl.java:830)
at com.ibm.ws.LocalTransaction.LocalTranCoordImpl.end(LocalTranCoordImpl.java:1424)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.endLocalTx(UOWNativeWASStrategyImpl.java:1676)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionImportExport(UOWNativeWASStrategyImpl.java:447)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:168)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler$2.run(TargetSecurityContextHandler.java:165)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5365)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSpecified(ContextManagerImpl.java:5433)
at com.ibm.ws.sca.internal.managed.security.SecurityContextManager.runAs(SecurityContextManager.java:568)
at com.ibm.ws.sca.internal.securitycontext.handler.TargetSecurityContextHandler.processMessage(TargetSecurityContextHandler.java:161)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionLocalActivitySessionFalse(UOWNativeWASStrategyImpl.java:1319)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.deliverAsyncAtCall(UOWNativeWASStrategyImpl.java:86)
at com.ibm.ws.sca.uow.handler.DeliverAsyncAtUOWNativeHandler.processMessage(DeliverAsyncAtUOWNativeHandler.java:72)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1775)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.SCAInvocationAction.invokeAsyncOneWay(SCAInvocationAction.java:497)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeByStyle(RetryControl.java:373)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.RetryControl.invokeWithRetry(RetryControl.java:245)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.ServiceInvocation.complete(ServiceInvocation.java:125)
at com.ibm.ws.sibx.mediation.flowaction.impl.sca.FlowActionFactoryImpl.create(FlowActionFactoryImpl.java:732)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.invokeFlowAction(ServiceInvokeMediation.java:663)
at com.ibm.ws.sibx.mediation.primitives.serviceinvoke.ServiceInvokeMediation.mediate(ServiceInvokeMediation.java:576)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:739)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.SIBXJavaMediationPrimitive.invoke(SIBXJavaMediationPrimitive.java:108)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminalOnly(JavaMediationPrimitive.java:558)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:433)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminalOnly(JavaMediationPrimitive.java:558)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:433)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.SIBXJavaMediationPrimitive.invoke(SIBXJavaMediationPrimitive.java:108)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminalOnly(JavaMediationPrimitive.java:558)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:433)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.SIBXJavaMediationPrimitive.invoke(SIBXJavaMediationPrimitive.java:108)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.fireOutputTerminals(JavaMediationPrimitive.java:841)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocation(JavaMediationPrimitive.java:761)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.performInvocationWithCatch(JavaMediationPrimitive.java:575)
at com.ibm.ws.sibx.scax.mediation.engine.JavaMediationPrimitive.invoke(JavaMediationPrimitive.java:395)
at com.ibm.ws.sibx.scax.mediation.engine.MediationPrimitive.invokeConnections(MediationPrimitive.java:344)
at com.ibm.ws.sibx.scax.mediation.engine.Input.invoke(Input.java:228)
at com.ibm.ws.sibx.scax.mediation.engine.RequestFlow.invokeFlow(RequestFlow.java:135)
at com.ibm.ws.sibx.scax.mediation.engine.MediationFlow.invokeRequestFlow(MediationFlow.java:168)
at com.ibm.wsspi.sibx.mediation.flow.ejb.MediationFlowBean.invokeRequestFlow(MediationFlowBean.java:292)
at com.ibm.ws.sibx.scax.mediation.component.SIBXMediationFlowComponentImpl.invokeRequestFlow(SIBXMediationFlowComponentImpl.java:135)
at com.ibm.ws.sibx.scax.runtime.handler.MFCImplementationHandler.processMessage(MFCImplementationHandler.java:228)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:774)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:999)
at com.ibm.ws.sca.uow.handler.UOWNativeWASStrategyImpl.transactionGlobalActivitySessionAny(UOWNativeWASStrategyImpl.java:854)
at com.ibm.ws.sca.uow.handler.JoinUOWNativeHandler.processMessage(JoinUOWNativeHandler.java:214)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessageWithPCI(MessageDispatcherImpl.java:785)
at com.ibm.ws.sca.internal.message.impl.MessageDispatcherImpl.processMessage(MessageDispatcherImpl.java:1626)
at com.ibm.ws.sca.internal.message.impl.ManagedMessageImpl.process(ManagedMessageImpl.java:990)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessageWithSCAContext(HTTPExportHandler.java:1204)
at com.ibm.ws.http.export.handler.HTTPExportHandler.sendSCAMessage(HTTPExportHandler.java:1180)
at com.ibm.ws.http.export.handler.HTTPExportHandler.processRequest(HTTPExportHandler.java:391)
at com.ibm.ws.http.export.handler.HTTPExportHandler.doPost(HTTPExportHandler.java:233)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:816)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)

Thanks,
Hetal