Topic
  • No replies
Brent_Shikoski
Brent_Shikoski
7 Posts

Pinned topic Soap connector hang

‏2013-03-19T15:41:42Z |
We are seeing these errors over and over again. Usually the are fairly sporadic, currently they are extremely frequent.

nodeagent:
3/19/13 9:47:40:140 CDT 00000003 ThreadMonitor W WSVR0605W: Thread "SoapConnectorThreadPool : 0" (00000020) has been active for 647459 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:196)
at com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:334)
at com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:182)
at com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext.read(SSLReadServiceContext.java:265)
at com.ibm.ws.management.connector.ipc.ClientAccessor.clientReadMoreData(ClientAccessor.java:264)
at com.ibm.ws.management.connector.ipc.ClientAccessor.request(ClientAccessor.java:169)
at com.ibm.ws.management.connector.ipc.IPCConnectorClient.sendReceive(IPCConnectorClient.java:470)
at com.ibm.ws.management.connector.ipc.IPCConnectorClient.queryNames(IPCConnectorClient.java:814)
at com.ibm.ws.management.AdminClientImpl.queryNames(AdminClientImpl.java:108)
at com.ibm.ws.management.AdminServiceImpl.queryNames(AdminServiceImpl.java:664)
at com.ibm.ws.management.connector.AdminServiceDelegator.queryNames(AdminServiceDelegator.java:113)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.ibm.ws.management.connector.soap.SOAPConnector.invoke(SOAPConnector.java:480)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:309)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)
3/19/13 10:09:29:552 CDT 00000020 ThreadMonitor W WSVR0606W: Thread "SoapConnectorThreadPool : 0" (00000020) was previously reported to be hung but has completed. It was active for approximately 1956879 milliseconds. There is/are 0 thread(s) in total in the server that still may be hung.
dmgr:
3/19/13 9:18:11:719 CDT 00000003 ThreadMonitor W WSVR0605W: Thread "WebContainer : 3" (00000031) has been active for 679371 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:140)
at com.ibm.jsse2.b.a(b.java:232)
at com.ibm.jsse2.b.a(b.java:95)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:116)
at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:143)
at com.ibm.jsse2.f.read(f.java:51)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:229)
at java.io.BufferedInputStream.read(BufferedInputStream.java:248)
at org.apache.soap.util.net.HTTPUtils.post(HTTPUtils.java:590)
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:425)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:451)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient$8.run(SOAPConnectorClient.java:844)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:837)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:682)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:672)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.queryNames(SOAPConnectorClient.java:584)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:509)
at $Proxy32.queryNames(Unknown Source)
at com.ibm.ws.management.AdminClientImpl.queryNames(AdminClientImpl.java:108)
at com.ibm.ws.management.AdminServiceImpl.queryNames(AdminServiceImpl.java:664)
at com.ibm.ws.management.commands.AdminServiceCommands$QueryNamesCmd.execute(AdminServiceCommands.java:182)
at com.ibm.ws.console.core.mbean.MBeanHelper.queryNames(MBeanHelper.java:519)
at com.ibm.ws.console.core.mbean.MBeanHelper.getMBeans(MBeanHelper.java:140)
at com.ibm.ws.console.resources.database.jca.lifecycle.ManageMessageEndpointsController.populateDetailForms(ManageMessageEndpointsController.java:135)
at com.ibm.ws.console.resources.database.jca.lifecycle.ManageMessageEndpointsController.setupCollectionForm(ManageMessageEndpointsController.java:104)
at com.ibm.ws.console.core.action.BaseController.perform(BaseController.java:402)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(Unknown Source)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(Unknown Source)
at org.apache.struts.action.RequestProcessor.process(Unknown Source)
at org.apache.struts.action.ActionServlet.process(Unknown Source)
at org.apache.struts.action.ActionServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1229)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.setUpCommandAssistance(WSCUrlFilter.java:950)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.continueStoringTaskState(WSCUrlFilter.java:499)
at com.ibm.ws.console.core.servlet.WSCUrlFilter.doFilter(WSCUrlFilter.java:320)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3751)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:962)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)
WebSphere Platform 8.0.0.5 http://ND 8.0.0.5 cf051243.01
Host Operating System is AIX, version 6.1
Java version = 1.6.0, Java Compiler = j9jit26, Java VM name = IBM J9 VM

Any suggestions on how to fix to debug this?
Enabled trace logging on the node agent:

*=info:com.ibm.ws.management.connector.ipc.ClientAccessor=all:com.ibm.ws.management.connector.ipc.IPCConnectorClient=all:com.ibm.ws.management.AdminClientImpl=all.

The output hasn't been much help to me. Yes, I have restarted all the node agents and the deployment manager.