Troubleshooting
Problem
When network latency issues exist, WebSphere Application Server might experience SOAP connectivity issues if a response is not received within 5 seconds of the original SOAP request.
Symptom
Symptoms include:
- addNode/removeNode command, wsadmin scripting, or SOAP connectivity errors
- "java.lang.SocketException: Connection reset" or "java.net.SocketTimeoutException: Read timed out" exceptions
Cause
Network latency/delays
Environment
WebSphere Application Server Express, Base, Network Deployment - All versions
Diagnosing The Problem
The following pieces of data can be gathered to help determine the cause of the SOAP connectivity issues:
1) Use the -trace option on the addNode/removeNode Qshell commands
i.e. app_server_root/bin/addNode <host> <port> -conntype SOAP -profileName <profile> -trace
2) Specify the following diagnostic trace string for the Deployment Manager you are attempting to connect to:
*=info:com.ibm.ws.management.*=all:com.ibm.websphere.management.application.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:TCPChannel=all:GenericBNF=all
For detailed instructions on how to set up diagnostic tracing in WebSphere Application Server, please refer to the following URL:
http://www-01.ibm.com/support/docview.wss?uid=swg21254706
3) TCP/IP traces from the both the DMGR and Application Server perspectives.
Use the TRCCNN command to gather the TCP/IP traces.
Command from the Application Server (who is connecting to the DMGR) perspective:
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP () (SOAPPORT))
TRCCNN SET(*OFF) TRCTBL(IBM)
Command from the DMGR perspective:
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP (SOAPPORT) ())
TRCCNN SET(*OFF) TRCTBL(IBM)
where SOAPPORT is the SOAP port of the DMGR you are attempting to connect to.
=======================================================================
com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke
[8/19/13 11:20:00:614 EDT] 00000088 SOAPUtils 3 unable to extract the SOAP Call
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.read(SocketInputStream.java:140)
at com.ibm.ws.io.Stream.read(Stream.java:17)
at com.ibm.ws.io.ReadStream.read(ReadStream.java:184)
at com.ibm.ws.http.ContentLengthInputStream.read(ContentLengthInputStream.java:49)
at com.ibm.ws.io.ReadStream.read(ReadStream.java:184)
at org.apache.soap.transport.TransportMessage.<init>(TransportMessage.java:141)
at org.apache.soap.server.ServerUtils.readEnvelopeFromInputStream(ServerUtils.java:107)
at com.ibm.ws.management.connector.soap.SOAPUtils.extractSoapCall(SOAPUtils.java:97)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:166)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
[8/19/13 11:20:00:615 EDT] 00000088 SOAPConnector 3 Caught exception
[SOAPException: faultCode=SOAP-ENV:Server.Exception; msg=ADMC0011E: The SOAP remote procedure call (RPC) cannot be unmarshalled.; targetException=java.net.SocketTimeoutException: Read timed out]
at com.ibm.ws.management.connector.soap.SOAPUtils.extractSoapCall(SOAPUtils.java:182)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:166)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
1) Use the -trace option on the addNode/removeNode Qshell commands
i.e. app_server_root/bin/addNode <host> <port> -conntype SOAP -profileName <profile> -trace
2) Specify the following diagnostic trace string for the Deployment Manager you are attempting to connect to:
*=info:com.ibm.ws.management.*=all:com.ibm.websphere.management.application.*=all:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:TCPChannel=all:GenericBNF=all
For detailed instructions on how to set up diagnostic tracing in WebSphere Application Server, please refer to the following URL:
http://www-01.ibm.com/support/docview.wss?uid=swg21254706
3) TCP/IP traces from the both the DMGR and Application Server perspectives.
Use the TRCCNN command to gather the TCP/IP traces.
Command from the Application Server (who is connecting to the DMGR) perspective:
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP () (SOAPPORT))
TRCCNN SET(*OFF) TRCTBL(IBM)
Command from the DMGR perspective:
TRCCNN SET(*ON) TRCTYPE(*IP) TRCTBL(IBM) SIZE(160000) TCPDTA(*TCP (SOAPPORT) ())
TRCCNN SET(*OFF) TRCTBL(IBM)
where SOAPPORT is the SOAP port of the DMGR you are attempting to connect to.
=======================================================================
Here are a few examples of typical error message that can occur when experiencing a SOAP connectivity issue with WebSphere Application Server:
[SOAPException: faultCode=SOAP-ENV:Client; msg=Connection reset; targetException=java.net.SocketException: Connection reset]com.ibm.websphere.management.exception.ConnectorException: ADMC0009E: The system failed to make the SOAP RPC call: invoke
[8/19/13 11:20:00:614 EDT] 00000088 SOAPUtils 3 unable to extract the SOAP Call
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.read(SocketInputStream.java:140)
at com.ibm.ws.io.Stream.read(Stream.java:17)
at com.ibm.ws.io.ReadStream.read(ReadStream.java:184)
at com.ibm.ws.http.ContentLengthInputStream.read(ContentLengthInputStream.java:49)
at com.ibm.ws.io.ReadStream.read(ReadStream.java:184)
at org.apache.soap.transport.TransportMessage.<init>(TransportMessage.java:141)
at org.apache.soap.server.ServerUtils.readEnvelopeFromInputStream(ServerUtils.java:107)
at com.ibm.ws.management.connector.soap.SOAPUtils.extractSoapCall(SOAPUtils.java:97)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:166)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
[8/19/13 11:20:00:615 EDT] 00000088 SOAPConnector 3 Caught exception
[SOAPException: faultCode=SOAP-ENV:Server.Exception; msg=ADMC0011E: The SOAP remote procedure call (RPC) cannot be unmarshalled.; targetException=java.net.SocketTimeoutException: Read timed out]
at com.ibm.ws.management.connector.soap.SOAPUtils.extractSoapCall(SOAPUtils.java:182)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:166)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
Resolving The Problem
Review the trace data (especially the TCP/IP traces) to determine if the time out is occurring due to a network delay. If you find the SOAP request/response is not being received by the DMGR in a timely manner, the client will need to work with their network administration team to resolve the latency in the network leading to the SOAP connection timeouts.
The following options are available as a temporary work around until the SOAP connectivity issue can be resolved:
a) Use the connection type of RMI and specify the bootstrap port on the addNode command.
The following options are available as a temporary work around until the SOAP connectivity issue can be resolved:
a) Use the connection type of RMI and specify the bootstrap port on the addNode command.
Configure the Deployment Manager and Node Agent to use the RMI connector.
i.e. addNode <DMGR_hostName> <DMGR_bootstrapPort> -conntype RMI -profilleName <profile>
How to configure the Deployment Manager to use the RMIConnector:
How to configure the Deployment Manager to use the RMIConnector:
1) Open a session to your WebSphere Application Server Integration Solutions Console in your web browser.
2) Expand the "System administration" section.
3) Click the "Deployment manager" link.
4) Under the "Additional Properties" section, click the "Administration services" link.
5) Change the "Remote connector" from SOAPConnector to RMIConnector.
6) Click the OK button and then the "Save" link at the top to save the changes to the master configuration.
7) Restart the Deployment Manager application server for the changes to take affect.
2) Expand the "System administration" section.
3) Click the "Deployment manager" link.
4) Under the "Additional Properties" section, click the "Administration services" link.
5) Change the "Remote connector" from SOAPConnector to RMIConnector.
6) Click the OK button and then the "Save" link at the top to save the changes to the master configuration.
7) Restart the Deployment Manager application server for the changes to take affect.
How to configure the Node Agent to use the RMIConnector:
1) Open a session to your WebSphere Application Server Integration Solutions Console in your web browser.
2) Expand the "System administration" section.
3) Click the "Node agents" link.
4) Click your node agent name.
5) Under the "Additional Properties" section, click the "Administration services" link.
6) Change the "Remote connector" from SOAPConnector to RMIConnector.
7) Click the OK button and then the "Save" link at the top to save the changes to the master configuration.
8) Restart the Node Agent application server for the changes to take affect.
2) Expand the "System administration" section.
3) Click the "Node agents" link.
4) Click your node agent name.
5) Under the "Additional Properties" section, click the "Administration services" link.
6) Change the "Remote connector" from SOAPConnector to RMIConnector.
7) Click the OK button and then the "Save" link at the top to save the changes to the master configuration.
8) Restart the Node Agent application server for the changes to take affect.
b) If the issue exists on the addNode/removeNode commands, you can set the following generic JVM argument for the Deployment Manager to increase the timeout wait period for SOAP connections.
1) Open a session to your WebSphere Application Server Integration Solutions Console in your web browser.
2) Expand the "System administration" section.
3) Click the "Deployment manager" link.
4) Under "Server Infrastructure", expand "Java and Process Management", and click the "Process definition" link.
5) Under "Additional Properties", click the "Java virtual machine" link.
6) Specify the following generic jvm argument in the "Generic JVM arguments" field. Each argument should be separated by a space.
-Dcom.ibm.ws.http.ConnectionIOTimeout=60000
NOTE: This property will increase the connection io timeout to 60 seconds from the default of 5 seconds. This value may need to be adjusted based on the amount of delay on your network. This is only a temporary work around and should not be used as a permanent solution. If you are having to increase this timeout, there is likely a bigger issue with your network; which is causing a delay in the SOAP requests/responses leading to the timeout of 5 seconds to be reached.
7) Restart the application server.
c) If the SOAP connection exists with an already federated node, follow the instructions in the URL below to implement the Read and Write timeout custom HTTP Transport properties to increase the connection timeout.2) Expand the "System administration" section.
3) Click the "Deployment manager" link.
4) Under "Server Infrastructure", expand "Java and Process Management", and click the "Process definition" link.
5) Under "Additional Properties", click the "Java virtual machine" link.
6) Specify the following generic jvm argument in the "Generic JVM arguments" field. Each argument should be separated by a space.
-Dcom.ibm.ws.http.ConnectionIOTimeout=60000
NOTE: This property will increase the connection io timeout to 60 seconds from the default of 5 seconds. This value may need to be adjusted based on the amount of delay on your network. This is only a temporary work around and should not be used as a permanent solution. If you are having to increase this timeout, there is likely a bigger issue with your network; which is causing a delay in the SOAP requests/responses leading to the timeout of 5 seconds to be reached.
7) Restart the application server.
https://www.ibm.com/docs/en/was/8.5.5?topic=cpctc-http-transport-channel-settings
============================================================================
For more information on Web Services client runtime errors, please refer the following URL:
https://www.ibm.com/docs/en/was/8.5.5?topic=tws-web-services-client-runtime-troubleshooting-tips
Related Information
[{"Type":"MASTER","Line of Business":{"code":"LOB57","label":"Power"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SWG60","label":"IBM i"},"ARM Category":[{"code":"a8m0z0000001iPIAAY","label":"WebSphere Application Server-\u003EApplications"}],"ARM Case Number":"","Platform":[{"code":"PF012","label":"IBM i"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
More support for:
IBM i
Component:
WebSphere Application Server->Applications
Software version:
All Versions
Operating system(s):
IBM i
Document number:
644887
Modified date:
06 July 2022
UID
nas8N1019742
Manage My Notification Subscriptions