IBM Support

Performance Issues Due to Remote Resources: socketRead

Troubleshooting


Problem

Symptoms of performance or hang issues may be observed with any number of threads executing on java.net.SocketInputStream.socketRead for a prolonged period of time. The following will detail how to determine if such threads are problematic and offer some recommendations for further investigation.

Symptom

Many performance issues are not due to an issue with the application server (or JVM) itself but with one or more of the remote resources to which the application server is connected. Threads executing in socketRead for a prolonged period of time may be a symptom of such an issue. The following is an example of a thread executing in socketRead:

      "WebContainer : 0"
          at java/net/SocketInputStream.socketRead0()
          at java/net/SocketInputStream.read()
          at com.xyz.methodABC()
          ...

The bottom part of the stack leading up to the socketRead will vary depending on what type of remote resource is being used and what code is initiating the activity.

Diagnosing The Problem

Not every thread found in a state such as the above is problematic. The stack above is in fact a normal and expected state for any number of scenarios where the application server is reading data from a remote resource. For example, it is common to see threads in a similar state while they are retrieving results from a database or an LDAP server. Any time the application server is reading data from a remote resource, it is likely that socketRead is being used.

Suspicion only arises when we observe threads executing on socketRead for a prolonged period of time. That being said, it is also important to note that some prolonged socketRead operations may be expected or necessary (reader threads, large but expected result sets, ...).

There are a number of different ways to determine if a thread has been executing on socketRead for a prolonged period of time. Here are two specific methods:


    Method One: Using a series of javacores / thread dumps

    With a series of javacores or thread dumps, the state of a particular thread over a period of time can be analyzed. If a specific thread is executing on socketRead over the course of a series of javacores or thread dumps, this is likely to be an issue that warrants further investigation.

    Method Two: Using hung thread warning messages from the SystemOut.log

    By default, WebSphere Application Server will print a hung thread warning message to the SystemOut.log if a thread has been executing on the same method for more than 10 minutes. Further, in more recent versions of WebSphere Application Server the message also includes the stack of the potentially hung thread. If a thread has been marked as potentially hung and is executing on socketRead, a deeper investigation is likely in order.


Regardless of the method used above, once a thread of interest has been identified, the thread stack leading up to the socketRead must be analyzed in order to determine more information about the remote resource. What is the remote resource? Is the remote resource a database, an LDAP server, another WebSphere Application Server process, or something else altogether? What code or 'use case' is initiating the activity?

With an understanding of what type of remote resource is being used and how the activity is being initiated, steps to resolve the problem can be taken.

Resolving The Problem

The socketRead method is classified as a blocking operation. As such, this method will execute until all the data has been read from the remote resource or until the connection is terminated by the remote resource. There are any number of reasons why an executing socketRead could take longer than expected and result in a hang or performance issue. The points below outline a number of aspects to keep in mind while resolving this type of issue.

  1. What is the health of the remote resource at the time of the issue?
    • Health of database, LDAP, ...
    • Is the remote resource responding to any other requests?
  2. What is the health of the connection between the application server and the remote resource at the time of the issue?
    • Health of networking, firewalls, routers, ...
    • Is there anything impeding the free flow of data from one end to the other?
  3. How much data is being requested / sent by the application server / remote resource at the time of the issue?
    • Could it be that the application just requested a massive amount of data that is simply taking a long time to transfer?
    • Did the application server just execute a query for all the records in a database?
    • Did the application server just request all the users from an LDAP server?

Again, each situation is different and there are a number of different ways in which issues like these can come about. Using these points as a guide should assist in addressing performance issues with threads executing in socketRead for a prolonged period of time.

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5;8.0;7.0","Edition":"Base;Express;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
15 June 2018

UID

swg21610001