Business Processes stuck in Waiting on IO – a guide to data gathering.

Body

There are times when Active business processes may stay in the status of Waiting on IO. This is a status that can occur with any of the protocol adapters when they are acting as clients and it may occur in any of the adapter's client steps (i.e. begin session, list, put, post, etc).

The client services can go to this state during any file write/read operations on disk/database. Ideally, the waiting on io status is temporary and the business process continues its operations. However, sometimes the business process may hang around in this state longer than expected, raising questions as to why they are in this status.

When you encounter a business process in waiting on io indefinitely or longer than expected, IBM Support recommends the following information gathering in a PMR:

1. Enable the specific client adapter's logs in DEBUG mode or ON. From the SBI UI go to Operation → System → Logs and locate the specific client adapter logs. For example, for the HTTP client adapter, look for the httpclient.log and set that to ON. If SBI is on a cluster please enable these logs in each node of the cluster. Please note that this may generate quite a bit of logging. It is important to capture the exact timing of when the BP went into Waiting on IO in the log. Please submit all the SI logs under the install/logs directory when attempting to troubleshoot Waiting on IO issues.

2. Ensure that while the issue is occurring, you take a screenshot of the “Communication Sessions” for that process from the SBI UI. To do this, from the SBI UI, go to Business Processes → Monitor → Advanced Search → Communication Sessions.

   From that screen you can do a search either by specific protocol or a search by process id (business process id). The result will give you specifics of that communication that are helpful to locate any message or error in the SBI logs.

3. Take thread dumps while the issue is occurring. We need at least 3 thread dumps while the Business Processes are in the status of Waiting on IO. Details on how to generate a thread dumps are documented in the SBI documentation: https://www-304.ibm.com/support/knowledgecenter/SS3JSW_5.2.0/com.ibm.help.performance_mgmt.doc/SIPM_ThreadDump.html

   While the business processes that are in Waiting on IO may not show up in the thread dump, other communication information will that may help understand the state of the communication session from the JVM perspective.

4. Take a screenshot of the Business Process steps specifically the steps leading to the Waiting on IO status. We want to ensure we know which step was executed before(and after if applicable) the Waiting on IO status step.

5. Capture network traces at the SBI server level while the issue is occurring. This will give us insight into the network communication between SBI and the end point communication. Your network team should assist you on capturing network traces.

IBM Support will be able to assist on getting to the root cause of the Business Processes which end up in Waiting on IO indefinitely or longer than expected when these items are collected during the time of the issue.