Application server may stop responding to requests and hung threads may be seen on Windows with Asynchronous I/O enabled

Content

Versions affected:
Versions 7.0.0.31 and 8.0.0.8 only.

Problem Description:
WebSphere Application Server V7.0.0.31 and V8.0.0.8 include the fix for APAR PM95795. A change introduced by the fix for this APAR may cause threads to hang and an application server to stop responding to incoming requests when running on a Windows operating system with AIO enabled. This problem will not occur on WebSphere Application Server V8.5, and it will not occur on other operating systems besides Windows.

When this problem occurs, the SystemOut.log will show many WSVR0605W messages indicating that there are possibly hung threads. The stack traces for these threads can be seen in the SystemOut.log or in a Javacore that is taken while the threads are hung. Many threads will be hung with a stack trace similar to the following:

Java callstack:
 at com/ibm/io/async/AsyncLibrary.terminateIOCB(AsyncLibrary.java:788(Compiled Code))
 at com/ibm/io/async/AsyncSocketChannel.close(AsyncSocketChannel.java:314(Compiled Code))
 at com/ibm/ws/tcp/channel/impl/AioSocketIOChannel.close(AioSocketIOChannel.java:435(Compiled Code))
 at com/ibm/ws/tcp/channel/impl/TCPConnLink.destroy(TCPConnLink.java:530(Compiled Code))
 at com/ibm/wsspi/channel/base/OutboundConnectorLink.close(OutboundConnectorLink.java:50(Compiled Code))
 at com/ibm/ws/tcp/channel/impl/TCPConnLink.close(TCPConnLink.java:506(Compiled Code))
 at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.close(HttpInboundLink.java:785(Compiled Code))
 at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.error(HttpICLReadCallback.java:141)
 at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:220(Compiled Code))
 at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
 at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
 at com/ibm/io/async/AbstractAsyncFuture.completed(AbstractAsyncFuture.java:170)
 at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:213(Compiled Code))
 at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:775(Compiled Code))
 at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
 at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1660(Compiled Code))

The Javacores will show that these threads are blocked by another thread with the following stack trace:

Java callstack:
 at com/ibm/io/async/AsyncLibrary.aio_cancel2(Native Method)
 at com/ibm/io/async/AsyncLibrary.cancel2(AsyncLibrary.java:462)
 at com/ibm/io/async/AbstractAsyncChannel.cancel(AbstractAsyncChannel.java:287)
 at com/ibm/io/async/AsyncChannelFuture.cancel(AsyncChannelFuture.java:114)
 at com/ibm/io/async/AsyncSocketChannelHelper$tCallback.timerTriggered(AsyncSocketChannelHelper.java:116)
 at com/ibm/io/async/Timer.checkForTimeouts(Timer.java:426(Compiled Code))
 at com/ibm/io/async/Timer.run(Timer.java:241(Compiled Code))


Solutions:
To work around the problem without installing any fixes, AIO can be disabled. This will result in Native I/O (NIO) being used, utilizing the Java Socket classes, rather than the direct utilization of the Windows libraries that is done by AIO. Disabling AIO could potentially cause a slight performance degradation in some environments. Instructions for disabling AIO can be found here.

Applying Interim Fix APAR PI15453, which replaces the original APAR PM95795 (as specified below), or a Fix Pack containing the APAR (as specified below), resolves this issue.

For IBM WebSphere Application Server for Distributed:

For V7.0.0.31:

• Apply Interim Fix APAR PI15453
  --OR--
• Apply Fix Pack 33 or later (V7.0.0.33 targeted to be available in June 2014)

• Apply Interim Fix APAR PI15453
  --OR--
• Apply Fix Pack 9 or later (V8.0.0.9 targeted to be available in June 2014)

• For Distributed, see Recommended fixes for WebSphere Application Server.
• For i5/OS, see WebSphere Application Server for i5/OS.
• For z/OS, see WebSphere Application Server for z/OS.