IBM Support

Application server may stop responding to requests and hung threads may be seen on Windows with Asynchronous I/O enabled

Flashes (Alerts)


Abstract

After WebSphere Application Server V7.0 Fix Pack 31 (V7.0.0.31) or V8.0 Fix Pack 8 (V8.0.0.8) is installed, threads may hang and an application server may stop responding to requests due to a problem introduced by APAR PM95795, which is included in these fix pack levels. The problem will only occur on Windows operating systems, and it will only occur if Asynchronous I/O (AIO) is enabled. AIO is enabled by default.

Content

Versions affected:
Versions 7.0.0.31 and 8.0.0.8 only.

Problem Description:
WebSphere Application Server V7.0.0.31 and V8.0.0.8 include the fix for APAR PM95795. A change introduced by the fix for this APAR may cause threads to hang and an application server to stop responding to incoming requests when running on a Windows operating system with AIO enabled. This problem will not occur on WebSphere Application Server V8.5, and it will not occur on other operating systems besides Windows.

When this problem occurs, the SystemOut.log will show many WSVR0605W messages indicating that there are possibly hung threads. The stack traces for these threads can be seen in the SystemOut.log or in a Javacore that is taken while the threads are hung. Many threads will be hung with a stack trace similar to the following:

Java callstack:
 at com/ibm/io/async/AsyncLibrary.terminateIOCB(AsyncLibrary.java:788(Compiled Code))
 at com/ibm/io/async/AsyncSocketChannel.close(AsyncSocketChannel.java:314(Compiled Code))
 at com/ibm/ws/tcp/channel/impl/AioSocketIOChannel.close(AioSocketIOChannel.java:435(Compiled Code))
 at com/ibm/ws/tcp/channel/impl/TCPConnLink.destroy(TCPConnLink.java:530(Compiled Code))
 at com/ibm/wsspi/channel/base/OutboundConnectorLink.close(OutboundConnectorLink.java:50(Compiled Code))
 at com/ibm/ws/tcp/channel/impl/TCPConnLink.close(TCPConnLink.java:506(Compiled Code))
 at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.close(HttpInboundLink.java:785(Compiled Code))
 at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.error(HttpICLReadCallback.java:141)
 at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:220(Compiled Code))
 at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
 at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code))
 at com/ibm/io/async/AbstractAsyncFuture.completed(AbstractAsyncFuture.java:170)
 at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:213(Compiled Code))
 at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:775(Compiled Code))
 at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
 at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1660(Compiled Code))

The Javacores will show that these threads are blocked by another thread with the following stack trace:

Java callstack:
 at com/ibm/io/async/AsyncLibrary.aio_cancel2(Native Method)
 at com/ibm/io/async/AsyncLibrary.cancel2(AsyncLibrary.java:462)
 at com/ibm/io/async/AbstractAsyncChannel.cancel(AbstractAsyncChannel.java:287)
 at com/ibm/io/async/AsyncChannelFuture.cancel(AsyncChannelFuture.java:114)
 at com/ibm/io/async/AsyncSocketChannelHelper$tCallback.timerTriggered(AsyncSocketChannelHelper.java:116)
 at com/ibm/io/async/Timer.checkForTimeouts(Timer.java:426(Compiled Code))
 at com/ibm/io/async/Timer.run(Timer.java:241(Compiled Code))


Solutions:
To work around the problem without installing any fixes, AIO can be disabled. This will result in Native I/O (NIO) being used, utilizing the Java Socket classes, rather than the direct utilization of the Windows libraries that is done by AIO. Disabling AIO could potentially cause a slight performance degradation in some environments. Instructions for disabling AIO can be found here.

Applying Interim Fix APAR PI15453, which replaces the original APAR PM95795 (as specified below), or a Fix Pack containing the APAR (as specified below), resolves this issue.

For IBM WebSphere Application Server for Distributed:

For V7.0.0.31:

  • Apply Interim Fix APAR PI15453
    --OR--
  • Apply Fix Pack 33 or later (V7.0.0.33 targeted to be available in June 2014)

For V8.0.0.8:
  • Apply Interim Fix APAR PI15453
    --OR--
  • Apply Fix Pack 9 or later (V8.0.0.9 targeted to be available in June 2014)

Additional documentation:
For additional details and information on WebSphere Application Server product updates:

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Channel Framework","Platform":[{"code":"PF033","label":"Windows"}],"Version":"8.0.0.8;7.0.0.31","Edition":"Base;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
15 June 2018

UID

swg21670780