PK32985: ORB DEADLOCKS WHEN THREADPOOL QUEUE IS FULL

6.1.0.7: WebSphere Application Server V6.1 Fix Pack 7 for Solaris
6.1.0.7: WebSphere Application Server V6.1 Fix Pack 7 for HP-UX
6.1.0.7: WebSphere Application Server V6.1 Fix Pack 7 for Linux
6.1.0.5: WebSphere Application Server V6.1.0 Fix Pack 5 for Linux
6.1.0.7: WebSphere Application Server V6.1 Fix Pack 7 for Windows
6.1.0.7 WebSphere Application Server V6.1 Fix Pack 7 for AIX
6.1.0.5: WebSphere Application Server V6.1.0 Fix Pack 5 for AIX
6.1.0.5: WebSphere Application Server V6.1.0 Fix Pack 5 for i5/OS
6.1.0.7: WebSphere Application Server V6.1 Fix Pack 7 for i5/OS
6.1.0.5: WebSphere Application Server V6.1.0 Fix Pack 5 for HP-UX
6.1.0.5: WebSphere Application Server V6.1.0 Fix Pack 5 for Windows
Java SDK 1.5 SR8 Cumulative Fix for WebSphere Application Server
Java SDK 1.5 SR8 Cumulative Fix for WebSphere Application Server
Java SDK 1.5 SR10 Cumulative Fix for WebSphere Application Server
6.1.0.31: Java SDK 1.5 SR11 FP1 Cumulative Fix for WebSphere Application Server
6.1.0.33: Java SDK 1.5 SR12 FP1 Cumulative Fix for WebSphere
6.1.0.29: Java SDK 1.5 SR11 Cumulative Fix for WebSphere Application Server
6.1.0.35: Java SDK 1.5 SR12 FP2 Cumulative Fix for WebSphere
6.1.0.37: Java SDK 1.5 SR12 FP3 Cumulative Fix for WebSphere
6.1.0.39: Java SDK 1.5 SR12 FP4 Cumulative Fix for WebSphere Application Server
6.1.0.41: Java SDK 1.5 SR12 FP5 Cumulative Fix for WebSphere Application Server
6.1.0.43: Java SDK 1.5 SR13 Cumulative Fix for WebSphere Application Server
6.1.0.45: Java SDK 1.5 SR14 Cumulative Fix for WebSphere Application Server
6.1.0.47: WebSphere Application Server V6.1 Fix Pack 47
6.1.0.47: Java SDK 1.5 SR16 Cumulative Fix for WebSphere Application Server
6.1.0.5: WebSphere Application Server V6.1.0 Fix Pack 5 for Solaris
6.1.0.9: WebSphere Application Server V6.1 Fix Pack 9 for Solaris

APAR status

• Closed as program error.

Error description

• Thread dump shows that one thread that has locked a
  connection and is IO blocked in socketWrite0. There are
  multiple threads waiting to lock the same connection and send
  the data. The waiting threads show that the Reader thread
  associated with the port in question are hung in
  Object.wait()...
  
  "RT=96:P=668372:O=0:WSSSLTransportConnection
  [addr=xxx.xxx.xxx.xxx,port=31029,local=10102]" daemon prio=5
  tid=0x01849828 nid=0x9d9 in Object.wait()
  [0xcf87f000..0xcf87fc30]
  at java.lang.Object.wait(Native Method)
  at java.lang.Object.wait(Object.java:429)
  at com.ibm.ws.util.BoundedBuffer.put(BoundedBuffer.java:171)
  - locked <0xdb3b3908> (a
  com.ibm.ws.util.BoundedBuffer$BoundedBufferLock)
  at com.ibm.ws.util.ThreadPool.execute(ThreadPool.java:1125)
  at com.ibm.ws.util.ThreadPool.execute(ThreadPool.java:992)
  at com.ibm.ejs.oa.pool.ThreadPool.startWorkerThread
  (ThreadPool.java:66)
  at
  com.ibm.rmi.iiop.Connection.processInput(Connection.java:1325)
  at com.ibm.rmi.iiop.Connection.doReaderWorkOnce
  (Connection.java:2666)
  at com.ibm.rmi.transport.ReaderThread.run
  (ReaderPoolImpl.java:137)
  
  Since the Reader thread is blocked here, the data from the
  connection is not read and causes the buffer to be full and
  threads on the other side seen in socketWrite0.
  

Local fix

• Restart the server to clear the hang
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: WebSphere Application Server version 6       *
  *                 users with exceptionally large amounts of    *
  *                 Remote Method Invocation (RMI) requests.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: The Object Request Broker (ORB) can     *
  *                      enter a state where it is unable to     *
  *                      process requests without restarting the *
  *                      server.                                 *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  Each inbound and outbound request that is made via the ORB
  requires a thread from the ORB thread pool.  In heavy load
  scenarios or scenarios where ORB requests nest deeply, it is
  possible for a Java Virtual Machine (JVM) to have all threads
  from the ORB thread pool attempting to send requests.
  Meanwhile, the remote JVM ORB that would process these requests
  also has all threads from its ORB thread pool attempting to
  send requests. As a result, no progress is ever made, no
  threads are released back to the ORB thread pool, and the ORB
  is unable to process requests.
  

Problem conclusion

• A new ORB custom property,
  com.ibm.websphere.orb.threadPoolTimeout, was added that
  controls the maximum time in milliseconds that the ORB will
  wait for a thread from the ORB thread pool before rejecting a
  request.  The default value is 0, which corresponds to the
  current behavior of waiting indefinitely.
  
  The fix for this APAR is currently targeted for inclusion in
  fixpaks 6.0.2.19 and 6.1.0.5.
  Please refer to the recommended updates page for delivery
  information:
  http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• RUNTIME
  

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels

• R60A PSY

     UP

• R60H PSY

     UP

• R60I PSY

     UP

• R60P PSY

     UP

• R60S PSY

     UP

• R60W PSY

     UP

• R60Z PSY

     UP

• R61A PSY

     UP

• R61H PSY

     UP

• R61I PSY

     UP

• R61P PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R61Z PSY

     UP