PK51013: UNEXPECTED TIME-OUT EXCEPTIONS IN SERVER PROCESS BEING LOGGED TO SYSTEMOUT.LOG

APAR status

• Closed as program error.

Error description

• When running Websphere Application Server w/ Web Services
  Feature Pack sporadic timed out exceptions are showing up in the
   the SystemOut.log file. Here is a segment of the stacktrace:
  
  stacktrace:
  -----------------------------
  [6/13/07 22:02:56:167 CDT] 00000231 SenderWorker  E
  org.apache.sandesha2.workers.SenderWorker run Sandesha2 got an
  exception when
  sending a message: org.apache.axis2.AxisFault: WSWS7089I: The
  connection wait has
  timed out.; nested exception is:
   org.apache.axis2.AxisFault: WSWS7089I: The connection wait
  has timed out.; nested
  exception is:
   org.apache.axis2.AxisFault: WSWS7089I: The connection wait
  has timed out.; nested
  exception is:
   org.apache.axis2.AxisFault: WSWS7089I: The connection wait
  has timed out.; nested
  exception is:
   org.apache.axis2.AxisFault: WSWS7089I: The connection wait
  has timed out.; nested
  exception is:
   org.apache.axis2.AxisFault: WSWS7089I: The connection wait
  has timed out.; nested
  exception is:
   org.apache.axis2.AxisFault: WSWS7089I: The connection wait
  has timed out.; nested
  exception is:
   org.apache.axis2.AxisFault: WSWS7089I: The connection wait
  has timed out..
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: IBM Websphere Application Server users of    *
  *                 Web Service Feature Pack                     *
  ****************************************************************
  * PROBLEM DESCRIPTION: Unexpected timed out exceptions are     *
  *                      occurring due to timing window in       *
  *                      JAXWS code                              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The symptom is that on the server side, outbound connections
  hang. There is a pool size of 50, as we hang connections we
  effectively reduce the pool size, eventually exhausting it.
  After the pool is exhausted new connections queue up, and
  eventually timeout. Java core here would show a lot of threads
  in OutboundConnectionCache::
  findGroupAndReturnConnection()
  However, the server isn't the root cause. Looking at the
  message sent by one of the blocked connections, and it was an
  async response message, and that message was successfully
  delivered to the user code.
  Java core on the client side showed a lot of threads (50
  connections, in fact) waiting in the JAXWS CallbackFuture
  class, line 193. That's the place where there is a 3 minute
  sleep that is supposed to be notified() by the
  CallbackFutureTask.
  
  If the notify doesn't happen, then the transport connection is
  blocked for 3 minutes, and under heavy load this happens often
  enough that our pool of 50 connections are all used up. Under
  lighter load the system will sort itself out.
  

Problem conclusion

• Basically, the fix closes out the windows of failure (mentioned
  above)
  
  This fix is targeted for fixpack 6.1.0.11.
  Please refer to the recommended updates page for delivery
  information:
  http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• WEBSRVC
  

Fix information

• Fixed component name

  WEBSERVIC FEATU

• Fixed component ID

  5724J0850

Applicable component levels

• R610 PSY

     UP