PH02450: WS-AT KEEPS CONSUMING WEBCONTAINER THREADS

8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
9.0.0.11: WebSphere Application Server traditional V9.0 Fix Pack 11
9.0.5.0: WebSphere Application Server traditional Version 9.0.5 Refresh Pack
9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21

APAR status

• Closed as program error.

Error description

• When the root coordinator sends a prepare request via WS-AT to
  a downstream WS-AT Participant, and this is holding a lock. For
  some reason the prepare operation hangs on something. Later we
  can see many WebContainer threads hanging on WS-AT code as
  follows:
  3XMTHREADINFO "WebContainer : 62"
  J9VMThread:0x000000000E55CB00, j9thread_t:0x000003FFA2AFFC10,
  java/lang/Thread:0x00000000960A44F0, state:B, prio=5
  3XMJAVALTHREAD (java/lang/Thread getId:0xB77, isDaemon:true)
  3XMTHREADINFO1 (native thread ID:0xB44A, native priority:0x5,
  native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201)
  3XMTHREADINFO2 (native stack address range
  from:0x000003FFD6492000, to:0x000003FFD64D2000, size:0x40000)
  3XMCPUTIME CPU usage total: 0.485163181 secs
  3XMTHREADBLOCK Blocked on:
  com/ibm/ws/Transaction/JTS/TransactionWrapper@0x00000000951E99C8
  Owned by: "WebContainer : 10" (J9VMThread:0x0000000004340600,
  java/lang/Thread:0x000000009A026AE0)
  3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
  3XMTHREADINFO3 Java callstack:
  4XESTACKTRACE at
  com/ibm/ws/Transaction/wstx/WSATParticipantPortDistImpl.rollback
  Operation(WSATParticipantPortDistImpl.java:210(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/Transaction/wstx/WSATParticipantBindingImpl.rollbackO
  peration(WSATParticipantBindingImpl.java:183(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/Transaction/wstx/ns0410/ParticipantPortSoapBindingImp
  l.rollbackOperation(ParticipantPortSoapBindingImpl.java:104(Comp
  iled Code))
  4XESTACKTRACE at
  sun/reflect/GeneratedMethodAccessor277.invoke(Bytecode
  PC:40(Compiled Code))
  4XESTACKTRACE at
  sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
  AccessorImpl.java:56(Compiled Code))
  4XESTACKTRACE at
  java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/dispatchers/java/JavaDispatcher.in
  vokeMethod(JavaDispatcher.java:203(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/dispatchers/java/JavaDispatcher.in
  vokeOperation(JavaDispatcher.java:158(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/dispatchers/SoapRPCProcessor.proce
  ssOneway(SoapRPCProcessor.java:453(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/dispatchers/SoapRPCProcessor.proce
  ssMessage(SoapRPCProcessor.java:395(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/dispatchers/BasicDispatcher.proces
  sMessage(BasicDispatcher.java:134(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/dispatchers/java/SessionDispatcher
  .invoke(SessionDispatcher.java:204(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa
  ndlerWrapper.java:264(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/handlers/jaxrpc/JAXRPCHandler.invo
  ke(JAXRPCHandler.java:153(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/handlers/WrappedHandler.invoke(Wra
  ppedHandler.java:64(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa
  ndlerWrapper.java:264(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa
  ndlerWrapper.java:264(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/engine/WebServicesEngine.invoke(WebServic
  esEngine.java:336(Compiled Code))
  4XESTACKTRACE at
  com/ibm/ws/webservices/sysengine/SystemEngineRouter.doPost(Syste
  mEngineRouter.java:147(Compiled Code))
  ...
  

Local fix

• N/A
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:  WebSphere Application Server users of Web   *
  *                  Services Atomic Transactions                *
  ****************************************************************
  * PROBLEM DESCRIPTION: The WebContainer runs out of threads    *
  *                      after a hang in a Resource Adapter      *
  *                      processing a transaction completion     *
  *                      protocol request                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  An issue in a Resource Adapter caused it to hang while
  processing a transaction completion protocol request, for
  example processing an JTA-XA prepare request.  The transaction
  being completed was imported to the application server via
  Web Service Atomic Transaction (WS-AT) Context on a web service
  request.
  In such a case the superior transaction coordinator resends
  transaction completion protocol requests via the WS-AT
  protocol.  Such protocol requests are processed on
  WebContainer threads and will be periodically resent by the
  superior until it receives the corresponding WS-AT protocol
  response.
  The WS-AT component held an intrinsic lock while making the
  call to the Resource Adapter.  Each time the superior resent a
  WS-AT protocol request a new WebContainer thread was used to
  process the new request and each of these threads blocked while
  trying to obtain the same intrinsic lock.  Eventually the
  WebContainer ran out of threads.
  The root cause of the issue was the hanging Resource Adapter,
  however the WS-AT component must be able to cope with such an
  issue without adversely affecting other work.
  

Problem conclusion

• The handling of inbound WS-AT protocol requests was augmented
  with a mechanism for detecting multiple concurrent requests
  for the same transaction allowing a fast fail mechanism to be
  employed avoiding the unlimited consumption of threads.
  
  The fix for this APAR is currently targeted for inclusion in
  fix packs 8.5.5.15 and 9.0.0.11.  Please refer to the
  Recommended Updates page for delivery information:
  http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels

• R800 PSY

     UP

• R850 PSY

     UP

• R900 PSY

     UP