IBM Support

PH02450: WS-AT KEEPS CONSUMING WEBCONTAINER THREADS

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • When the root coordinator sends a prepare request via WS-AT to
    a downstream WS-AT Participant, and this is holding a lock. For
    some reason the prepare operation hangs on something. Later we
    can see many WebContainer threads hanging on WS-AT code as
    follows:
    3XMTHREADINFO "WebContainer : 62"
    J9VMThread:0x000000000E55CB00, j9thread_t:0x000003FFA2AFFC10,
    java/lang/Thread:0x00000000960A44F0, state:B, prio=5
    3XMJAVALTHREAD (java/lang/Thread getId:0xB77, isDaemon:true)
    3XMTHREADINFO1 (native thread ID:0xB44A, native priority:0x5,
    native policy:UNKNOWN, vmstate:B, vm thread flags:0x00000201)
    3XMTHREADINFO2 (native stack address range
    from:0x000003FFD6492000, to:0x000003FFD64D2000, size:0x40000)
    3XMCPUTIME CPU usage total: 0.485163181 secs
    3XMTHREADBLOCK Blocked on:
    com/ibm/ws/Transaction/JTS/TransactionWrapper@0x00000000951E99C8
    Owned by: "WebContainer : 10" (J9VMThread:0x0000000004340600,
    java/lang/Thread:0x000000009A026AE0)
    3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
    3XMTHREADINFO3 Java callstack:
    4XESTACKTRACE at
    com/ibm/ws/Transaction/wstx/WSATParticipantPortDistImpl.rollback
    Operation(WSATParticipantPortDistImpl.java:210(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/Transaction/wstx/WSATParticipantBindingImpl.rollbackO
    peration(WSATParticipantBindingImpl.java:183(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/Transaction/wstx/ns0410/ParticipantPortSoapBindingImp
    l.rollbackOperation(ParticipantPortSoapBindingImpl.java:104(Comp
    iled Code))
    4XESTACKTRACE at
    sun/reflect/GeneratedMethodAccessor277.invoke(Bytecode
    PC:40(Compiled Code))
    4XESTACKTRACE at
    sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethod
    AccessorImpl.java:56(Compiled Code))
    4XESTACKTRACE at
    java/lang/reflect/Method.invoke(Method.java:620(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/dispatchers/java/JavaDispatcher.in
    vokeMethod(JavaDispatcher.java:203(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/dispatchers/java/JavaDispatcher.in
    vokeOperation(JavaDispatcher.java:158(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/dispatchers/SoapRPCProcessor.proce
    ssOneway(SoapRPCProcessor.java:453(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/dispatchers/SoapRPCProcessor.proce
    ssMessage(SoapRPCProcessor.java:395(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/dispatchers/BasicDispatcher.proces
    sMessage(BasicDispatcher.java:134(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/dispatchers/java/SessionDispatcher
    .invoke(SessionDispatcher.java:204(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa
    ndlerWrapper.java:264(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/handlers/jaxrpc/JAXRPCHandler.invo
    ke(JAXRPCHandler.java:153(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/handlers/WrappedHandler.invoke(Wra
    ppedHandler.java:64(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa
    ndlerWrapper.java:264(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/PivotHandlerWrapper.invoke(PivotHa
    ndlerWrapper.java:264(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/engine/WebServicesEngine.invoke(WebServic
    esEngine.java:336(Compiled Code))
    4XESTACKTRACE at
    com/ibm/ws/webservices/sysengine/SystemEngineRouter.doPost(Syste
    mEngineRouter.java:147(Compiled Code))
    ...
    

Local fix

  • N/A
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  WebSphere Application Server users of Web   *
    *                  Services Atomic Transactions                *
    ****************************************************************
    * PROBLEM DESCRIPTION: The WebContainer runs out of threads    *
    *                      after a hang in a Resource Adapter      *
    *                      processing a transaction completion     *
    *                      protocol request                        *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    An issue in a Resource Adapter caused it to hang while
    processing a transaction completion protocol request, for
    example processing an JTA-XA prepare request.  The transaction
    being completed was imported to the application server via
    Web Service Atomic Transaction (WS-AT) Context on a web service
    request.
    In such a case the superior transaction coordinator resends
    transaction completion protocol requests via the WS-AT
    protocol.  Such protocol requests are processed on
    WebContainer threads and will be periodically resent by the
    superior until it receives the corresponding WS-AT protocol
    response.
    The WS-AT component held an intrinsic lock while making the
    call to the Resource Adapter.  Each time the superior resent a
    WS-AT protocol request a new WebContainer thread was used to
    process the new request and each of these threads blocked while
    trying to obtain the same intrinsic lock.  Eventually the
    WebContainer ran out of threads.
    The root cause of the issue was the hanging Resource Adapter,
    however the WS-AT component must be able to cope with such an
    issue without adversely affecting other work.
    

Problem conclusion

  • The handling of inbound WS-AT protocol requests was augmented
    with a mechanism for detecting multiple concurrent requests
    for the same transaction allowing a fast fail mechanism to be
    employed avoiding the unlimited consumption of threads.
    
    The fix for this APAR is currently targeted for inclusion in
    fix packs 8.5.5.15 and 9.0.0.11.  Please refer to the
    Recommended Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PH02450

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2018-09-04

  • Closed date

    2018-10-30

  • Last modified date

    2018-10-30

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R800 PSY

       UP

  • R850 PSY

       UP

  • R900 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
15 October 2021