IBM Support

PI73588: Deadlock occurs in SIP Proxy under heavy TCP load.

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A hanglike situation may be encountered due to a deadlock in
    SIP code.  Javacores may show a deadlock similar to the
    following:
    
    Deadlock detected !!!
    ---------------------
    
     Thread "Proxy : 48" (0x000000004399D400)
       is waiting for:
         sys_mon_t:0x00000000670C63A8 infl_mon_t:0x00000000670C6420:
         java/util/Vector@0x00000000113C3498
       which is owned by:
     Thread "Default : 4" (0x0000000043A0B000)
       which is waiting for:
         sys_mon_t:0x0000000067066688 infl_mon_t:0x0000000067066700:
    
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler@0x00
    00000011385AA8
       which is owned by:
     Thread "Proxy : 48" (0x000000004399D400)
    
    
    When examining the threads in the deadlock:
    
    
    "Proxy : 48"
    J9VMThread:0x000000004399D400,j9thread_t:0x0000000000E07FC0,
    java/lang/Thread:0x000000000AAAE7E0,state:B, prio=5
           (java/lang/Thread getId:0xED, isDaemon:true)
           (native thread ID:0x1814, native priority:0x5, native
    policy:UNKNOWN, vmstate:B, vm thread flags:0x01000201)
          CPU usage total: 1098.543441900 secs, user: 693.315244300
    secs, system: 405.228197600 secs
    
    Blocked on: java/util/Vector@0x00000000113C3498 Owned by:
    "Default : 4"(J9VMThread:0x0000000043A0B000,
    java/lang/Thread:0x000000000492EE58)
          Heap bytes allocated since last GC cycle=0 (0x0)
    
          Java callstack:
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection.shutdown(SipProx
    yConnection.java:1352(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$ReadHandler.erro
    r(SipProxyConnection.java:2730(Compiled Code))
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler.erro
    r(SIPTcpConnection.java:736(Compiled Code))
              at
    com/ibm/ws/tcp/channel/impl/AioTCPReadRequestContextImpl.process
    AsyncReadRequest(AioTCPReadRequestContextImpl.java:141(Compiled
    Code))
              at
    com/ibm/ws/tcp/channel/impl/TCPReadRequestContextImpl.readIntern
    al(TCPReadRequestContextImpl.java:169(Compiled Code))
              at
    com/ibm/ws/tcp/channel/impl/TCPReadRequestContextImpl.read(TCPRe
    adRequestContextImpl.java:142(Compiled Code))
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler.read
    (SIPTcpConnection.java:387(Compiled Code))
                 (entered
    lock:com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler
    @0x0000000011385AA8, entry count: 1)
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$ReadHandler.read
    Data(SipProxyConnection.java:2787(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$ReadHandler.mess
    ageReceived(SipProxyConnection.java:2743(Compiled Code))
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler.comp
    lete(SIPTcpConnection.java:613(Compiled Code))
              at
    com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureComp
    leted(AioReadCompletionListener.java:175(Compiled Code))
    
    
    
    "Default : 4"
    J9VMThread:0x0000000043A0B000,j9thread_t:0x0000000000E0B630,
    java/lang/Thread:0x000000000492EE58,state:B, prio=5
           (java/lang/Thread getId:0xA1, isDaemon:true)
           (native thread ID:0x1930, native priority:0x5, native
    policy:UNKNOWN, vmstate:B, vm thread flags:0x01000201)
          CPU usage total: 789.723862300 secs, user: 529.014991100
    secs,system: 260.708871200 secs
    
    Blocked on:
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler@0x00
    00000011385AA8 Owned by: "Proxy : 48"
    (J9VMThread:0x000000004399D400,java/lang/Thread:0x000000000AAAE7
    E0)
          Heap bytes allocated since last GC cycle=0 (0x0)
    
          Java callstack:
    
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler.clea
    nup(SIPTcpConnection.java:743(Compiled Code))
                 (entered
    lock:com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler
    @0x0000000011385AA8, entry count: 1)
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection.destroy(SIPTcpConne
    ction.java:189(Compiled Code))
              at
    com/ibm/ws/sip/channel/inbound/impl/SIPTcpInboundConnLink.destro
    y(SIPTcpInboundConnLink.java:69(Compiled Code))
              at
    com/ibm/wsspi/channel/base/OutboundConnectorLink.destroy(Outboun
    dConnectorLink.java:63(Compiled Code))
              at
    com/ibm/ws/tcp/channel/impl/TCPConnLink.destroy(TCPConnLink.java
    :540(Compiled Code))
              at
    com/ibm/wsspi/channel/base/OutboundConnectorLink.close(OutboundC
    onnectorLink.java:50(Compiled Code))
              at
    com/ibm/ws/tcp/channel/impl/TCPConnLink.close(TCPConnLink.java:5
    06(Compiled Code))
              at
    com/ibm/ws/sip/channel/inbound/impl/SIPTcpInboundConnLink.close(
    SIPTcpInboundConnLink.java:103(Compiled Code))
              at
    com/ibm/wsspi/channel/base/InboundApplicationLink.close(InboundA
    pplicationLink.java:58(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnectionLink.close(SipPro
    xyConnectionLink.java:150(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection.closeConnection(
    SipProxyConnection.java:1264(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$WriteHandler.err
    or(SipProxyConnection.java:2945(Compiled Code))
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPWriteHandler.err
    or(SIPTcpConnection.java:1037(Compiled Code))
              at
    com/ibm/ws/tcp/channel/impl/AioTCPWriteRequestContextImpl.proces
    sAsyncWriteRequest(AioTCPWriteRequestContextImpl.java:103(Compil
    ed Code))
              at
    com/ibm/ws/tcp/channel/impl/TCPWriteRequestContextImpl.writeInte
    rnal(TCPWriteRequestContextImpl.java:382(Compiled Code))
              at
    com/ibm/ws/tcp/channel/impl/TCPWriteRequestContextImpl.write(TCP
    WriteRequestContextImpl.java:353(Compiled Code))
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPWriteHandler.wri
    te(SIPTcpConnection.java:897(Compiled Code))
                 (entered lock:
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPWriteHandler@0x0
    000000011385AF8, entry count: 1)
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$WriteHandler.flu
    shWriteQueue(SipProxyConnection.java:3084(Compiled Code))
                 (entered lock:
    java/util/Vector@0x00000000113C3498, entrycount: 1)
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$WriteHandler.acc
    ess$000(SipProxyConnection.java:2890(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection.writeMessage(Sip
    ProxyConnection.java:1214(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection.processMessage(S
    ipProxyConnection.java:2141(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection.readIndication(S
    ipProxyConnection.java:1733(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection.access$600(SipPr
    oxyConnection.java:106(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$ReadHandler.read
    Data(SipProxyConnection.java:2771(Compiled Code))
              at
    com/ibm/ws/proxy/channel/sip/SipProxyConnection$ReadHandler.mess
    ageReceived(SipProxyConnection.java:2743(Compiled Code))
              at
    com/ibm/ws/sip/channel/impl/SIPTcpConnection$SIPReadHandler.comp
    lete(SIPTcpConnection.java:613(Compiled Code))
    

Local fix

  • Restarting the server may temporarily relieve the issue.
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere SIP Proxy Server *
    *                                                              *
    ****************************************************************
    * PROBLEM DESCRIPTION: Deadlock occurs in the SIP Proxy when   *
    *                      using TCP for transporting SIP          *
    *                      messages.                               *
    *                      This occurs under extremely heavy       *
    *                      load.                                   *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Under extremely heavy loads (and usually with full traces
    enabled), we see a deadlock occur between the SIP Channel and
    SIP
    Proxy code.   The "deadlock detected" notification will be shown
    in a javacore.
    

Problem conclusion

Temporary fix

  • N/A
    

Comments

APAR Information

  • APAR number

    PI73588

  • Reported component name

    WEBS APP SERV N

  • Reported component ID

    5724H8800

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-12-12

  • Closed date

    2017-06-06

  • Last modified date

    2019-05-16

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBS APP SERV N

  • Fixed component ID

    5724H8800

Applicable component levels

  • R850 PSY

       UP

  • R900 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
19 October 2021