IBM Support

PI73529: High CPU in ODR code caching

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • This apar is opened for cpu being driven by the ODR
    processing an unusually high number of treeRequest messages,
    building  the treeResponse messages in reply can be quite
    costly.
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server WAS Base edition- Virtual            *
    *                  Enterprise/IM Component                     *
    ****************************************************************
    * PROBLEM DESCRIPTION: On Demand Router (ODR) processes are    *
    *                      consuming an abnormally high amount of  *
    *                      CPU time.                               *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    On Demand Router (ODR) processes are consuming an abnormally
    high amount of CPU time and performance analysis leads to a
    thread with the following stack:
    ...
    java/util/zip/DeflaterOutputStream.deflate(DeflaterOutputStream.
    java:263(Compiled Code))
    java/util/zip/DeflaterOutputStream.write(DeflaterOutputStream.ja
    va:223(Compiled Code))
    java/util/zip/GZIPOutputStream.write(GZIPOutputStream.java:158(C
    ompiled Code))
    (entered lock:
    java/util/zip/GZIPOutputStream@0x000000002970C100, entry count:
    1)
    java/io/PrintStream.write(PrintStream.java:505(Compiled Code))
    (entered lock: java/io/PrintStream@0x000000002970C3B8, entry
    count: 3)
    sun/nio/cs/StreamEncoder.writeBytes(StreamEncoder.java:234(Compi
    led Code))
    sun/nio/cs/StreamEncoder.implFlushBuffer(StreamEncoder.java:304(
    Compiled Code))
    sun/nio/cs/StreamEncoder.flushBuffer(StreamEncoder.java:117(Comp
    iled Code))
    (entered lock: java/io/PrintStream@0x000000002970C3B8, entry
    count: 2)
    java/io/OutputStreamWriter.flushBuffer(OutputStreamWriter.java:2
    04(Compiled Code))
    java/io/PrintStream.write(PrintStream.java:552(Compiled Code))
    (entered lock: java/io/PrintStream@0x000000002970C3B8, entry
    count: 1)
    java/io/PrintStream.print(PrintStream.java:694(Compiled Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.printPropertyValue(ODCNodeImpl.jav
    a:1086(Compiled Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1047(Compil
    ed Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
    ed Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
    ed Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
    ed Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
    ed Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1008(Compil
    ed Code))
    com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1000(Compil
    ed Code))
    com/ibm/ws/odc/nd/ODCTreeImpl.getCompressedXml(ODCTreeImpl.java:
    671(Compiled Code))
    (entered lock: com/ibm/ws/odc/nd/ODCTreeImpl@0x0000000000F6EDE0,
    entry count: 2)
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.sendGetTreeResponse(ODCGroup.
    java:1190(Compiled Code))
    (entered lock: com/ibm/ws/odc/nd/ODCTreeImpl@0x0000000000F6EDE0,
    entrycount: 1)
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.handleGetTreeRequest(ODCGroup
    .java:1181(Compiled Code))
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.consumerOnMessage(ODCGroup.ja
    va:533(Compiled Code))
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.processMessage(ODCGroup.java:
    710(Compiled Code))
    com/ibm/ws/wsgroup/nd/odc/ODCGroup.onMessage(ODCGroup.java:667(C
    ompiled Code))
    (entered lock: java/util/LinkedList@0x00000000015BDA50, entry
    count: 1)
    com/ibm/ws/wsgroup/WsGroup.deliverMessage(WsGroup.java:219(Compi
    led Code))
    ...
    

Problem conclusion

  • The CPU consumption is being driven by the On Demand Router
    (ODR) processes receiving a high number of getTreeRequests from
    other processes. Building the On Demand Configuration (ODC) tree
    response message to reply to the influx of getTreeRequests
    messages is quite costly in a larger environments. Currently ODC
    does cache the last tree response reply message so that is can
    be resent without incurring the cost associated with building
    the message. However the cache is aggressively cleared to ensure
    an ODC tree response messages are not sent out with stale data.
    The ODC tree response caching routine has been altered such that
    the cache is only cleared when absolutely necessary to better
    cope with a spike in received tree request messages.
    
    The fix for this APAR is currently targeted for inclusion  in
    fix packs 8.5.5.12 and 9.0.0.4.  Please refer to the Recommended
    Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PI73529

  • Reported component name

    WEBSPHERE APP S

  • Reported component ID

    5724J0800

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-12-09

  • Closed date

    2017-03-08

  • Last modified date

    2017-03-08

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE APP S

  • Fixed component ID

    5724J0800

Applicable component levels

  • R850 PSY

       UP

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSCVS22","label":"General"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Edition":"","Line of Business":{"code":"","label":""}}]

Document Information

Modified date:
08 March 2017