PI73529: High CPU in ODR code caching

Fixes are available

APAR status

Closed as program error.

Error description

This apar is opened for cpu being driven by the ODR
processing an unusually high number of treeRequest messages,
building  the treeResponse messages in reply can be quite
costly.

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  All users of IBM WebSphere Application      *
*                  Server WAS Base edition- Virtual            *
*                  Enterprise/IM Component                     *
****************************************************************
* PROBLEM DESCRIPTION: On Demand Router (ODR) processes are    *
*                      consuming an abnormally high amount of  *
*                      CPU time.                               *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
On Demand Router (ODR) processes are consuming an abnormally
high amount of CPU time and performance analysis leads to a
thread with the following stack:
...
java/util/zip/DeflaterOutputStream.deflate(DeflaterOutputStream.
java:263(Compiled Code))
java/util/zip/DeflaterOutputStream.write(DeflaterOutputStream.ja
va:223(Compiled Code))
java/util/zip/GZIPOutputStream.write(GZIPOutputStream.java:158(C
ompiled Code))
(entered lock:
java/util/zip/GZIPOutputStream@0x000000002970C100, entry count:
1)
java/io/PrintStream.write(PrintStream.java:505(Compiled Code))
(entered lock: java/io/PrintStream@0x000000002970C3B8, entry
count: 3)
sun/nio/cs/StreamEncoder.writeBytes(StreamEncoder.java:234(Compi
led Code))
sun/nio/cs/StreamEncoder.implFlushBuffer(StreamEncoder.java:304(
Compiled Code))
sun/nio/cs/StreamEncoder.flushBuffer(StreamEncoder.java:117(Comp
iled Code))
(entered lock: java/io/PrintStream@0x000000002970C3B8, entry
count: 2)
java/io/OutputStreamWriter.flushBuffer(OutputStreamWriter.java:2
04(Compiled Code))
java/io/PrintStream.write(PrintStream.java:552(Compiled Code))
(entered lock: java/io/PrintStream@0x000000002970C3B8, entry
count: 1)
java/io/PrintStream.print(PrintStream.java:694(Compiled Code))
com/ibm/ws/odc/nd/ODCNodeImpl.printPropertyValue(ODCNodeImpl.jav
a:1086(Compiled Code))
com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1047(Compil
ed Code))
com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
ed Code))
com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
ed Code))
com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
ed Code))
com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1069(Compil
ed Code))
com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1008(Compil
ed Code))
com/ibm/ws/odc/nd/ODCNodeImpl.print(ODCNodeImpl.java:1000(Compil
ed Code))
com/ibm/ws/odc/nd/ODCTreeImpl.getCompressedXml(ODCTreeImpl.java:
671(Compiled Code))
(entered lock: com/ibm/ws/odc/nd/ODCTreeImpl@0x0000000000F6EDE0,
entry count: 2)
com/ibm/ws/wsgroup/nd/odc/ODCGroup.sendGetTreeResponse(ODCGroup.
java:1190(Compiled Code))
(entered lock: com/ibm/ws/odc/nd/ODCTreeImpl@0x0000000000F6EDE0,
entrycount: 1)
com/ibm/ws/wsgroup/nd/odc/ODCGroup.handleGetTreeRequest(ODCGroup
.java:1181(Compiled Code))
com/ibm/ws/wsgroup/nd/odc/ODCGroup.consumerOnMessage(ODCGroup.ja
va:533(Compiled Code))
com/ibm/ws/wsgroup/nd/odc/ODCGroup.processMessage(ODCGroup.java:
710(Compiled Code))
com/ibm/ws/wsgroup/nd/odc/ODCGroup.onMessage(ODCGroup.java:667(C
ompiled Code))
(entered lock: java/util/LinkedList@0x00000000015BDA50, entry
count: 1)
com/ibm/ws/wsgroup/WsGroup.deliverMessage(WsGroup.java:219(Compi
led Code))
...

Problem conclusion

The CPU consumption is being driven by the On Demand Router
(ODR) processes receiving a high number of getTreeRequests from
other processes. Building the On Demand Configuration (ODC) tree
response message to reply to the influx of getTreeRequests
messages is quite costly in a larger environments. Currently ODC
does cache the last tree response reply message so that is can
be resent without incurring the cost associated with building
the message. However the cache is aggressively cleared to ensure
an ODC tree response messages are not sent out with stale data.
The ODC tree response caching routine has been altered such that
the cache is only cleared when absolutely necessary to better
cope with a spike in received tree request messages.

The fix for this APAR is currently targeted for inclusion  in
fix packs 8.5.5.12 and 9.0.0.4.  Please refer to the Recommended
Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PI73529
Reported component name
WEBSPHERE APP S
Reported component ID
5724J0800
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-12-09
Closed date
2017-03-08
Last modified date
2017-03-08

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WEBSPHERE APP S
Fixed component ID
5724J0800

Applicable component levels

R850 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
04 May 2022

Tips

PI73529: High CPU in ODR code caching

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R850 PSY

Document Information

Share your feedback

Need support?