Fixes are available
9.0.5.1: WebSphere Application Server traditional Version 9.0.5 Fix Pack 1
9.0.5.2: WebSphere Application Server traditional Version 9.0.5 Fix Pack 2
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
9.0.5.3: WebSphere Application Server traditional Version 9.0.5 Fix Pack 3
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
APAR status
Closed as program error.
Error description
Log shows: ODCF8041I: Detected process ..... stopped. ODCF8040I: Detected process ..... started. CWOBB1009W: Process ..... rejoined. The rapid leave join is caused by unhandled exceptions that propagate all the way back up to the main P2P job queue loop causing it to exit, that job queue loop rooted at com.ibm.son.mesh.Peer.run()is suppose to run for the entire lifetime of the process. In this case the unhandled exception was an ´java.lang.IllegalArgumentException´ encountered during a TCP read Additional symptoms: Data fills up the receive queue for up on the UDP port for endpoint OVERLAY_UDP_LISTENER_ADDRESS
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server WAS ND edition * **************************************************************** * PROBLEM DESCRIPTION: SystemOut log files contain rapidly * * repeating * * ODCF8041I/ODCF8040I/CWOBB1009W * * messages * * accompanied by slow memory growth. * **************************************************************** * RECOMMENDATION: * **************************************************************** This issue can be identified by rapidly repeating ODCF8041I/ODCF8040I (Detected process {0} stopped/started) and CWOBB1009W (Process {0} rejoined messages) in the SystemOut.log files accompanied by slow memory growth in maps owned by the com.ibm.son.mesh.Peer object. This issue is caused by exceptions, most predominantly ´java.lang.IllegalArgumentException(s)´ encountered during a TCP read, that were not properly caught and handled which were allowed to propagate all the way back up to the main P2P/SON, the Intelligent Management communication layer, Peer job queue loop causing it to exit prematurely.
Problem conclusion
The Intelligent Management communication layer (P2P/SON) has been hardened to catch and properly handle all known exceptions as well as catching any potentially unknown exception types, outputting an FFDC in that case, to ensure the main P2P/SON Peer job queue loop can never exit prematurely while the JVM is still running. The fix for this APAR is currently targeted for inclusion in fix packs 8.5.5.16 and 9.0.5.1. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PH12773
Reported component name
WEBSPHERE FOR Z
Reported component ID
5655I3500
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2019-05-31
Closed date
2022-03-29
Last modified date
2022-03-29
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBSPHERE FOR Z
Fixed component ID
5655I3500
Applicable component levels
R850 PSY
UP
Document Information
Modified date:
27 April 2022