IBM Support

PH12773: ADD/REMOVE NEIGHBOR LOOP WITH MESSAGES ODCF8041I, ODCF8040I, CWOBB1009W

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • Log shows:
    ODCF8041I: Detected process ..... stopped.
    ODCF8040I: Detected process ..... started.
    CWOBB1009W: Process ..... rejoined.
    
    The rapid leave join is caused by unhandled exceptions that
    propagate all the way back up to the main P2P job queue loop
    causing it to exit, that job queue loop rooted at
    com.ibm.son.mesh.Peer.run()is suppose to run for the entire
    lifetime of the process.
    
    In this case the unhandled exception was an
    ´java.lang.IllegalArgumentException´ encountered during a
    TCP
    read
    
    Additional symptoms:
    Data fills up the receive queue for up on the UDP port for
    endpoint OVERLAY_UDP_LISTENER_ADDRESS
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server WAS ND edition                       *
    ****************************************************************
    * PROBLEM DESCRIPTION: SystemOut log files contain rapidly     *
    *                      repeating                               *
    *                      ODCF8041I/ODCF8040I/CWOBB1009W messages *
    *                      accompanied by slow memory growth.      *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    This issue can be identified by rapidly repeating
    ODCF8041I/ODCF8040I (Detected process {0} stopped/started) and
    CWOBB1009W (Process {0} rejoined messages) in the SystemOut.log
    files accompanied by slow memory growth in maps owned by the
    com.ibm.son.mesh.Peer object. This issue is caused by
    exceptions, most predominantly
    ´java.lang.IllegalArgumentException(s)´ encountered during a TCP
    read, that were not properly caught and handled which were
    allowed to propagate all the way back up to the main P2P/SON,
    the Intelligent Management communication layer, Peer job queue
    loop causing it to exit prematurely.
    

Problem conclusion

  • The Intelligent Management communication layer (P2P/SON) has
    been hardened to catch and properly handle all known exceptions
    as well as catching any potentially unknown exception types,
    outputting an FFDC in that case, to ensure the main P2P/SON Peer
    job queue loop can never exit prematurely while the JVM is still
    running.
    
    The fix for this APAR is currently targeted for inclusion in fix
    packs 8.5.5.16 and 9.0.5.1.  Please refer to the Recommended
    Updates page for delivery information:
    http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
    

Temporary fix

Comments

APAR Information

  • APAR number

    PH12773

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    850

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2019-05-31

  • Closed date

    2019-08-26

  • Last modified date

    2019-08-26

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

  • R850 PSY

       UP

[{"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
17 October 2021