IBM Support

PH34576: DURING SHUTDOWN OF SERVER, CONTROL REGION EXPERIENCES HANG IN COM/IBM/SON/MESH/CFWTCPIMPL.COMPLETE

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • Control region experiences hang during shutdown of server.
    The server has Intelligent Management enabled.
    
    Formatting out the threads from a console dump will show many
    threads with stacktrace:
    
    com/ibm/son/mesh/CfwTCPImpl.complete
    com/ibm/ws/ssl/channel/impl/SSLReadServiceContext$SSLReadComplet
    edCallback.complete
    com/ibm/ws/tcp/channel/impl/ZAioTCPReadRequestContextImpl.readCo
    mpleted
    com/ibm/ws/tcp/channel/impl/ZAioTCPConnLink.readCompleted
    com/ibm/ws/tcp/channel/impl/ZAioTCPChannel.readCompleted
    com/ibm/ws/tcp/channel/impl/ZAioTCPChannelCPPUtilities.readCompl
    eted
    
    Java deadlock:
    thread: com.ibm.son.mesh.Peer-tcp-port-11091 id: 0x45c52000
    (owns monitor for com/ibm/son/mesh/Peer object : 0x81c0fe78)
    waiting for =>
    
    thread: WebSphere t=009af250 id: 0x45c6c000 (owns monitor for
    java/lang/Object object : 0x82b2b038) waiting for =>
    
    thread: com.ibm.son.mesh.Peer-tcp-port-11091 id: 0x45c52000
    (owns monitor for com/ibm/son/mesh/Peer object : 0x81c0fe78)
    

Local fix

Problem summary

  • ****************************************************************
    * USERS AFFECTED:  All users of IBM WebSphere Application      *
    *                  Server V9.0 for z/OS                        *
    ****************************************************************
    * PROBLEM DESCRIPTION: Controller region with Intelligent      *
    *                      Management enabled hangs during         *
    *                      server stop.                            *
    ****************************************************************
    * RECOMMENDATION:                                              *
    ****************************************************************
    Controller region with Intelligent Management enabled hangs
    during server stop. The stop does not complete because two
    threads are in a deadlock. The first thread has the
    ZAioTCPConnLink read lock and wants the IM Mesh lock.
    The second thread has the IM Mesh lock and wants the
    ZAioTCPConnLink read lock.
    First thread
    com/ibm/son/mesh/CfwTCPImpl.complete
    com/ibm/ws/ssl/channel/impl/
    SSLReadServiceContext$SSLReadCompletedCallback.complete
    com/ibm/ws/tcp/channel/impl/
    ZAioTCPReadRequestContextImpl.readCompleted
    com/ibm/ws/tcp/channel/impl/ZAioTCPConnLink.readCompleted
    com/ibm/ws/tcp/channel/impl/ZAioTCPChannel.readCompleted
    com/ibm/ws/tcp/channel/impl/
    ZAioTCPChannelCPPUtilities.readCompleted
    Second thread
    com/ibm/ws/tcp/channel/impl/ZAioTCPConnLink.destroyCommon
    com/ibm/ws/tcp/channel/impl/ZAioTCPConnLink.destroy
    com/ibm/wsspi/channel/base/OutboundConnectorLink.close
    com/ibm/ws/ssl/channel/impl/SSLConnectionLink.close
    com/ibm/wsspi/channel/base/InboundApplicationLink.close
    com/ibm/son/channelfw/SonTCPLink.close
    com/ibm/son/mesh/CfwTCPImpl.closeLinks
    com/ibm/son/mesh/CfwTCPImpl.hardClose
    com/ibm/son/mesh/Neighbor.hardClose
    com/ibm/son/mesh/MemberMgr.handleIOExceptionGlobalCleanup
    com/ibm/son/mesh/MemberMgr.handleIOException
    com/ibm/son/mesh/RandNeighborMaintenanceJob.dropNbrs
    com/ibm/son/mesh/RandNeighborMaintenanceJob.dropNbrs()
    com/ibm/son/mesh/RandNeighborMaintenanceJob.run
    com/ibm/son/mesh/JobQueue.run()
    com/ibm/son/mesh/Peer.run()
    

Problem conclusion

  • Updated the code to avoid the deadlock when custom property
    com.ibm.ws390.tcpchannel.skipConLinkReadLockOnReadComplete is
    set to true in a controller region.
    
    The following shows how to set the custom property for
    server1.
    servers > Server Types > WebSphere application servers >
    server1 > Server Infrastructure > Java and Process Management >
    Process definition > Control > Java Virtual Machine >
    Custom properties
    
    Do not use this custom property without the direction of
    the WebSphere Support team.
    
    The fix for this APAR is targeted for inclusion in fix pack
    9.0.5.8. For more information, see 'Recommended Updates for
    WebSphere Application Server':
    https://www.ibm.com/support/pages/node/715553
    

Temporary fix

Comments

APAR Information

  • APAR number

    PH34576

  • Reported component name

    WEBSPHERE FOR Z

  • Reported component ID

    5655I3500

  • Reported release

    900

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2021-02-17

  • Closed date

    2021-03-10

  • Last modified date

    2022-03-15

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WEBSPHERE FOR Z

  • Fixed component ID

    5655I3500

Applicable component levels

[{"Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"900"}]

Document Information

Modified date:
16 March 2022