PH45943: Changing node maintenance mode state fails and hung thread warnings (WSVR0605W) are observed in SystemOut.log.

APAR status

• Closed as program error.

Error description

• Attempts to change node maintenance mode state fail and hung
  thread warnings (WSVR0605W) related to the
  SoapConnectorThreadPool with the stack below are observed in
  SystemOut.log until the soap connection timeout is exceeded.
  
  [4/13/22 11:55:07:594 CDT] 000000cc ThreadMonitor W
  WSVR0605W: Thread "SoapConnectorThreadPool : 64" (00000643)
  has been active for 653673 milliseconds and may be hung.
  There is/are 1 thread(s) in total in the server that may be
  hung.
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:189)
          at
  com.ibm.ws.xd.console.distmanagement.topology.nodegroup.NodeGrou
  pXDUtilFactory$MyCollab.lockNodeMMLock(NodeGroupXDUtilFactory.ja
  va:829)
          at
  com.ibm.ws.xd.console.distmanagement.topology.nodegroup.NodeGrou
  pXDUtilFactory$MyCollab.setBatchMaintenanceMode(NodeGroupXDUtilF
  actory.java:869)
          at
  com.ibm.ws.xd.console.distmanagement.topology.nodegroup.NodeGrou
  pXDUtilFactory$MyCollab.setMaintenanceMode(NodeGroupXDUtilFactor
  y.java:491)
  
  There are exception paths through the NodeGroupManager MBean
  setMaintenanceMode operation calls that can leave a lock held
  that will prevent subsequent setMaintenanceMode calls from
  executing correctly leading to the WSVR0605W messages and
  inability to change node maintenance mode state. One specific
  way to enter this error path is attempting to take a node out
  of maintenance mode that does not exist in the cell which
  would result in the the messages below also being seen.
  
  [4/13/22 11:44:01:071 CDT] 00000642 NodeGroupXDUt I
  NGRP2007I: unsetMaintenanceMode: An attempt is made to take
  the node NodeNameThatDoesNotExistInTheCell out of maintenance
  mode
  [4/13/22 11:44:01:074 CDT] 00000642 NodeGroupXDUt E
  NGRP0009E: Error invoking setMaintenanceMode on node
  NodeNameThatDoesNotExistInTheCell, newValue: false
  leaveProcessesRunning: true Exception:
  com.ibm.ws.console.distmanagement.topology.nodegroup.NodeGroupMe
  mberNotFoundException:null
  

Local fix

• NA
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server                                      *
  *                  WAS ND edition that use Node Maintenance    *
  *                  Mode.                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION: Changing node maintenance mode state    *
  *                      fails and hung thread warnings          *
  *                      (WSVR0605W) are observed in             *
  *                      SystemOut.log.                          *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  Attempts to change node maintenance mode state fail and hung
  thread warnings (WSVR0605W) related to the
  SoapConnectorThreadPool with the stack below are observed in
  SystemOut.log until the soap connection timeout is exceeded.
  [4/13/22 11:55:07:594 CDT] 000000cc ThreadMonitor W   WSVR0605W:
  Thread "SoapConnectorThreadPool : 64" (00000643) has been active
  for 653673 milliseconds and may be hung.  There is/are 1
  thread(s) in total in the server that may be hung.
  at java.lang.Object.wait(Native Method)
  at java.lang.Object.wait(Object.java:189)
  at
  com.ibm.ws.xd.console.distmanagement.topology.nodegroup.NodeGrou
  pXDUtilFactory$MyCollab.lockNodeMMLock(NodeGroupXDUtilFactory.ja
  va:829)
  at
  com.ibm.ws.xd.console.distmanagement.topology.nodegroup.NodeGrou
  pXDUtilFactory$MyCollab.setBatchMaintenanceMode(NodeGroupXDUtilF
  actory.java:869)
  at
  com.ibm.ws.xd.console.distmanagement.topology.nodegroup.NodeGrou
  pXDUtilFactory$MyCollab.setMaintenanceMode(NodeGroupXDUtilFactor
  y.java:491)
  There are exception paths through the NodeGroupManager MBean
  setMaintenanceMode operation calls that can leave a lock held
  that will prevent subsequent setMaintenanceMode calls from
  executing correctly leading to the WSVR0605W messages and
  inability to change node maintenance mode state. One specific
  way to enter this error path is attempting to take a node out of
  maintenance mode that does not exist in the cell which would
  result in the the messages below also being seen.
  [4/13/22 11:44:01:071 CDT] 00000642 NodeGroupXDUt I   NGRP2007I:
  unsetMaintenanceMode: An attempt is made to take the node
  NodeNameThatDoesNotExistInTheCell out of maintenance mode
  [4/13/22 11:44:01:074 CDT] 00000642 NodeGroupXDUt E   NGRP0009E:
  Error invoking setMaintenanceMode on node
  NodeNameThatDoesNotExistInTheCell, newValue: false
  leaveProcessesRunning: true Exception:
  com.ibm.ws.console.distmanagement.topology.nodegroup.NodeGroupMe
  mberNotFoundException:null
  

Problem conclusion

• The exception paths residing in the NodeGroupManager MBean
  maintenance mode operation calls were altered such that all
  locks are properly released on exit allowing subsequent calls to
  node maintenance mode operations to execute correctly.
  
  The fix for this APAR is targeted for inclusion in fix packs
  8.5.5.22 and 9.0.5.13. For more information, see 'Recommended
  Updates for WebSphere Application Server':
  https://www.ibm.com/support/pages/node/715553
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WEBS APP SERV N

• Fixed component ID

  5724H8800

Applicable component levels