mmdelnode command

Removes one or more nodes from a GPFS cluster.

Synopsis

mmdelnode {-a | -N Node[,Node...] | NodeFile | NodeClass [--on-error-continue]}

Availability

Available on all IBM Storage Scale editions.

Description

Use the mmdelnode command to delete one or more nodes from the GPFS cluster. You may issue the mmdelnode command on any GPFS node.

A node cannot be deleted if any of the following are true:
  1. It is a primary or secondary GPFS cluster configuration server.

    The node being deleted cannot be the primary or secondary GPFS cluster configuration server unless you intend to delete the entire cluster.

    You can determine whether a node is the primary or secondary configuration server by issuing the mmlscluster command. If the node is listed as of the servers and you still want to delete it without deleting the cluster, first use the mmchcluster command to assign another node as the server.

  2. It is defined as an NSD server.

    The node being deleted cannot be defined as an NSD server for any disk unless you intend to delete the entire cluster.

    You can determine whether a node is an NSD server for one or more disks by issuing the mmlsnsd command. If the node is listed as an NSD server and you still want to delete it without deleting the cluster, first use the mmchnsd command to assign another node as an NSD server for the affected disks.

  3. If the GPFS state is unknown and the node is reachable on the network.
    You cannot delete a node if both of the following are true:
    • The node responds to a TCP/IP ping command from another node.
    • The status of the node shows unknown when you use the mmgetstate command from another node in the cluster.
    Note: You will probably be able to delete such a node if you physically power it off.
  4. If the node is configured as a performance monitoring collector. In such cases, you need to remove the node from the performance monitoring configuration by using the mmperfmon config update --collectors command before deleting the node. Deleting a collector node causes loss of all the collected perfmon stats data on the collector node.
  5. If the node is defined as a Transparent cloud tiering node. You can determine whether a node is a Transparent cloud tiering node by issuing the mmcloudgateway node list command. If the node is listed as the Transparent cloud tiering node, and you still want to delete it without deleting the cluster, first use the mmchnode command to disable the Transparent cloud tiering node role.
    • If the node is a Transparent cloud tiering node, disable Transparent cloud tiering from the node by using the mmchnode --cloud-gateway-disable command, and then uninstall the Transparent cloud tiering rpms. Doing so ensures that the mmdelnode command does not fail on a Transparent cloud tiering node.
You must follow these rules when deleting nodes:
  1. Unless all nodes in the cluster are being deleted, run the mmdelnode command from a node that will remain in the cluster.
  2. Before you can delete a node, unmount all of the GPFS file systems and stop GPFS on the node to be deleted.
  3. Exercise caution when shutting down GPFS on quorum nodes. If the number of remaining quorum nodes falls below the requirement for a quorum, you will be unable to perform file system operations. For more information, see Quorum.

Each GPFS cluster is managed independently, so there is no automatic coordination and propagation of changes between clusters like there is between the nodes within a cluster. This means that if you permanently delete nodes that are being used as contact nodes by other GPFS clusters that can mount your file systems, you should notify the administrators of those GPFS clusters so that they can update their own environments.

Results

Upon successful completion of the mmdelnode command, the specified nodes are deleted from the GPFS cluster.

Parameters

-a
Delete all nodes in the cluster.
-N {Node[,Node...] | NodeFile | NodeClass}
Specifies the set of nodes to be deleted from the cluster.

For general information on how to specify node names, see Specifying nodes as input to GPFS commands.

This command does not support a NodeClass of mount.

--on-error-continue
Enables the command to trap invalid nodes, remove those nodes from the node list, and continue to process the command. Without selecting this option, the command may exit for any reason when a node is found to be invalid.
Note: Not all error conditions relating to a node or the program function can allow the command to continue.
When this option is used, the delete node process runs in three stages and the progress is displayed on the console for each stage as it runs. The --on-error-continue option provides the following functions:
  1. Some errors are trapped and the nodes that generated the errors are removed from the list of valid nodes to process. This allows the program to continue as long as there is at least one valid node remaining in the delete node process.
  2. Presents the console output in more of a report format.
  3. Creates three stages in the delete node process. The first two stages is where some node errors can be trapped and thus the node gets removed from the process. The third stage is more critical and will error out on any significant errors.
  4. In stage one and two, if nodes failed to get removed from the cluster, two files are created at each stage to list the valid nodes and invalid nodes separately. The user gets informed of these files. This allows the user to rerun with just the valid node list from the last successful stage, after the reasons for failure are corrected.
Refer to Example 3 for the example of the delete node process with the --on-error-continue option enabled.

Exit status

0
Successful completion.
nonzero
A failure has occurred.

Security

You must have root authority to run the mmdelnode command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.

You may issue the mmdelnode command from any node that will remain in the GPFS cluster.

Examples

  1. To delete all of the nodes in the cluster, issue this command: 
    mmdelnode -a
    The system displays information similar to: 
    Verifying GPFS is stopped on all affected nodes ...
mmdelnode: Command successfully completed
mmdelnode: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
  2. To delete nodes k145n12, k145n13, and k145n14, issue this command: 
    mmdelnode -N k145n12,k145n13,k145n14
    The system displays information similar to: 
    Verifying GPFS is stopped on all affected nodes ...
mmdelnode: Command successfully completed
mmdelnode: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
  3. In the following example, the mmdelnode command is run with the --on-error-continue option. This option helps to continue the process of deleting a list of nodes from the cluster even if certain nodes in the list fail to get deleted from the cluster. 
    # mmdelnode -N notvalidhost.gpfs.net,c21m2n04.gpfs.net,c71f2u33p1.gpfs.net  --on-error-continue
mmdelnode: On Error Continue option detected

====================================================================
                          Stage 1
Running basic validation...
====================================================================
--------------------------------------------------------------------
PROCESSING NODE: 1
Tue Feb 16 01:35:15 EST 2021: mmdelnode: Processing node notvalidhost.gpfs.net
mmdelnode: Incorrect node notvalidhost.gpfs.net specified for command.
mmdelnode: No nodes were found that matched the input specification.
--------------------------------------------------------------------
PROCESSING NODE: 2
Tue Feb 16 01:35:15 EST 2021: mmdelnode: Processing node c21m2n04.gpfs.net
--------------------------------------------------------------------
PROCESSING NODE: 3
Tue Feb 16 01:35:15 EST 2021: mmdelnode: Processing node c71f2u33p1.gpfs.net
--------------------------------------------------------------------
====================================================================
                        Stage 1 Complete
Processed node count:   3
Number of valid nodes:  2
====================================================================

Stage 1 generated files:
      /tmp/mmdelnode_rpt__2021_02_16_01_35_15/valid_nodes_list
      /tmp/mmdelnode_rpt__2021_02_16_01_35_15/invalid_nodes_list

      If stage 2 fails, address any concerns, then rerun the command
      using the 'valid_nodes_list' file with the -N option to avoid
      processing invalid nodes again.
      Rerun the command with the invalid_nodes_list file with the -N
      option once you have addressed the concerns with those invalid
      nodes.

====================================================================
                          Stage 2
Running cluster config and feature validation checks...
====================================================================
--------------------------------------------------------------------
PROCESSING NODE: 1
mmdelnode: : Processing node c21m2n04.gpfs.net
mmdelnode: Node c21m2n04.gpfs.net must be disabled as a CES node before trying to remove it from the GPFS cluster.
mmdelnode: [E] Node c21m2n04.gpfs.net can not be deleted.
               It is marked for use by Transparent Cloud Tiering.
               To remove this node, first disable it with
               mmchnode --cloud-gateway-disable.

mmdelnode: The node c21m2n04.gpfs.net hit validation errors
           and was removed from the node deletion list.
--------------------------------------------------------------------
PROCESSING NODE: 2
mmdelnode: : Processing node c71f2u33p1.gpfs.net
--------------------------------------------------------------------
====================================================================
                        Stage 2 Complete
Processed node count:   2
Number of valid nodes:  1
====================================================================
Stage 2 generated files:
      /tmp/mmdelnode_rpt__2021_02_16_01_35_15/valid_nodes_list2
      /tmp/mmdelnode_rpt__2021_02_16_01_35_15/invalid_nodes_list2

      If stage 3 fails, address any concerns, then rerun the command
      using the 'valid_nodes_list2' file with the -N option to avoid
      processing invalid nodes again from Stage 2.
      Rerun the command with the invalid_nodes_list2 file with the -N
      option once you have addressed the concerns with those invalid
      nodes from Stage 2.


====================================================================
                          Stage 3
Running additional checks and committing node deletion updates.
This stage has critical checks.  It does not generate valid/invalid
node reports and will exit on any serious errors.
====================================================================
Verifying GPFS is stopped on all affected nodes ...
mmdelnode: Removing GPFS system files on all deleted nodes ...
mmdelnode: Command successfully completed
====================================================================
                       Stage 3 Complete
====================================================================
mmdelnode: Propagating the cluster configuration data to all affected nodes.

See also

Location

/usr/lpp/mmfs/bin