Maintenance procedure for IBM Spectrum Scale node using Ambari maintenance mode

If you need to do maintenance (servicing) of the IBM Spectrum® Scale nodes, you can set the Ambari maintenance mode for the GPFS_NODE and the corresponding NameNode and DataNode components residing on the same node.

Note: The Ambari maintenance mode is not supported in the FPO (local) filesystems.

General procedure for maintenance

The following procedure is for GPFS_NODE that is not colocated with the Ambari server:

  1. In an HA environment, if the active NameNode 1 requires to be serviced, ensure that the failover of NameNode 1 has completed and that NameNode 1 is now on standby. NameNode 2 should now become the active NameNode.
  2. To service only a subset of the IBM Spectrum Scale nodes, ensure that there are enough IBM Spectrum Scale quorum nodes for IBM Spectrum Scale to stay healthy.
    Run the following command and check the Designation field on the quorum node:
    # mmlscluster

    In order to avoid losing quorum, if the quorum nodes are not enough, move the quorum designation to the other IBM Spectrum Scale nodes that are not being serviced.

    For information on checking and setting the quorum nodes, see the Which nodes in my cluster are quorum nodes? topic in IBM Storage Scale: Problem Determination Guide.

  3. In Ambari, to set the component-level maintenance mode for each node, perform the following:
    • Stop the GPFS_NODE and the NameNode or DataNode components on the node that is to be serviced.
    • Set the GPFS_NODE and the NameNode and DataNode components to the Ambari maintenance mode on the node to be serviced.
  4. Service the IBM Spectrum Scale nodes that were set to the Ambari maintenance mode.
  5. From Ambari GUI, disable the Ambari maintenance mode for the GPFS_NODE and the NameNode or DataNode components on the node that was serviced. Perform this for each node.
  6. Start the GPFS_NODE and the NameNode or DataNode components on each node or perform an Ambari START ALL.
  7. If the quorum designation was moved in step 2, you can move it back to the original designated IBM Spectrum Scale node.

Maintenance procedure for the Ambari server node with colocated GPFS_NODE and GPFS_MASTER

The GPFS_MASTER and GPFS_NODE on the Ambari server cannot honor the Ambari maintenance mode. The GPFS_MASTER is also responsible for executing the IBM Spectrum Scale commands in the cluster. Therefore, this node requires to be serviced on its own.

  1. In order to avoid quorum loss, if the GPFS_MASTER and GPFS_NODE on the Ambari server have quorum designation, move the quorum designation to other IBM Spectrum Scale nodes that are not being serviced.

    For information on checking and setting the quorum nodes, see the Which nodes in my cluster are quorum nodes? topic in IBM Storage Scale: Problem Determination Guide.

    If there is an Active NameNode present on the GPFS_MASTER node, initiate a failover and ensure that the current NameNode is on standby.

  2. To set the component-level maintenance mode on the Ambari server, perform the following:
    • Stop the GPFS_MASTER, GPFS_NODE and NameNode or DataNode components.
    • Set the GPFS_MASTER, GPFS_NODE and NameNode or DataNode components to the maintenance mode.
      Note: After you stop the IBM Spectrum Scale service components, you cannot manage the IBM Spectrum Scale service from the Ambari server.
  3. On the Ambari server, service the IBM Spectrum Scale node.
  4. From the Ambari GUI, disable the maintenance mode for GPFS_MASTER, GPFS_NODE and NameNode or DataNode components on the Ambari server that was serviced.
  5. On the Ambari server, start the GPFS_MASTER, GPFS_NODE and the NameNode or DataNode components.
  6. If the quorum designation was moved in step 1, you can move it back to the original designated Ambari server node.