Removing a healthy node (IBM Concert Operate on Linux)

If you no longer want a worker node or a control plane node in your Linux cluster to be used by your IBM Concert Operate deployment, then use the following procedure to remove it.

The following procedures are also available:

• if you want to uninstall IBM Concert Operate completely, follow the instructions in Uninstalling IBM Concert Operate on Linux.
• if you want to delete and recover from a single unresponsive worker node in your IBM Concert Operate deployment, follow the instructions in Removing a failed node.

• You must check that the removal of a node will not cause there to be insufficient resources for IBM Concert Operate. For more information, see Planning an installation. You must always have three control plane nodes.
• If you have only one control plane node, do not remove it as this will cause IBM Concert Operate to stop functioning.

1. Before you remove a node, ensure that you first add a new node. For more information, see Adding a control plane node or Adding a worker node.

2. Set environment variables.

   1. Run the following command to source the shell script aiops_var.sh that you created when you installed IBM Concert Operate on your Linux cluster.

      . ./aiops_var.sh
      

      For more information about aiops_var.sh, see Create environment variables in Online installation of IBM Concert Operate on Linux or Offline installation of IBM Concert Operate on Linux.

   2. Set an additional temporary environment variable for the node that you are removing.

      export NODE_TO_REMOVE=<node_to_remove>
      

      Where <node_to_remove> is the FQDN or IP address of the node that you want to remove. For example, node1.example.com.

3. Check if it is safe to delete the node.

   Run the node deletion command with the dry-run flag from a control plane node, to check whether it is safe to remove the node from the cluster.

   ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster delete-node --dry-run ${NODE_TO_REMOVE}
   

   You are prompted with a warning that data loss might occur, and a query whether to continue. To avoid data loss, examine the Data stored on node table to check that the replication of your data stores is sufficient such that data loss does not occur if you continue to delete the node. Check that the last column Replicas remaining after deletion does not have 0 for any row, and that an extra warning message to contact IBM Support is not displayed. If Replicas remaining after deletion is 0 for any row, do not proceed and contact IBM Support.

   Example output for a cluster where it is safe to continue and remove the node:

   $ aiopsctl cluster delete-node --dry-run agent1.acme.com                 
   [WARN] Removing node agent1.acme.com. This may cause data loss, read the information below carefully before continuing.
   
   Data stored on node:
   
   Name                          Total replicas  Replicas on node  Replicas remaining after deletion
   Kafka                         3               0                 3
   Identity Management Postgres  2               0                 2
   Zen MinIO                     3               1                 2
   CouchDB                       3               0                 3
   Opensearch                    3               0                 3
   Zen Postgres                  2               0                 2
   AIOps Postgres                3               1                 2
   AIOps MinIO                   5               0                 5
   Redis                         3               1                 2
   Cassandra                     3               0                 3
   
   Are you sure you wish to continue? (y/n): n
   

   Type n at the prompt to exit without deleting the node.

4. Mark the node as down.

   Run one of the following commands on the node that you want to remove from your IBM Concert Operate deployment, depending on whether it is a worker node or a control plane node.

   Worker node:

   ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} ssh ${TARGET_USER}@${NODE_TO_REMOVE} aiopsctl cluster node down --role worker
   

   Control plane node:

   ssh ${TARGET_USER}@${NODE_TO_REMOVE} aiopsctl cluster node down --role control-plane
   

   If you have only one control plane node, then you cannot delete that node, and aiopsctl prevents you from doing so.

5. Wait for 5 minutes for the application services to detect the node's removal.

6. Delete the node.

   From a control plane node that is not the target node, run the delete-node command.

   ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster delete-node ${NODE_TO_REMOVE}
   

   You are prompted with a warning that data loss might occur, and a query whether to continue. To avoid data loss, examine the Data stored on node table to check that the replication of your data stores is sufficient such that data loss does not occur if you continue to delete the node. Check that the last column Replicas remaining after deletion does not have 0 for any row, and that an extra warning message to contact IBM Support is not displayed. If Replicas remaining after deletion is 0 for any row, do not proceed and contact IBM Support. Otherwise, type y at the prompt if you want to continue.

   Example output for a cluster where it is safe to continue and remove the node:

   $ aiopsctl cluster delete-node agent1.acme.com                 
   [WARN] Removing node agent1.acme.com. This may cause data loss, read the information below carefully before continuing.
   
   Data stored on node:
   
   Name                          Total replicas  Replicas on node  Replicas remaining after deletion
   Kafka                         3               0                 3
   Identity Management Postgres  2               0                 2
   Zen MinIO                     3               1                 2
   CouchDB                       3               0                 3
   Opensearch                    3               0                 3
   Zen Postgres                  2               0                 2
   AIOps Postgres                3               1                 2
   AIOps MinIO                   5               0                 5
   Redis                         3               1                 2
   Cassandra                     3               0                 3
   
   Are you sure you wish to continue? (y/n): y
   
   o- [24 Jul 24 08:44 CDT] Deleting node agent1.acme.com...
   o- [24 Jul 24 08:44 CDT] Recovering datastores...
   
   # Node is removed
   

   If you have only one control plane node, then you cannot delete that node, and aiopsctl prevents you from doing so.

7. Update the environment variables file aiops_var.sh

   When your node is successfully removed, remove its entry from aiops_var.sh. This is to maintain the accuracy of the environment variables file.

   For more information about aiops_var.sh, see Create environment variables in Online installation of IBM Concert Operate on Linux or Offline installation of IBM Concert Operate on Linux.

   If you are removing a worker node:

   Remove the entry for the worker node that you removed from the WORKER_NODES array.

   If you are removing a control plane node:

   • If the control plane node that you removed is in the ADDITIONAL_CONTROL_PLANES array, then remove the entry for the control plane node from the ADDITIONAL_CONTROL_PLANES array.
   • If the control plane node that you removed is the CONTROL_PLANE_NODE, then replace the value of CONTROL_PLANE_NODE with the value of the new main control plane node that you created in step 1, if you have not already done so.

8. If you removed a control plane node, then update your load balancer's configuration.