Removing a node (IBM Cloud Pak for AIOps on Linux)

Learn how to remove a worker node or a control plane node from a deployment of IBM Cloud Pak® for AIOps on Linux®.

Overview

If you no longer want a worker node or a control plane node in your Linux cluster to be used by your IBM Cloud Pak for AIOps deployment, then use the following procedure to remove it.

Note: The following procedures are also available:

if you want to uninstall IBM Cloud Pak for AIOps completely, follow the instructions in Uninstalling IBM Cloud Pak for AIOps on Linux.
if you want to delete and recover from a single unresponsive worker node in your IBM Cloud Pak for AIOps deployment, follow the instructions in Recovering from node failure.

Procedure

Important:

You must check that the removal of a node will not cause there to be insufficient resources for IBM Cloud Pak for AIOps. For more information, see Planning an installation. You must always have three control plane nodes.
If you have only one control plane node, do not remove it as this will cause IBM Cloud Pak for AIOps to stop functioning.

Before you remove a node, ensure that you first add a new node. For more information, see Adding a control plane node or Adding a worker node.
Set environment variables.
1. Run the following command to source the shell script aiops_var.sh that you created when you installed IBM Cloud Pak for AIOps on your Linux cluster.
```
. ./aiops_var.sh
```
  For more information about aiops_var.sh, see Create environment variables in Online installation of IBM Cloud Pak for AIOps on Linux or Offline installation of IBM Cloud Pak for AIOps on Linux.
2. Set an additional temporary environment variable for the node that you are removing.
```
export NODE_TO_REMOVE=<node_to_remove>
```
  Where <node_to_remove> is the FQDN or IP address of the node that you want to remove. For example, node1.example.com.

Run the following command to uninstall k3s on the node that you want to remove.

ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} aiopsctl cluster delete-node ${NODE_TO_REMOVE}

You are prompted with a warning that data loss might occur, and a query whether to continue. To avoid data loss, examine the Data stored on node table to check that the replication of your data stores is sufficient such that data loss does not occur if you continue to delete the node. Check that the last column Replicas remaining after deletion does not have 0 for any row, and that an extra warning message to contact IBM Support is not displayed. Type y at the prompt if you want to continue.

Example output for a cluster where it is safe to continue and remove the node:

$ aiopsctl cluster delete-node agent1.acme.com                 
[WARN] Removing node agent1.acme.com. This may cause data loss, read the information below carefully before continuing.

Data stored on node:

Name                          Total replicas  Replicas on node  Replicas remaining after deletion
Kafka                         3               0                 3
Identity Management Postgres  2               0                 2
Zen MinIO                     3               1                 2
CouchDB                       3               0                 3
Elasticsearch                 3               0                 3
Zen Postgres                  2               0                 2
AIOps Postgres                3               1                 2
AIOps MinIO                   5               0                 5
Redis                         3               1                 2
Cassandra                     3               0                 3

Are you sure you wish to continue? (y/n): y

o- [24 Jul 24 08:44 CDT] Deleting node agent1.acme.com...
o- [24 Jul 24 08:44 CDT] Recovering datastores...

# Node is removed

Note: If you have only one control plane node, then you cannot delete that node, and aiopsctl prevents you from doing so.

Run one of the following commands to remove the node from your IBM Cloud Pak for AIOps deployment, depending on whether it is a worker node or a control plane node.

Worker node:

ssh ${TARGET_USER}@${CONTROL_PLANE_NODE} ssh ${TARGET_USER}@${NODE_TO_REMOVE} aiopsctl cluster node down --role worker --app-storage ${APP_STORAGE_PATH}

Control plane node:

ssh ${TARGET_USER}@${NODE_TO_REMOVE} aiopsctl cluster node down --role control-plane

Update the environment variables file aiops_var.sh

When your node is successfully removed, remove its entry from aiops_var.sh. This is to maintain the accuracy of the environment variables file.

For more information about aiops_var.sh, see Create environment variables in Online installation of IBM Cloud Pak for AIOps on Linux or Offline installation of IBM Cloud Pak for AIOps on Linux.

If you are removing a worker node:

Remove the entry for the worker node that you removed from the WORKER_NODES array.

If you are removing a control plane node:
- If the control plane node that you removed is in the ADDITIONAL_CONTROL_PLANES array, then remove the entry for the control plane node from the ADDITIONAL_CONTROL_PLANES array.
- If the control plane node that you removed is the CONTROL_PLANE_NODE, then replace the value of CONTROL_PLANE_NODE with the value of the new main control plane node that you created in step 1, if you have not already done so.
If you removed a control plane node, then update your load balancer's configuration.