Upgrading the nodes of a Linux cluster that is hosting IBM Concert Operate

Use this procedure if you need to complete maintenance tasks on the Linux nodes that host your IBM Concert Operate deployment.

Overview

The Linux nodes that host IBM Concert Operate inevitably require periodic maintenance for operating system upgrades, security updates, and restarts. IBM Concert Operate is highly available, so you can use this procedure to take the Linux nodes offline one at a time with minimal or no disruption to IBM Concert Operate.

Alternatively, if you want to take down all the Linux nodes simultaneously, then consider first shutting down IBM Concert Operate with the procedure in Restarting the environment.

Warning: Always follow one of these procedures if you need to shut down or restart multiple Linux nodes simultaneously. Failure to do so might cause unrecoverable errors.

Node maintenance procedure

Use the following procedure for each node that you need to complete maintenance on.

1. Verify application health

Run the following command to verify the status of IBM Concert Operate:
```
aiopsctl status
```
All components must be listed as Ready. Do not proceed if any components are not listed as Ready.
Run the following command to verify that all the IBM Concert Operate pods are healthy:
```
kubectl get po -A | egrep -v ' Completed | ([0-9]+)/\1 +Running '
```
The command should not return any pods. Based on the command’s outcome, take one of the following actions:
- No pods are returned: Proceed to step 2. Drain the node.
- A pod is listed with a STATUS of Error: Run the following command to check whether the ownerReference is a Job resource:
```
kubectl describe pod <error_pod>
```
  If the output of the command shows that ownerReference is a Job resource, then this is indicative of a past problem and the pod can be safely deleted with the following command:
```
kubectl delete pod <error_pod>
```
- Other pods are listed: Contact IBM Support to investigate these pods and resolve any problems before proceeding.

2. Drain the node

Export the name of the Linux node that you want to complete a maintenance task on.
```
export MAINTENANCE_NODE=<node_name>
```
Where <node_name> is the name of the Linux node that you want to complete a maintenance task on.
Run the following command to mark the Linux node as unschedulable and to safely evict its running pods:
```
kubectl drain ${MAINTENANCE_NODE} --delete-emptydir-data --ignore-daemonsets
```
Wait for the drain command to complete. Based on the command's outcome, take one of the following actions:
- Command succeeds with "node/${MAINTENANCE_NODE} drained": Proceed to step 3. Verify drain completion.
- Pod disruption budget error: An error message is displayed similar to the following example:
```
error when evicting pods/"iaf-system-kafka-1" -n "my_namespace" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
```
  Run the following command, replacing <pod_name> with the pod name from the error message (for example, iaf-system-kafka-1):
```
kubectl delete pod <pod_name>
```
  The drain should complete successfully after the pod is deleted.
- Other errors: If the drain fails for any other reason, or if it still does not complete after manually deleting the pod, contact IBM Support.

3. Verify drain completion

When the drain is complete, run the following command to verify that only the load balancer pod for the Gateway service still has a status of Running:

oc get po -o wide -A | grep ${MAINTENANCE_NODE}

Expected output:

kube-system   svclb-envoy-aiops-aiops-gateway-4f91d3be-f9692a90-24d76   1/1   Running   0   19h   10.42.0.9   node-1.mycompany.com   <none>   <none>

Most pods are moved to other nodes. You might see some pods with a status of Pending if they use local disk storage and cannot be moved.

4. Complete maintenance

Complete your maintenance tasks on the Linux node.

When your maintenance task is complete, run the following command to verify that the Linux node is Ready:

kubectl get no | grep ${MAINTENANCE_NODE}

Example output for a Linux node that is Ready:

kubectl get no | grep node-1.mycompany.com
node-1.mycompany.com   Ready,SchedulingDisabled   control-plane,etcd   31h   v1.34.4+k3s1

5. Uncordon the node

Run the following command to uncordon the Linux node so that it is schedulable again:

kubectl uncordon ${MAINTENANCE_NODE}

Example output:

kubectl uncordon node-1.mycompany.com
node/node-1.mycompany.com uncordoned

6. Verify that pending pods are running

If Pending pods remained in step 3, run the following command to verify that they are now successfully Running:

kubectl get po -o wide -A | grep ${MAINTENANCE_NODE}

Example output:

kubectl get po -o wide -A | grep node-1.mycompany.com
aiops         aiops-ir-analytics-postgres-2                                   1/1   Running   0   29m   10.42.2.118   node-1.mycompany.com   <none>   <none>
aiops         zen-metastore-1                                                 1/1   Running   0   32m   10.42.2.117   node-1.mycompany.com   <none>   <none>
kube-system   svclb-envoy-aiops-aiops-gateway-4f91d3be-f9692a90-24d76         1/1   Running   0   19h   10.42.0.9     node-1.mycompany.com   <none>   <none>

7. Repeat for remaining nodes

Repeat all the steps for the remaining Linux nodes in the cluster that you want to complete maintenance tasks on.

Warning: Do not skip the repetition of step 1. Verify application health. Failure to do so can cause loss of function and potentially unrecoverable errors.