Replacing an operational VMware node on installer-provisioned infrastructure

Use this information to replace an operational VMware node on an installer-provisioned infrastructure.

Procedure

Log in to the OpenShift Web Console, and click Compute > Nodes.
Identify the faulty node that you need to replace.
Take note of its Machine Name.
Mark the node as unscheduable, where <node_name> specifies the name of node that you need to replace.
oc adm cordon <node_name>
Drain the node.
oc adm drain <node_name> --force --delete-emptydir-data=true --ignore-daemonsets
Important: This activity might take at least 5 - 10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when you label the new node, and it is functional.
Click Compute > Machines and search for the required machine.
For the required machine, click Action menu > Delete Machine.
Click Delete to confirm that the machine is deleted.
A new machine is automatically created.
Wait for the new machine to start and transition into Running state.

Important: This activity might take at least 5 - 10 minutes or more.
Go to Compute > Nodes and confirm that the new node is in a Ready state.
Apply the Fusion Data Foundation label to the new node using one of the following steps:
From the user interface
1. Go to Action Menu > Edit Labels > .
2. Add cluster.ocs.openshift.io/openshift-storage, and click Save.
From the command-line interface
Apply the Fusion Data Foundation label to the new node:, where <new_node_name> specifies the name of the new node.
```
oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
```

Verify that the new node and all pods are running.

Verify that the new node is present in the output:

oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

Workloads > Pods and confirm that at least the following pods on the new node are in a Running state:
- csi-cephfsplugin-*
- csi-rbdplugin-*
Verify that all the other required Fusion Data Foundation pods are in Running state.
Verify that the new Object Storage Device (OSD) pods are running on the replacement node:
```
oc get pods -o wide -n openshift-storage| egrep -i <new_node_name> | egrep osd
```
If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.
For each of the new nodes identified in the previous step, do the following:
1. Create a debug pod and open a chroot environment for the one or more selected hosts:
```
oc debug node/<node_name>
```
```
chroot /host
```
2. Display the list of available block devices:, using the lsblk command.
  Check for the crypt keyword beside the one or more ocs-deviceset names.
If the verification steps fail, contact IBM Support.