Replacing operational nodes on IBM Z or LinuxONE infrastructure

Use this procedure to replace an operational node on IBM zSystems or LinuxONE infrastructure.

Procedure

Identify the node, and get the labels on the node that you need to replace, where <node_name> specifies the name of the node that needs to be replaced.
```
oc get nodes --show-labels | grep <node_name>
```
Identify the monitor pod (mon) (if any), and OSDs that are running in the node that you need to replace.
```
oc get pods -n openshift-storage -o wide | grep -i <node_name>
```

Scale down the deployments of the pods identified in the previous step.

oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage

oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage

oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

Mark the node as unschedulable
```
oc adm cordon <node_name>
```

Remove the pods which are in Terminating state.

oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

Drain the node.

oc adm drain <node_name> --force --delete-emptydir-data=true --ignore-daemonsets

Delete the node.
```
oc delete node <node_name>
```
Get a new IBM zSystem storage node as a replacement.
Check for certificate signing requests (CSRs) related to Fusion Data Foundation that are in Pending state.
```
oc get csr
```
Approve all required Fusion Data Foundation CSRs for the new node, where <certificate_name> specifies the name of the CSR.
```
oc adm certificate approve <certificate_name>
```
From the OpenShift Web Console, go to Compute > Nodes and confirm that the new node is in Ready state.
Apply the Fusion Data Foundation label to the new node using one of the following steps:
From the user interface
1. Go to Action Menu > Edit Labels > .
2. Add cluster.ocs.openshift.io/openshift-storage, and click Save.
From the command-line interface
Apply the Fusion Data Foundation label to the new node:, where <new_node_name> specifies the name of the new node.
```
oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
```

Add a new worker node to the localVolumeDiscovery and localVolumeSet.

Update the localVolumeDiscovery definition to include the new node, and remove the failed node.

oc edit -n local_storage_project localvolumediscovery auto-discover-devices

Example output, where server3.example.com is removed, and newnode.example.com is the new node:

[...]
   nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - server1.example.com
            - server2.example.com
            #- server3.example.com
            - newnode.example.com
[...]

Remember: Save before exiting the editor.

Determine the localVolumeSet to edit.
Replace local-storage-project with the name of your local storage project. The default project name is openshift-local-storage in Fusion Data Foundation.
```
oc get -n local-storage-project localvolumeset
```
Example output:
```
NAME          AGE
localblock   25h
```

Update the localVolumeSet definition.

oc edit -n $local_storage_project localvolumeset localblock

Example output, where server3.example.com is removed, and newnode.example.com is the new node:

[...]
   nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
            - server1.example.com
            - server2.example.com
            #- server3.example.com
            - newnode.example.com
[...]

Remember: Save before exiting the editor.

Verify that the new localblock PV is available.

 oc get pv | grep localblock


          CAPA- ACCESS RECLAIM                                STORAGE
NAME      CITY  MODES  POLICY  STATUS     CLAIM               CLASS       AGE
local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
3e8964d3                                  ocs-deviceset-2-0
                                          -79j94
local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
414755e0                                  ocs-deviceset-1-0
                                          -959rp
local-pv- 931Gi  RWO   Delete  Available                      localblock  3m24s
b481410
local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
d9c5cbd6                                  ocs-deviceset-0-0
                                          -nvs68

Change to the openshift-storage project.
```
oc project openshift-storage
```
Remove the failed OSD from the cluster.
Multiple failed OSDs can be specified, if required.
1. Identify the PVC.
  The associated PVs later get deleted.
```
 osd_id_to_remove=1
oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc
```
  In this example, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix.
  The deployment name here is rook-ceph-osd-1.
  Example output:
```
ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-g2mmc
    ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-g2mmc
```
  In this example, the PVC name is ocs-deviceset-localblock-0-data-0-g2mmc.
2. Remove the failed OSD from the cluster.
```
oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS=${osd_id_to_remove} |oc create -f -
```
  You can remove more than one OSD by adding comma separated OSD IDs in the command. (For example: FAILED_OSD_IDS=0,1,2)
  Warning: This step results in OSD being completely removed from the cluster. Ensure that the correct value of osd_id_to_remove is provided.
Remove the failed OSD from the cluster. You can specify multiple failed OSDs if required.
Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal pod.
A status of Completed confirms that the OSD removal job succeeded.
```
oc get pod -l job-name=ocs-osd-removal-osd_id_to_remove -n openshift-storage
```
Note: If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:
```
oc logs -l job-name=ocs-osd-removal-osd_id_to_remove -n openshift-storage
--tail=-1
```
It may be necessary to manually cleanup the removed OSD as follows:
```
ceph osd crush remove osd.osd_id_to_remove
ceph osd rm osd_id_to_remove
ceph auth del osd.osd_id_to_remove
ceph osd crush rm osd_id_to_remove
```

Delete the PV associated with the failed node.

Identify the PV associated with the PVC.

The PVC name must be identical to the name that is obtained while removing the failed OSD from the cluster.

 oc get pv -L kubernetes.io/hostname | grep localblock | grep Released

local-pv-5c9b8982 500Gi RWO Delete Released openshift-storage/ocs-deviceset-localblock-0-data-0-g2mmc localblock 24h worker-0

If there is a PV in Released state, delete it.

oc delete pv <persistent-volume>

For example:

oc delete pv local-pv-5c9b8982
persistentvolume "local-pv-5c9b8982" deleted

Identify the crashcollector pod deployment.

oc get deployment --selector=app=rook-ceph-crashcollector,node_name=<failed_node_name> -n openshift-storage

If there is an existing crashcollector pod deployment, delete it.

oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<failed_node_name> -n openshift-storage

Delete the ocs-osd-removal job.

oc delete job ocs-osd-removal-${osd_id_to_remove}

Example output:

job.batch "ocs-osd-removal-0" deleted

What to do next

Verify that the new node and all pods are running.

Verify that the new node is present in the output:

oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

Workloads > Pods and confirm that at least the following pods on the new node are in a Running state:
- csi-cephfsplugin-*
- csi-rbdplugin-*
Verify that all the other required Fusion Data Foundation pods are in Running state.
Verify that the new Object Storage Device (OSD) pods are running on the replacement node:
```
oc get pods -o wide -n openshift-storage| egrep -i <new_node_name> | egrep osd
```
If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.
For each of the new nodes identified in the previous step, do the following:
1. Create a debug pod and open a chroot environment for the one or more selected hosts:
```
oc debug node/<node_name>
```
```
chroot /host
```
2. Display the list of available block devices:, using the lsblk command.
  Check for the crypt keyword beside the one or more ocs-deviceset names.
If the verification steps fail, contact IBM Support.