Replacing operational nodes on IBM Z or LinuxONE infrastructure

Use this procedure to replace an operational node on IBM zSystems or LinuxONE infrastructure.

Procedure

  1. Identify the node, and get the labels on the node that you need to replace, where <node_name> specifies the name of the node that needs to be replaced.
    oc get nodes --show-labels | grep <node_name>
  2. Identify the monitor pod (mon) (if any), and OSDs that are running in the node that you need to replace.
    oc get pods -n openshift-storage -o wide | grep -i <node_name>
  3. Scale down the deployments of the pods identified in the previous step.
    oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
    oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
    oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage
  4. Mark the node as unscheduable
    oc adm cordon <node_name>
  5. Remove the pods which are in Terminating state.
    oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'
  6. Drain the node.
    oc adm drain <node_name> --force --delete-emptydir-data=true --ignore-daemonsets
  7. Delete the node.
    oc delete node <node_name>
  8. Get a new IBM zSystem storage node as a replacement.
  9. Check for certificate signing requests (CSRs) related to Fusion Data Foundation that are in Pending state.
    oc get csr
  10. Approve all required Fusion Data Foundation CSRs for the new node, where <certificate_name> specifies the name of the CSR.
    oc adm certificate approve <certificate_name>
  11. From the OpenShift Web Console, go to Compute > Nodes and confirm that the new node is in Ready state.
  12. Apply the Fusion Data Foundation label to the new node using one of the following steps:
    From the user interface
    1. Go to Action Menu > Edit Labels > .
    2. Add cluster.ocs.openshift.io/openshift-storage, and click Save.
    From the command-line interface
    Apply the Fusion Data Foundation label to the new node:, where <new_node_name> specifies the name of the new node.
    oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
  13. Add a new worker node to the localVolumeDiscovery and localVolumeSet.
    1. Update the localVolumeDiscovery definition to include the new node, and remove the failed node.
      oc edit -n local_storage_project localvolumediscovery auto-discover-devices
      Example output, where server3.example.com is removed, and newnode.example.com is the new node:
      [...]
         nodeSelector:
          nodeSelectorTerms:
            - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - server1.example.com
                  - server2.example.com
                  #- server3.example.com
                  - newnode.example.com
      [...]
      Remember: Save before exiting the editor.
    2. Determine the localVolumeSet to edit.
      Replace local-storage-project with the name of your local storage project. The default project name is openshift-local-storage in Fusion Data Foundation.
      oc get -n local-storage-project localvolumeset
      Example output:
      NAME          AGE
      localblock   25h
    3. Update the localVolumeSet definition.
      oc edit -n $local_storage_project localvolumeset localblock
      Example output, where server3.example.com is removed, and newnode.example.com is the new node:
      [...]
         nodeSelector:
          nodeSelectorTerms:
            - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - server1.example.com
                  - server2.example.com
                  #- server3.example.com
                  - newnode.example.com
      [...]
      Remember: Save before exiting the editor.
  14. Verify that the new localblock PV is available.
     oc get pv | grep localblock
    
              CAPA- ACCESS RECLAIM                                STORAGE
    NAME      CITY  MODES  POLICY  STATUS     CLAIM               CLASS       AGE
    local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
    3e8964d3                                  ocs-deviceset-2-0
                                              -79j94
    local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
    414755e0                                  ocs-deviceset-1-0
                                              -959rp
    local-pv- 931Gi  RWO   Delete  Available                      localblock  3m24s
    b481410
    local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
    d9c5cbd6                                  ocs-deviceset-0-0
                                              -nvs68
  15. Change to the openshift-storage project.
    oc project openshift-storage
  16. Remove the failed OSD from the cluster.
    Multiple failed OSDs can be specified, if required.
    1. Identify the PVC.
      The associated PVs later get deleted.
       osd_id_to_remove=1
      oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc
      In this example, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix.
      The deployment name here is rook-ceph-osd-1.
      Example output:
      ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-g2mmc
          ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-g2mmc
      In this example, the PVC name is ocs-deviceset-localblock-0-data-0-g2mmc.
    2. Remove the failed OSD from the cluster.
      oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS=${osd_id_to_remove} |oc create -f -
      You can remove more than one OSD by adding comma separated OSD IDs in the command. (For example: FAILED_OSD_IDS=0,1,2)
      Warning: This step results in OSD being completely removed from the cluster. Ensure that the correct value of osd_id_to_remove is provided.

    Remove the failed OSD from the cluster. You can specify multiple failed OSDs if required.

  17. Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal pod.
    A status of Completed confirms that the OSD removal job succeeded.
    oc get pod -l job-name=ocs-osd-removal-osd_id_to_remove -n openshift-storage
    Note: If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:
    oc logs -l job-name=ocs-osd-removal-osd_id_to_remove -n openshift-storage
    --tail=-1
    It may be necessary to manually cleanup the removed OSD as follows:
    ceph osd crush remove osd.osd_id_to_remove
    ceph osd rm osd_id_to_remove
    ceph auth del osd.osd_id_to_remove
    ceph osd crush rm osd_id_to_remove
  18. Delete the PV associated with the failed node.
    1. Identify the PV associated with the PVC.
      The PVC name must be identical to the name that is obtained while removing the failed OSD from the cluster.
       oc get pv -L kubernetes.io/hostname | grep localblock | grep Released
      local-pv-5c9b8982 500Gi RWO Delete Released openshift-storage/ocs-deviceset-localblock-0-data-0-g2mmc localblock 24h worker-0
    2. If there is a PV in Released state, delete it.
      oc delete pv <persistent-volume>
      For example:
      oc delete pv local-pv-5c9b8982
      persistentvolume "local-pv-5c9b8982" deleted
  19. Identify the crashcollector pod deployment.
    oc get deployment --selector=app=rook-ceph-crashcollector,node_name=<failed_node_name> -n openshift-storage

    If there is an existing crashcollector pod deployment, delete it.

    oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<failed_node_name> -n openshift-storage
  20. Delete the ocs-osd-removal job.
    oc delete job ocs-osd-removal-${osd_id_to_remove}
    Example output:
    job.batch "ocs-osd-removal-0" deleted

What to do next

Verify that the new node and all pods are running.
  1. Verify that the new node is present in the output:
    oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1
  2. Workloads > Pods and confirm that at least the following pods on the new node are in a Running state:
    • csi-cephfsplugin-*
    • csi-rbdplugin-*
  3. Verify that all the other required Fusion Data Foundation pods are in Running state.
  4. Verify that the new Object Storage Device (OSD) pods are running on the replacement node:
    oc get pods -o wide -n openshift-storage| egrep -i <new_node_name> | egrep osd
  5. If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.

    For each of the new nodes identified in the previous step, do the following:

    1. Create a debug pod and open a chroot environment for the one or more selected hosts:
      oc debug node/<node_name>
      chroot /host
    2. Display the list of available block devices:, using the lsblk command.

      Check for the crypt keyword beside the one or more ocs-deviceset names.

  6. If the verification steps fail, contact IBM Support.