Disk replacement and OpenShift dedicated OSD failure recovery

Learn how to replace a faulty disk in a Red Hat® OpenShift® dedicated Object Storage Device (OSD) cluster.

Procedure

  1. Identify the failed OSD.
    Command example:
    oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide
    An example output where osd-19 is failed and the status is CrashLoopBackOff:
    NAME                                READY   STATUS             RESTARTS        AGE     IP             NODE                                          NOMINATED NODE   READINESS GATES
rook-ceph-osd-0-85fcb5fd9-5cvws     2/2     Running            0               9d      9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-1-5dd8bc8d9d-cbz7g    2/2     Running            0               37d     9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-10-7cc49487b5-bdw6w   2/2     Running            0               9d      9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-11-84cd6fb7d7-xpn22   2/2     Running            0               37d     9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-12-7bb579498c-25tzh   2/2     Running            0               9d      9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-13-866dbc7f57-kqqpn   2/2     Running            0               37d     9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-14-6f7f6dd89b-7skg8   2/2     Running            0               37d     9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-15-697c5b8577-9plff   2/2     Running            0               6h58m   9.42.107.146   control-1-ru3.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-16-8d78df98c-khbh7    2/2     Running            0               37d     9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-17-c8ffbb5bf-q4xqb    2/2     Running            0               37d     9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-18-65f847c8d4-zpz2f   2/2     Running            0               37d     9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-19-85d475d68d-5kvmq   1/2     CrashLoopBackOff   6 (2m10s ago)   37d     9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
.
.
.
 <none>           <none>
rook-ceph-osd-7-7f79d4b76-jxz8x     2/2     Running            0               37d     9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-8-75497d6999-wz296    2/2     Running            0               37d     9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-9-56d5cc7c59-w29n2    2/2     Running            0               37d     9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
  2. Scale down ocs-operator first, then rook-ceph-operator.
    1. Scale down ocs-operator.
      1. Log in to the Red Hat OpenShift Container Platform web console and select your project.
      2. Go to Workloads > Pods page and verify that ocs-operator is in the Running status in the Pods list.
      3. Go to Workloads > Deployments page.

      4. Click ocs-operator and check the Deployment details tab.

        For example, the Deployment details tab displays that ocs-operator has 1 Pod.

      5. Scale down the Pod number to 0.

      6. Go back to the Pods tab and check for the ocs-operator.

        The ocs-operator does not appear or is no longer available in the Pods list.

    2. Scale down rook-ceph-operator.
      1. Go to Workloads > Pods page and verify that rook-ceph-operator is in the Running status in the Pods list.
      2. Go to Workloads > Deployments page.

      3. Click rook-ceph-operator and check the Deployment details tab.

        For example, the Deployment details tab displays that rook-ceph-operator has 1 Pod.

      4. Scale down the Pod number to 0.

      5. Go back to the Pods tab and check for the rook-ceph-operator.

        The rook-ceph-operator does not appear or is no longer available in the Pods list.

  3. Optional: Verify whether Pods for ocs-operator and rook-ceph-operator are removed as expected.
    Command example:
    oc get pods |grep operator
    Example output:
    noobaa-operator-79446cc789-gj66l                                  1/1     Running            0              38d
ocs-client-operator-console-7d85dc6bf9-vdbhn                      1/1     Running            0              9d
ocs-client-operator-controller-manager-85bbcc7bfd-znvx4           2/2     Running            1 (6d1h ago)   37d
odf-operator-controller-manager-59465654c-jmj5z                   2/2     Running            0              38d
  4. Clean up failed OSD.
    1. Scale down the rook-ceph-osd deployment.
      Command example:
      osd_id_to_remove=<replace-it-with-osd-id>
oc scale -n openshift-storage deployment rook-ceph-osd-${osd_id_to_remove} --replicas=0
      Example output:
      deployment.apps/rook-ceph-osd-19 scaled
    2. Verify that the rook-ceph-osd Pod for the faulty OSD is deleted.
      Command example:
      oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide
      Example output:
      NAME                                READY   STATUS    RESTARTS   AGE     IP             NODE                                          NOMINATED NODE   READINESS GATES
rook-ceph-osd-0-85fcb5fd9-5cvws     2/2     Running   0          9d      9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-1-5dd8bc8d9d-cbz7g    2/2     Running   0          37d     9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-10-7cc49487b5-bdw6w   2/2     Running   0          9d      9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-11-84cd6fb7d7-xpn22   2/2     Running   0          37d     9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-12-7bb579498c-25tzh   2/2     Running   0          9d      9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-13-866dbc7f57-kqqpn   2/2     Running   0          37d     9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-14-6f7f6dd89b-7skg8   2/2     Running   0          37d     9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-15-697c5b8577-9plff   2/2     Running   0          7h16m   9.42.107.146   control-1-ru3.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-16-8d78df98c-khbh7    2/2     Running   0          37d     9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-17-c8ffbb5bf-q4xqb    2/2     Running   0          37d     9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-18-65f847c8d4-zpz2f   2/2     Running   0          37d     9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-2-5fd5548c56-gclrv    2/2     Running   0          37d     9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-20-b88868db5-h55h9    2/2     Running   0          37d     9.42.107.147   control-1-ru4.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
.
.
.
.
rook-ceph-osd-8-75497d6999-wz296    2/2     Running   0          37d     9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-9-56d5cc7c59-w29n2    2/2     Running   0          37d     9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
  5. Remove the old OSD from the cluster.
    1. Delete any existing ocs-osd-removal jobs, if present.
      Command example:
      oc delete -n openshift-storage job ocs-osd-removal-job
      Example output:
      job.batch "ocs-osd-removal-job" deleted
    2. Remove the old OSD from the cluster
      Command example:
      oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS=${osd_id_to_remove} FORCE_OSD_REMOVAL=true |oc create -n openshift-storage -f -
      Important: Ensure that you set the correct osd_id_to_remove.
      Example output:
      job.batch/ocs-osd-removal-job created
    3. Verify that the ocs-osd-removal-job has completed.
      Command example:
      oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage
      Example output:
      NAME                        READY   STATUS      RESTARTS   AGE
ocs-osd-removal-job-rh72h   0/1     Completed   0          39s

      After the ocs-osd-removal-job completes, the associated Persistent Volumes (PVs) are released, and the Persistent Volume Claims (PVCs) are either deleted or move to the Pending state.

    4. Check the PV state.
      Command example:
      oc get pv |grep Released
      Example output:
      local-pv-141605d2                          7153Gi     RWO            Delete           Released   openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465                          ibm-spectrum-fusion-local     <unset>                          37d
    5. If encryption is enabled, remove the associated configuration.

      Do the following steps:

      1. Retrieve PVC and node details.

        Command example:
        oc describe pv local-pv-141605d2
        Example output:
        Name:              local-pv-141605d2
Labels:            kubernetes.io/hostname=control-1-ru2.isf-racka.rtp.raleigh.ibm.com
                   storage.openshift.com/owner-kind=LocalVolumeSet
                   storage.openshift.com/owner-name=ibm-spectrum-fusion-local
                   storage.openshift.com/owner-namespace=openshift-local-storage
Annotations:       pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/provisioned-by: local-volume-provisioner-control-1-ru2.isf-racka.rtp.raleigh.ibm.com
                   storage.openshift.com/device-id: nvme-MZWLJ7T6HALA-000V5_S5LLNE0R400096
                   storage.openshift.com/device-name: nvme1n1
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      ibm-spectrum-fusion-local
Status:            Released
Claim:             openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Block
Capacity:          7153Gi
Node Affinity:     
  Required Terms:  
    Term 0:        kubernetes.io/hostname in [control-1-ru2.isf-racka.rtp.raleigh.ibm.com]
Message:           
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /mnt/local-storage/ibm-spectrum-fusion-local/nvme-MZWLJ7T6HALA-000V5_S5LLNE0R400096
Events:
  Type     Reason              Age                   From     Message
  ----     ------              ----                  ----     -------
  Warning  VolumeFailedDelete  15s (x15 over 3m20s)  deleter  Error cleaning PV "local-pv-141605d2": failed to get volume mode of path "/mnt/local-storage/ibm-spectrum-fusion-local/nvme-MZWLJ7T6HALA-000V5_S5LLNE0R400096": Directory check for "/mnt/local-storage/ibm-spectrum-fusion-local/nvme-MZWLJ7T6HALA-000V5_S5LLNE0R400096" failed: open /mnt/local-storage/ibm-spectrum-fusion-local/nvme-MZWLJ7T6HALA-000V5_S5LLNE0R400096: no such file or directory

      2. Remove the dm-crypt–managed device-mapper mapping from the OSD devices.

        Command example:
        oc debug node/<node name
chroot /host
dmsetup ls| grep <pvc name>
        Example output:
        ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt	(253:3)

      3. Remove the mapped device.

        Command example:
        cryptsetup luksClose --debug --verbose ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt
        Example output:
        # cryptsetup 2.6.0 processing "cryptsetup luksClose --debug --verbose ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt"
# Verifying parameters for command close.
# Running command close.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device context by device ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt.
# Initialising device-mapper backend library.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.48.0.
# Detected dm-crypt version 1.24.0.
# Device-mapper backend running with UDEV support enabled.
# dm status ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount noflush ]   [16384] (*1)
# Releasing device-mapper backend.
# Allocating context for crypt device (none).
# Initialising device-mapper backend library.
Underlying device for crypt device ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt disappeared.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount flush securedata ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount flush ]   [16384] (*1)
# LUKS device header not available.
# Deactivating volume ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm status ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount noflush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount flush securedata ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount flush securedata ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Udev cookie 0xd4d2da5 (semid 0) created
# Udev cookie 0xd4d2da5 (semid 0) incremented to 1
# Udev cookie 0xd4d2da5 (semid 0) incremented to 2
# Udev cookie 0xd4d2da5 (semid 0) assigned to REMOVE task(2) with flags DISABLE_LIBRARY_FALLBACK         (0x20)
# dm remove ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465-block-dmcrypt  [ opencount flush retryremove ]   [16384] (*1)
# Udev cookie 0xd4d2da5 (semid 0) decremented to 0
# Udev cookie 0xd4d2da5 (semid 0) waiting for zero
# Udev cookie 0xd4d2da5 (semid 0) destroyed
# Releasing crypt device empty context.
# Releasing device-mapper backend.
Command successful.
    6. Identify and delete the failed PV.

      Do the following steps:

      1. Identify the failed PV.

        Command example:
        oc get pv -l kubernetes.io/hostname=control-1-ru2.isf-racka.rtp.raleigh.ibm.com

        The status of the failed PV shows as Released.

        Example output:
        NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                                                      STORAGECLASS                VOLUMEATTRIBUTESCLASS   REASON   AGE
local-pv-141605d2   7153Gi     RWO            Delete           Released   openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-32b6465   ibm-spectrum-fusion-local   <unset>                          37d
local-pv-4ce100b9   7153Gi     RWO            Delete           Bound      openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-26px6qd   ibm-spectrum-fusion-local   <unset>                          37d
local-pv-5b320962   7153Gi     RWO            Delete           Bound      openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-29w8c49   ibm-spectrum-fusion-local   <unset>                          37d
local-pv-6ba1664e   7153Gi     RWO            Delete           Bound      openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-334h7jp   ibm-spectrum-fusion-local   <unset>                          22d
local-pv-89429641   7153Gi     RWO            Delete           Bound      openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-9tc4vl    ibm-spectrum-fusion-local   <unset>                          37d
local-pv-a3bb3040   7153Gi     RWO            Delete           Bound      openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-5dtrs9    ibm-spectrum-fusion-local   <unset>                          37d
local-pv-e9ed407b   7153Gi     RWO            Delete           Bound      openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-17k74xw   ibm-spectrum-fusion-local   <unset>                          37d

      2. Delete the failed PV.

        Command example:
        oc delete pv local-pv-141605d2
        Example output:
        persistentvolume "local-pv-141605d2" deleted

      3. Verify that the failed PV is deleted.

        Command example:
        oc get pv -l kubernetes.io/hostname=control-1-ru2.isf-racka.rtp.raleigh.ibm.com
        Example output:
        NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                      STORAGECLASS                VOLUMEATTRIBUTESCLASS   REASON   AGE
local-pv-4ce100b9   7153Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-26px6qd   ibm-spectrum-fusion-local   <unset>                          37d
local-pv-5b320962   7153Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-29w8c49   ibm-spectrum-fusion-local   <unset>                          37d
local-pv-6ba1664e   7153Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-334h7jp   ibm-spectrum-fusion-local   <unset>                          22d
local-pv-89429641   7153Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-9tc4vl    ibm-spectrum-fusion-local   <unset>                          37d
local-pv-a3bb3040   7153Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-5dtrs9    ibm-spectrum-fusion-local   <unset>                          37d
local-pv-e9ed407b   7153Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-17k74xw   ibm-spectrum-fusion-local   <unset>                          37d
  6. Verify that the stale OSD entry is removed.
    Command example:
    oc get pods
    Example output:
    NAME                                                              READY   STATUS      RESTARTS       AGE
csi-addons-controller-manager-585444c85d-vlklj                    2/2     Running     0              38d
csi-cephfsplugin-5rc9r                                            2/2     Running     0              37d
csi-cephfsplugin-6zvlj                                            2/2     Running     0              37d
csi-cephfsplugin-8rpcc                                            2/2     Running     0              37d
csi-cephfsplugin-9xtqk                                            2/2     Running     4 (46h ago)    37d
csi-cephfsplugin-k6jf9                                            2/2     Running     0              37d
csi-cephfsplugin-provisioner-796d9bb95c-g6dsp                     5/5     Running     0              37d
csi-cephfsplugin-provisioner-796d9bb95c-klt8v                     5/5     Running     3 (9d ago)     9d
csi-cephfsplugin-srk79                                            2/2     Running     0              37d
csi-rbdplugin-dzfln                                               3/3     Running     0              37d
csi-rbdplugin-j5jk5                                               3/3     Running     5 (46h ago)    37d
csi-rbdplugin-lw5lt                                               3/3     Running     0              37d
csi-rbdplugin-mf6pp                                               3/3     Running     0              37d
csi-rbdplugin-nc9g9                                               3/3     Running     0              37d
csi-rbdplugin-provisioner-7cf4b47dd9-89j27                        6/6     Running     0              37d
csi-rbdplugin-provisioner-7cf4b47dd9-9w4zv                        6/6     Running     0              37d
csi-rbdplugin-tb2wf                                               3/3     Running     0              37d
noobaa-core-0                                                     2/2     Running     0              37d
noobaa-db-pg-0                                                    1/1     Running     0              37d
noobaa-endpoint-68c99f7ddf-vlfbs                                  1/1     Running     0              37d
noobaa-operator-79446cc789-gj66l                                  1/1     Running     0              38d
ocs-client-operator-console-7d85dc6bf9-vdbhn                      1/1     Running     0              9d
ocs-client-operator-controller-manager-85bbcc7bfd-znvx4           2/2     Running     1 (6d1h ago)   37d
ocs-metrics-exporter-858847c98f-m6x4l                             1/1     Running     0              37d
ocs-osd-removal-job-rh72h                                         0/1     Completed   0              22m
ocs-provider-server-6d49cffc97-q4vfm                              1/1     Running     0              37d
odf-console-69bdbf8bc4-xt7jh                                      1/1     Running     0              38d
odf-operator-controller-manager-59465654c-jmj5z                   2/2     Running     0              38d
rook-ceph-crashcollector-18f4004ce1313cc06e57f6eb215a1321-lhpn6   1/1     Running     0              37d
rook-ceph-crashcollector-3f4d91760a68a6dbc7579b3e8b5915dd-m84jq   1/1     Running     0              8d
rook-ceph-crashcollector-6ff2efb9f9fe11fb194447b2fbf5c1e4-85ztt   1/1     Running     0              9d
rook-ceph-crashcollector-74af1983e44d54775407e88bf7382e9e-vchft   1/1     Running     0              37d
rook-ceph-crashcollector-9178b690870ea615c9dc0b5706acedea-84kcs   1/1     Running     0              37d
rook-ceph-crashcollector-e67f06af5747db0daafd2ddc1be00528-wlkn8   1/1     Running     0              46h
rook-ceph-exporter-compute-1-ru5.isf-racka.rtp.raleigh.ibm7p7df   1/1     Running     0              37d
rook-ceph-exporter-compute-1-ru6.isf-racka.rtp.raleigh.ibmz7qk8   1/1     Running     0              37d
rook-ceph-exporter-compute-1-ru7.isf-racka.rtp.raleigh.ibm4v9ct   1/1     Running     0              8d
rook-ceph-exporter-control-1-ru2.isf-racka.rtp.raleigh.ibmrd6f9   1/1     Running     0              9d
rook-ceph-exporter-control-1-ru3.isf-racka.rtp.raleigh.ibms5wxb   1/1     Running     0              46h
rook-ceph-exporter-control-1-ru4.isf-racka.rtp.raleigh.ibmtq9dr   1/1     Running     0              37d
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-6b557764fgdv7   2/2     Running     10 (46h ago)   37d
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-6d8c97d72fjts   2/2     Running     10 (46h ago)   9d
rook-ceph-mgr-a-864fb44895-pl2n8                                  3/3     Running     0              46h
rook-ceph-mgr-b-7cf6b49bcd-q5k49                                  3/3     Running     0              37d
rook-ceph-mon-a-79897dd546-hbstb                                  2/2     Running     0              37d
rook-ceph-mon-c-bf487b7c7-flcpf                                   2/2     Running     0              37d
rook-ceph-mon-d-76f96d4fd4-h4h9t                                  2/2     Running     0              46h
rook-ceph-osd-0-85fcb5fd9-5cvws                                   2/2     Running     0              9d
rook-ceph-osd-1-5dd8bc8d9d-cbz7g                                  2/2     Running     0              37d
rook-ceph-osd-10-7cc49487b5-bdw6w                                 2/2     Running     0              9d
rook-ceph-osd-11-84cd6fb7d7-xpn22                                 2/2     Running     0              37d
rook-ceph-osd-12-7bb579498c-25tzh                                 2/2     Running     0              9d
rook-ceph-osd-13-866dbc7f57-kqqpn                                 2/2     Running     0              37d
rook-ceph-osd-14-6f7f6dd89b-7skg8                                 2/2     Running     0              37d
rook-ceph-osd-15-697c5b8577-9plff                                 2/2     Running     0              7h41m
rook-ceph-osd-16-8d78df98c-khbh7                                  2/2     Running     0              37d
rook-ceph-osd-17-c8ffbb5bf-q4xqb                                  2/2     Running     0              37d
rook-ceph-osd-18-65f847c8d4-zpz2f                                 2/2     Running     0              37d
rook-ceph-osd-2-5fd5548c56-gclrv                                  2/2     Running     0              37d
rook-ceph-osd-20-b88868db5-h55h9                                  2/2     Running     0              37d
rook-ceph-osd-21-556947cb6f-k29mm                                 2/2     Running     0              37d
rook-ceph-osd-22-5d79578d54-b9qlb                                 2/2     Running     0              37d
rook-ceph-osd-23-84bcb674c5-6jzxx                                 2/2     Running     0              37d
rook-ceph-osd-24-6df4949c8c-qvczd                                 2/2     Running     0              37d
rook-ceph-osd-26-5d7b7b6f46-vqgz4                                 2/2     Running     0              46h
rook-ceph-osd-27-59b8c85b78-wz4pb                                 2/2     Running     0              46h
rook-ceph-osd-28-54dc8db898-ml8r5                                 2/2     Running     0              46h
rook-ceph-osd-29-5666b4975c-qrpm5                                 2/2     Running     0              46h
rook-ceph-osd-3-7cfc7899dc-px8xm                                  2/2     Running     0              9d
rook-ceph-osd-30-7cc88bfcfc-xsnvc                                 2/2     Running     0              37d
rook-ceph-osd-31-78cc567d99-8jhkh                                 2/2     Running     0              37d
rook-ceph-osd-32-6c5fcdfb45-4jz2q                                 2/2     Running     0              46h
rook-ceph-osd-33-694f48d4f8-4sflf                                 2/2     Running     0              22d
rook-ceph-osd-4-6f4fb99d7-w7vrb                                   2/2     Running     0              37d
rook-ceph-osd-5-7b7ff5cfb5-blpj7                                  2/2     Running     0              37d
rook-ceph-osd-6-5d9856b54f-bjpmm                                  2/2     Running     0              9d
rook-ceph-osd-7-7f79d4b76-jxz8x                                   2/2     Running     0              37d
rook-ceph-osd-8-75497d6999-wz296                                  2/2     Running     0              37d
rook-ceph-osd-9-56d5cc7c59-w29n2                                  2/2     Running     0              37d
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-668df44tbvtn   2/2     Running     0              37d
rook-ceph-tools-68d744d548-jwj5l                                  1/1     Running     0              23d
storageclient-737342087af10580-status-reporter-29061474-tv52t     0/1     Completed   0              51s
ux-backend-server-7796f96896-kwtsf                                2/2     Running     0              38d
  7. Add a new disk to the node.
    Important:
    • If a fresh or unused disk is inserted, a new PV is created in the Available state.
    • If a used disk is inserted as a replacement, it must be formatted for the corresponding PV to become available.

    After adding the new disk, verify the status of the new PV.

    Command example:
    oc get pv |grep local
    Example output:
    local-pv-159e1b8e                          7153Gi     RWO            Delete           Bound       openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-11ms68d                          ibm-spectrum-fusion-local     <unset>                          37d
local-pv-2ac8a85b                          7153Gi     RWO            Delete           Bound       openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-2xdlrk                           ibm-spectrum-fusion-local     <unset>                          37d
local-pv-406f1af8                          7153Gi     RWO            Delete           Bound       openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-128sb6h                          ibm-spectrum-fusion-local     <unset>                          37d
local-pv-451f6341                          7153Gi     RWO            Delete           Bound       openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-31wkjwp                          ibm-spectrum-fusion-local     <unset>                          37d
local-pv-4ce100b9                          7153Gi     RWO            Delete           Bound       openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-26px6qd                          ibm-spectrum-fusion-local     <unset>                          37d
local-pv-4da6a449                          7153Gi     RWO            Delete           Available                                                                                                     ibm-spectrum-fusion-local     <unset>                          2m23s
local-pv-521ee04f                          7153Gi     RWO            Delete           Bound       openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-30g5vj5  
.
.
.
.
local-pv-b7c4932c                          7153Gi     RWO            Delete           Bound       openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-2825ldc                          ibm-spectrum-fusion-local     <unset>                          37d
  8. Scale up ocs-operator first, then rook-ceph-operator.
    1. Scale up ocs-operator.
      1. Go to Workloads > Deployments page.

      2. Click ocs-operator and check the Deployment details tab.

        For example, the Deployment details tab displays that ocs-operator has 0 Pod.

      3. Scale up the Pod number to 1.
      4. Go to Workloads > Pods page and verify that ocs-operator is in the Running status in the Pods list.
    2. Scale up rook-ceph-operator.
      1. Go to Workloads > Deployments page.

      2. Click rook-ceph-operator and check the Deployment details tab.

        For example, the Deployment details tab displays that rook-ceph-operator has 0 Pod.

      3. Scale up the Pod number to 1.
      4. Go to Workloads > Pods page and verify that rook-ceph-operator is in the Running status in the Pods list.
  9. Verify that the new OSD Pod is running.
    Command example:
    oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide
    Example output:
    NAME                                READY   STATUS    RESTARTS   AGE    IP             NODE                                          NOMINATED NODE   READINESS GATES
rook-ceph-osd-0-85fcb5fd9-5cvws     2/2     Running   0          9d     9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-1-5dd8bc8d9d-cbz7g    2/2     Running   0          37d    9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-10-7cc49487b5-bdw6w   2/2     Running   0          9d     9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-11-84cd6fb7d7-xpn22   2/2     Running   0          37d    9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-12-7bb579498c-25tzh   2/2     Running   0          9d     9.42.107.150   compute-1-ru7.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-13-866dbc7f57-kqqpn   2/2     Running   0          37d    9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-14-6f7f6dd89b-7skg8   2/2     Running   0          37d    9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-15-697c5b8577-9plff   2/2     Running   0          8h     9.42.107.146   control-1-ru3.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-16-8d78df98c-khbh7    2/2     Running   0          37d    9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-17-c8ffbb5bf-q4xqb    2/2     Running   0          37d    9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-18-65f847c8d4-zpz2f   2/2     Running   0          37d    9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-19-74cc9c8bc6-cc8p4   2/2     Running   0          111s   9.42.107.145   control-1-ru2.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-2-5fd5548c56-gclrv    2/2     Running   0          37d    9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-20-b88868db5-h55h9    2/2     Running   0          37d    9.42.107.147   control-1-ru4.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
.
.
.
rook-ceph-osd-8-75497d6999-wz296    2/2     Running   0          37d    9.42.107.148   compute-1-ru5.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
rook-ceph-osd-9-56d5cc7c59-w29n2    2/2     Running   0          37d    9.42.107.149   compute-1-ru6.isf-racka.rtp.raleigh.ibm.com   <none>           <none>
  10. Verify that the PV and PVC are successfully created.
    Command example for PVC:
    oc get pvc -n openshift-storage |grep local-pv-4da6a449
    Command example for PV:
    oc get pv -n openshift-storage |grep local-pv-4da6a449
    Example output for PVC:
    ocs-deviceset-ibm-spectrum-fusion-local-0-data-35tw7jf   Bound    local-pv-4da6a449                          7153Gi     RWO            ibm-spectrum-fusion-local     <unset>                 13h
    Example output for PV:
    local-pv-4da6a449                          7153Gi     RWO            Delete           Bound    openshift-storage/ocs-deviceset-ibm-spectrum-fusion-local-0-data-35tw7jf                          ibm-spectrum-fusion-local     <unset>                          13h

Results

Faulty disk or failed OSD replacement is complete.

What to do next

Wait for the storage cluster to reconcile its health status. If it remains in the Warning state despite all Fusion Data Foundation Pods running, check for and resolve any previous crashes.

To verify the health status, do as follows:
  • Log in to the IBM Fusion web console and go to Storage > Local storage to check the health of the local storage.
  • In the Red Hat OpenShift Container Platform web console, go to Storage > Data Foundation and check the health of the storage.