Replacing operational or failed storage devices on IBM Power

Use this information to replace operational or failed storage devices on IBM Power infrastructure.

Before you begin

It is recommended that replacement devices are configured with similar infrastructure and resources to the device being replaced.
Ensure that the data is resilient.
In the OpenShift Web Console, click Storage > Data Foundation.
Click the Storage Systems tab, and then click ocs-storagecluster-storagesystem.
In the Status card of Block and File dashboard, under the Overview tab, verify that Data Resiliency has a green tick mark.

About this task

You can replace an object storage device (OSD) in Fusion Data Foundation deployed using local storage devices on IBM Power.

Note: One or more underlying storage devices may need to be replaced.

Procedure

Identify the OSD that needs to be replaced and the OpenShift Container Platform node that has the OSD scheduled on it.

oc get -n openshift-storage pods -l app=rook-ceph-osd -o wide

Example output:

rook-ceph-osd-0-86bf8cdc8-4nb5t   0/1     crashLoopBackOff   0   24h   10.129.2.26     worker-0     <none>       <none>
rook-ceph-osd-1-7c99657cfb-jdzvz   1/1     Running   0          24h     10.128.2.46     worker-1     <none>       <none>
rook-ceph-osd-2-5f9f6dfb5b-2mnw9    1/1     Running   0          24h     10.131.0.33    worker-2     <none>       <none>

In this example, rook-ceph-osd-0-86bf8cdc8-4nb5t needs to be replaced and worker-0 is the OpenShift Container platform node on which the OSD is scheduled.

Note: If the OSD to be replaced is healthy, the status of the pod will be Running.

Scale down the OSD deployment for the OSD to be replaced.
Each time you want to replace the OSD, update the osd_id_to_remove parameter with the OSD ID, and repeat this step.
```
$ osd_id_to_remove=0
oc scale -n openshift-storage deployment rook-ceph-osd-${osd_id_to_remove} --replicas=0
```
where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-0.
Example output:
```
deployment.extensions/rook-ceph-osd-0 scaled
```

Verify that the rook-ceph-osd pod is terminated.

oc get -n openshift-storage pods -l ceph-osd-id=${osd_id_to_remove}

Example output:

No resources found.

Important: If the rook-ceph-osd pod is in terminating state, use the force option to delete the pod.

oc delete pod rook-ceph-osd-0-86bf8cdc8-4nb5t --force --grace-period=0

Example output:

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
  pod "rook-ceph-osd-0-86bf8cdc8-4nb5t" force deleted

Remove the old OSD from the cluster so that you can add a new OSD.
1. Identify the DeviceSet associated with the OSD to be replaced.
```
oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc
```
  Example output:
```
ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-64xjl
    ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-64xjl
```
  In this example, the Persistent Volume Claim (PVC) name is ocs-deviceset-localblock-0-data-0-64xjl.
2. Identify the Persistent Volume (PV) associated with the PVC.
```
oc get -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix>
```
  where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.
  Example output:
```
NAME                                      STATUS        VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS   AGE
           ocs-deviceset-localblock-0-data-0-64xjl   Bound    local-pv-8137c873    256Gi      RWO           localblock     24h
```
  In this example, the associated PV is local-pv-8137c873.
3. Identify the name of the device to be replaced.
```
oc get pv local-pv-<pv-suffix> -o yaml | grep path
```
  where, pv-suffix is the value in the PV name identified in an earlier step.
  Example output:
```
path: /mnt/local-storage/localblock/vdc
```
  In this example, the device name is vdc.
4. Identify the prepare-pod associated with the OSD to be replaced.
```
oc describe -n openshift-storage pvc ocs-deviceset-<x>-<y>-<pvc-suffix> | grep Used
```
  where, x, y, and pvc-suffix are the values in the DeviceSet identified in an earlier step.
  Example output:
```
Used By:    rook-ceph-osd-prepare-ocs-deviceset-localblock-0-data-0-64knzkc
```
5. Delete any old ocs-osd-removal jobs.
```
$ oc delete -n openshift-storage job ocs-osd-removal-job
```
  Example output:
```
job.batch "ocs-osd-removal-job" deleted
```
6. Change to the openshift-storageproject.
```
$ oc project openshift-storage
```
7. Remove the old OSD from the cluster.
```
$ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS=${osd_id_to_remove} FORCE_OSD_REMOVAL=false |oc create -n openshift-storage -f -
```
  The FORCE_OSD_REMOVAL value must be changed to true in clusters that only have three OSDs, or clusters with insufficient space to restore all three replicas of the data after the OSD is removed.
  Warning: This step results in OSD being completely removed from the cluster. Ensure that the correct value of osd_id_to_remove is provided.
Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal-job pod.
A status of Completed confirms that the OSD removal job succeeded.
```
$ oc get pod -l job-name=ocs-osd-removal-job -n openshift-storage
```

Ensure that the OSD removal is completed.

$ oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1 | egrep -i 'completed removal'

Example output:

2022-05-10 06:50:04.501511 I | cephosd: completed removal of OSD

Important: If the ocs-osd-removal-job fails and the pod is not in the expected Completed state, check the pod logs for further debugging.

For example:

# oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1

If encryption was enabled at the time of install, remove dm-crypt managed device-mapper mapping from the OSD devices that are removed from the respective Fusion Data Foundation nodes.
1. Get the PVC name(s) of the replaced OSD(s) from the logs of ocs-osd-removal-job pod.
```
oc logs -l job-name=ocs-osd-removal-job -n openshift-storage --tail=-1  |egrep -i ‘pvc|deviceset’
```
```
2021-05-12 14:31:34.666000 I | cephosd: removing the OSD PVC "ocs-deviceset-xxxx-xxx-xxx-xxx"
```
2. For each of the previously identified nodes, do the following:
  1. Create a debug pod and chroot to the host on the storage node, where <node name> is the name of the node.
```
oc debug node/<node name>
```
```
$ chroot /host
```
  2. Find a relevant device name based on the PVC names identified in the previous step, where <pvc name> is the name of the PVC.
```
dmsetup ls| grep <pvc name>
```
    Example output:
    ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt (253:0)
  3. Remove the mapped device.
```
$ cryptsetup luksClose --debug --verbose ocs-deviceset-xxx-xxx-xxx-xxx-block-dmcrypt
```
    Important: If the above command gets stuck due to insufficient privileges, run the following commands:
    
    Press CTRL+Z to exit the above command.
    
    Find the PID of the process which was stuck.
    $ ps -ef | grep crypt
    
    Terminate the process using the kill command.
    kill -9 <PID>
    where <PID>is the process ID.
    
    Verify that the device name is removed.
    $ dmsetup ls

Find the PV that need to be deleted.

oc get pv -L kubernetes.io/hostname | grep localblock | grep Released

Example output:

local-pv-d6bf175b           1490Gi       RWO         Delete          Released            openshift-storage/ocs-deviceset-0-data-0-6c5pw      localblock      2d22h       compute-1

Delete the PV, where <pv-name> is name of the PV.
```
oc delete pv <pv-name>
```

Replace the old device and use the new device to create a new OpenShift Container Platform PV.

In this example, the OpenShift Container Platform node is worker-0.

$ oc debug node/worker-0

Example output:

Starting pod/worker-0-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.88.21
If you don't see a command prompt, try pressing enter.
# chroot /host

Record the /dev/disk that is to be replaced using the device name, vdc, identified earlier.

# ls -alh /mnt/local-storage/localblock

Example output:

total 0
drwxr-xr-x. 2 root root 17 Nov  18 15:23 .
drwxr-xr-x. 3 root root 24 Nov  18 15:23 ..
lrwxrwxrwx. 1 root root  8 Nov  18 15:23 vdc -> /dev/vdc

Find the name of the LocalVolume CR, and remove or comment out the device /dev/disk that is to be replaced.

$ oc get -n openshift-local-storage localvolume

Example output:

NAME          AGE
localblock   25h

# oc edit -n openshift-local-storage localvolume localblock

Example output:

[...]
    storageClassDevices:
    - devicePaths:
   #   - /dev/vdc
      storageClassName: localblock
      volumeMode: Block
[...]

Make sure to save the changes after editing the CR.

$ oc debug node/worker-0

Example output:

Starting pod/worker-0-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.88.21
If you don't see a command prompt, try pressing enter.
# chroot /host

Identify the old symlink for the device name to be replaced.

In this example, the device name is vdc.

# ls -alh /mnt/local-storage/localblock

Example output:

total 0
drwxr-xr-x. 2 root root 17 Nov  18 15:23 .
drwxr-xr-x. 3 root root 24 Nov  18 15:23 ..
lrwxrwxrwx. 1 root root  8 Nov  18 15:23 vdc -> /dev/vdc

Remove the symlink.
```
# rm /mnt/local-storage/localblock/vdc
```

Verify that the symlink is removed.

# ls -alh /mnt/local-storage/localblock

Example output:

total 0
drwxr-xr-x. 2 root root 6 Nov 18 17:11 .
drwxr-xr-x. 3 root root 24 Nov 18 15:23 ..

Replace the old device with the new device.

Log back into the correct OpenShift Container Platform node and identify the device name for the new drive.

The device name must change unless you are resetting the same device.

# lsblk

Example output:

NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
vda                          252:0    0   40G  0 disk
|-vda1                       252:1    0    4M  0 part
|-vda2                       252:2    0  384M  0 part /boot
`-vda4                       252:4    0 39.6G  0 part
  `-coreos-luks-root-nocrypt 253:0    0 39.6G  0 dm   /sysroot
vdb                          252:16   0  512B  1 disk
vdd                          252:32   0  256G  0 disk

In this example, the new device name is vdd.

After the new /dev/disk is available, you can add a new disk entry to the LocalVolume CR.

In this example, the new device is /dev/vdd.

# oc edit -n openshift-local-storage localvolume localblock

Example output:

[...]
    storageClassDevices:
    - devicePaths:
    #  - /dev/vdc
      - /dev/vdd
      storageClassName: localblock
      volumeMode: Block
[...]

Make sure to save the changes after editing the CR.

Verify that there is a new PV in Available state and of the correct size.

$ oc get pv | grep 256Gi

Example output:

local-pv-1e31f771   256Gi   RWO    Delete  Bound  openshift-storage/ocs-deviceset-localblock-2-data-0-6xhkf   localblock    24h
local-pv-ec7f2b80   256Gi   RWO    Delete  Bound  openshift-storage/ocs-deviceset-localblock-1-data-0-hr2fx   localblock    24h
local-pv-8137c873   256Gi   RWO    Delete  Available

Create the new OSD for the new device.

Deploy the new OSD.

Restart the rook-ceph-operator to force operator reconciliation.

Identify the name of the rook-ceph-operator.

$ oc get -n openshift-storage pod -l app=rook-ceph-operator

Example output:

NAME                                  READY   STATUS    RESTARTS   AGE
rook-ceph-operator-85f6494db4-sg62v   1/1     Running   0          1d20h

Delete the rook-ceph-operator.
```
$ oc delete -n openshift-storage pod rook-ceph-operator-85f6494db4-sg62v
```
Example output:
```
pod "rook-ceph-operator-85f6494db4-sg62v" deleted
```
In this example, the rook-ceph-operator pod name is rook-ceph-operator-85f6494db4-sg62v.

Verify that the rook-ceph-operator pod is restarted.

$ oc get -n openshift-storage pod -l app=rook-ceph-operator

Example output:

NAME                                  READY   STATUS    RESTARTS   AGE
rook-ceph-operator-85f6494db4-wx9xx   1/1     Running   0          50s

Creation of the new OSD may take several minutes after the operator restarts.

Delete the ocs-osd-removal jobs.
```
$ oc delete -n openshift-storage job ocs-osd-removal-job
```
Example output:
```
job.batch "ocs-osd-removal-job" deleted
```
Note: When using an external key management system (KMS) with data encryption, the old OSD encryption key can be removed from the Vault server as it is now an orphan key.

What to do next

Verify that there is a new OSD in a Running state.

oc get -n openshift-storage pods -l app=rook-ceph-osd

Example output:

rook-ceph-osd-0-76d8fb97f9-mn8qz   1/1     Running   0          23m
rook-ceph-osd-1-7c99657cfb-jdzvz   1/1     Running   1          25h
rook-ceph-osd-2-5f9f6dfb5b-2mnw9   1/1     Running   0          25h

Verify that a new PVC created.

oc get -n openshift-storage pvc | grep localblock

Example output:

ocs-deviceset-localblock-0-data-0-q4q6b   Bound    local-pv-8137c873       256Gi     RWO         localblock         10m
ocs-deviceset-localblock-1-data-0-hr2fx   Bound    local-pv-ec7f2b80       256Gi     RWO         localblock         1d20h
ocs-deviceset-localblock-2-data-0-6xhkf   Bound    local-pv-1e31f771       256Gi     RWO         localblock         1d20h

If cluster-wide encryption is enabled on the cluster, verify that the new OSD devices are encrypted.
1. Identify the nodes where the new OSD pods are running, where <OSD-pod-name>is the name of the OSD pod.
```
oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/<OSD-pod-name>
```
  For example:
```
oc get -n openshift-storage -o=custom-columns=NODE:.spec.nodeName pod/rook-ceph-osd-0-544db49d7f-qrgqm
```
  Example output:
```
NODE
compute-1
```
2. For each of the previously identified nodes, do the following:
  1. Create a debug pod and open a chroot environment for the selected host(s), where <node name> is the name of the node.
```
oc debug node/<node name>
```
```
$ chroot /host
```
  2. Check for the crypt keyword beside the ocs-deviceset name(s).
```
$ lsblk
```
Log in to OpenShift Web Console and view the storage dashboard.

Note: A full data recovery may take longer depending on the volume of data being recovered.