Restoring the monitor pods in Fusion Data Foundation

Use this information to manually restore the monitor pods Fusion Data Foundation, when necessary.

1. Scale down the rook-ceph-operator and ocs-operator.

   oc scale deployment rook-ceph-operator --replicas=0 -n openshift-storage

   oc scale deployment ocs-operator --replicas=0 -n openshift-storage

2. Create a backup of all deployments in openshift-storage namespace.

   mkdir backup

   cd backup

   oc project openshift-storage

   for d in $(oc get deployment|awk -F' ' '{print $1}'|grep -v NAME); do echo $d;oc get deployment $d -o yaml > oc_get_deployment.${d}.yaml; done

3. Patch the MDS deployments to remove the livenessProbe parameter and run it with the command parameter as sleep.

   for i in $(oc get deployment -l app=rook-ceph-osd -oname);do oc patch ${i} -n openshift-storage --type='json' -p '[{"op":"remove", "path":"/spec/template/spec/containers/0/livenessProbe"}]' ; oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "osd", "command": ["sleep", "infinity"], "args": []}]}}}}' ; done

4. Retrieve the monstore cluster map from all the OSDs.
   1. Create the recover_mon.sh script.

      !/bin/bash
      ms=/tmp/monstore
      
      rm -rf $ms
      mkdir $ms
      
      for osd_pod in $(oc get po -l app=rook-ceph-osd -oname -n openshift-storage); do
      
        echo "Starting with pod: $osd_pod"
      
        podname=$(echo $osd_pod|sed 's/pod\///g')
        oc exec $osd_pod -- rm -rf $ms
        oc cp $ms $podname:$ms
      
        rm -rf $ms
        mkdir $ms
      
        echo "pod in loop: $osd_pod ; done deleting local dirs"
      
        oc exec $osd_pod -- ceph-objectstore-tool --type bluestore --data-path /var/lib/ceph/osd/ceph-$(oc get $osd_pod -ojsonpath='{ .metadata.labels.ceph_daemon_id }') --op update-mon-db --no-mon-config --mon-store-path $ms
        echo "Done with COT on pod: $osd_pod"
      
        oc cp $podname:$ms $ms
      
        echo "Finished pulling COT data from pod: $osd_pod"
      done

   2. Run the recover_mon.sh script.

      chmod +x recover_mon.sh

      ./recover_mon.sh

5. Patch the MON deployments, and run it with the command parameter as sleep.
   1. Edit the MON deployments.

      for i in $(oc get deployment -l app=rook-ceph-mon -oname);do oc patch ${i} -n openshift-storage -p '{"spec": {"template": {"spec": {"containers": [{"name": "mon", "command": ["sleep", "infinity"], "args": []}]}}}}'; done

   2. Patch the MON deployments to increase the initialDelaySeconds.

      oc get deployment rook-ceph-mon-a -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      oc get deployment rook-ceph-mon-b -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

      oc get deployment rook-ceph-mon-c -o yaml | sed "s/initialDelaySeconds: 10/initialDelaySeconds: 2000/g" | oc replace -f -

6. Copy the previously retrieved monstore to the mon-a pod.

   oc cp /tmp/monstore/ $(oc get po -l app=rook-ceph-mon,mon=a -oname |sed 's/pod\///g'):/tmp/

7. Navigate into the MON pod and change the ownership of the retrieved monstore.

   oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

    chown -R ceph:ceph /tmp/monstore

8. Copy the keyring template file before rebuilding the mon db.

   oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

    cp /etc/ceph/keyring-store/keyring /tmp/keyring

    cat /tmp/keyring
     [mon.]
       key = AQCleqldWqm5IhAAgZQbEzoShkZV42RiQVffnA==
       caps mon = "allow *"
     [client.admin]
       key = AQCmAKld8J05KxAArOWeRAw63gAwwZO5o75ZNQ==
       auid = 0
       caps mds = "allow *"
       caps mgr = "allow *"
       caps mon = "allow *"
       caps osd = "allow *”

9. Identify the keyring of all other Ceph daemons (MGR, MDS, RGW, Crash, CSI and CSI provisioners) from its respective secrets.

   oc get secret rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-keyring -ojson  | jq .data.keyring | xargs echo | base64 -d
   
   [mds.ocs-storagecluster-cephfilesystem-a]
   key = AQB3r8VgAtr6OhAAVhhXpNKqRTuEVdRoxG4uRA==
   caps mon = "allow profile mds"
   caps osd = "allow *"
   caps mds = "allow"

   Example keyring file, /etc/ceph/ceph.client.admin.keyring:

   [mon.]
           key = AQDxTF1hNgLTNxAAi51cCojs01b4I5E6v2H8Uw==
           caps mon = "allow "
   [client.admin]
           key = AQDxTF1hpzguOxAA0sS8nN4udoO35OEbt3bqMQ==
           caps mds = "allow "
           caps mgr = "allow *"
           caps mon = "allow *"
           caps osd = "allow *"
   [mds.ocs-storagecluster-cephfilesystem-a]
           key = AQCKTV1horgjARAA8aF/BDh/4+eG4RCNBCl+aw==
           caps mds = "allow"
           caps mon = "allow profile mds"
           caps osd = "allow *"
   [mds.ocs-storagecluster-cephfilesystem-b]
           key = AQCKTV1hN4gKLBAA5emIVq3ncV7AMEM1c1RmGA==
           caps mds = "allow"
           caps mon = "allow profile mds"
           caps osd = "allow *"
   [client.rgw.ocs.storagecluster.cephobjectstore.a]
           key = AQCOkdBixmpiAxAA4X7zjn6SGTI9c1MBflszYA==
           caps mon = "allow rw"
           caps osd = "allow rwx"
   [mgr.a]
           key = AQBOTV1hGYOEORAA87471+eIZLZtptfkcHvTRg==
           caps mds = "allow *"
           caps mon = "allow profile mgr"
           caps osd = "allow *"
   [client.crash]
           key = AQBOTV1htO1aGRAAe2MPYcGdiAT+Oo4CNPSF1g==
           caps mgr = "allow rw"
           caps mon = "allow profile crash"
   [client.csi-cephfs-node]
           key = AQBOTV1hiAtuBBAAaPPBVgh1AqZJlDeHWdoFLw==
           caps mds = "allow rw"
           caps mgr = "allow rw"
           caps mon = "allow r"
           caps osd = "allow rw tag cephfs *="
   [client.csi-cephfs-provisioner]
           key = AQBNTV1hHu6wMBAAzNXZv36aZJuE1iz7S7GfeQ==
           caps mgr = "allow rw"
           caps mon = "allow r"
           caps osd = "allow rw tag cephfs metadata="
   [client.csi-rbd-node]
           key = AQBNTV1h+LnkIRAAWnpIN9bUAmSHOvJ0EJXHRw==
           caps mgr = "allow rw"
           caps mon = "profile rbd"
           caps osd = "profile rbd"
   [client.csi-rbd-provisioner]
           key = AQBNTV1hMNcsExAAvA3gHB2qaY33LOdWCvHG/A==
           caps mgr = "allow rw"
           caps mon = "profile rbd"
           caps osd = "profile rbd"

   Important:

   • For client.csi related keyring, refer to the previous keyring file output and add the default caps after fetching the key from its respective Fusion Data Foundation secret.
   • OSD keyring is added automatically post recovery.
10. Navigate into the mon-a pod, and verify that the monstore has monmap.
    1. Navigate into the mon-a pod.

       oc rsh $(oc get po -l app=rook-ceph-mon,mon=a -oname)

    2. Verify that the monstore has monmap.

        ceph-monstore-tool /tmp/monstore get monmap -- --out /tmp/monmap

        monmaptool /tmp/monmap --print

11. Optional: If the monmap is missing then create a new monmap.

     monmaptool --create --add <mon-a-id><mon-a-ip> --add <mon-b-id><mon-b-ip> --add <mon-c-id><mon-c-ip> --enable-all-features --clobber /root/monmap --fsid <fsid>

    <mon-a-id>

    Is the ID of the mon-a pod.

    <mon-a-ip>

    Is the IP address of the mon-a pod.

    <mon-b-id>

    Is the ID of the mon-b pod.

    <mon-b-ip>

    Is the IP address of the mon-b pod.

    <mon-c-id>

    Is the ID of the mon-c pod.

    <mon-c-ip>

    Is the IP address of the mon-c pod.

    <fsid>

    Is the file system ID.

12. Verify the monmap.

     monmaptool /root/monmap --print

13. Import the monmap.
    Important: Use the previously created keyring file.

     ceph-monstore-tool /tmp/monstore rebuild -- --keyring /tmp/keyring --monmap /root/monmap

     chown -R ceph:ceph /tmp/monstore

14. Create a backup of the old store.db file.

    mv /var/lib/ceph/mon/ceph-a/store.db /var/lib/ceph/mon/ceph-a/store.db.corrupted

    mv /var/lib/ceph/mon/ceph-b/store.db /var/lib/ceph/mon/ceph-b/store.db.corrupted

    mv /var/lib/ceph/mon/ceph-c/store.db /var/lib/ceph/mon/ceph-c/store.db.corrupted

15. Copy the rebuild store.db file to the monstore directory.

    mv /tmp/monstore/store.db /var/lib/ceph/mon/ceph-a/store.db

    chown -R ceph:ceph /var/lib/ceph/mon/ceph-a/store.db

16. After rebuilding the monstore directory, copy the store.db file from local to the rest of the MON pods, where <id> is the ID of the MON pod.

    oc cp $(oc get po -l app=rook-ceph-mon,mon=a -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-a/store.db /tmp/store.db

    oc cp /tmp/store.db $(oc get po -l app=rook-ceph-mon,mon=<id> -oname | sed 's/pod\///g'):/var/lib/ceph/mon/ceph-<id>

17. Navigate into the rest of the MON pods and change the ownership of the copied monstore, where <id> is the ID of the MON pod.

    oc rsh $(oc get po -l app=rook-ceph-mon,mon=<id> -oname)

    chown -R ceph:ceph /var/lib/ceph/mon/ceph-<id>/store.db

18. Revert the patched changes.
    Important: Ensure that the MON, MGR, and OSD pods are up and running.

    For MON deployments

    Use the following command, where <mon-deployment.yaml> is the MON deployment yaml file.

    oc replace --force -f <mon-deployment.yaml>

    For OSD deployments:

    Use the following command, where <osd-deployment.yaml> is the OSD deployment yaml file.

    oc replace --force -f <osd-deployment.yaml>

    For MGR deployments

    Use the following command, where <mgr-deployment.yaml> is the MGR deployment yaml file.

    oc replace --force -f <mgr-deployment.yaml>

19. Scale up the rook-ceph-operator and ocs-operator deployments.

    oc -n openshift-storage scale deployment ocs-operator --replicas=1