Cleanup Red Hat OpenShift nodes

IBM Storage Scale container native requires host path volume mounts and creates persistent directories and files on each of the Red Hat OpenShift nodes where core pods are deployed. Removal of these persistent files are needed during clean up and before reinstallation of the cluster.

Complete the following steps to remove the persistent files on the Red Hat OpenShift nodes.

1. Switch to the default project.

   oc project default
   

2. Enter the following command to list the nodes.

   oc get nodes -o jsonpath="{range .items[*]}{.metadata.name}{'\n'}"
   

   Use the nodeSelector to reduce the set of nodes to cleanup, or run cleanup against all the nodes.

3. From the list of nodes, use the oc debug node command to create a debug pod accessing the host and allowing for the ability to remove the kernel modules and the host path volume-mounted directories that are used by IBM Storage Scale container native:

   oc debug node/<openshift_node> -T -- chroot /host sh -c "rm -rf /var/mmfs; rm -rf /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   

   Example:

   oc debug node/worker0.example.com -T -- chroot /host sh -c "rm -rf /var/mmfs; rm -rf /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   Starting pod/worker0examplecom-debug ...
   To use host binaries, run `chroot /host`
   Removing debug pod ...
   

4. Check that none of the artifacts remain by entering the following validation command:

   oc debug node/<openshift_node> -T -- chroot /host sh -c "ls /var/mmfs; ls /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   

   Example:

   oc debug node/worker0.example.com -T -- chroot /host sh -c "ls /var/mmfs; ls /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   Starting pod/worker0examplecom-debug ...
   To use host binaries, run `chroot /host`
   ls: cannot access '/var/mmfs': No such file or directory
   ls: cannot access '/var/adm/ras': No such file or directory
   rmmod: ERROR: Module tracedev is not currently loaded
   rmmod: ERROR: Module mmfs26 is not currently loaded
   rmmod: ERROR: Module mmfslinux is not currently loaded
   Removing debug pod ...
   error: non-zero exit code from debug container
   

5. Remove any node labels that are created by the IBM Storage Scale container native operator:

   # clean up all 'scale.spectrum.ibm.com' labels on the nodes
   oc get nodes -ojson | jq -r '.items[].metadata.labels'  | \
   grep scale.spectrum.ibm.com | sort | cut -d: -f1 | \
   tr -d "\""| uniq | xargs -I{} oc label nodes --all {}\-
   
   # clean up 'scale' label on the nodes
   oc label node --all scale-