Cleanup Red Hat OpenShift nodes

Complete the following steps to remove the persistent files on the Red Hat OpenShift nodes.

1. Switch to the default project.

    oc project default
   

2. Enter the following command to list the nodes.

   kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name}{'\n'}"
   

   Use the nodeSelector to reduce the set of nodes to cleanup, or run cleanup against all the nodes.

3. From the list of nodes, use the oc debug node command to create a debug pod accessing the host and allowing for the ability to remove the kernel modules and the host path volume-mounted directories that are used by IBM Storage Scale container native:

   oc debug node/<openshift_node> -T -- chroot /host sh -c "rm -rf /var/mmfs; rm -rf /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   

   Example:

   oc debug node/worker0.example.com -T -- chroot /host sh -c "rm -rf /var/mmfs; rm -rf /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   Starting pod/worker0examplecom-debug ...
   To use host binaries, run `chroot /host`
   Removing debug pod ...
   

4. Check that none of the artifacts remain by entering the following validation command:

   oc debug node/<openshift_node> -T -- chroot /host sh -c "ls /var/mmfs; ls /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   

   Example:

   oc debug node/worker0.example.com -T -- chroot /host sh -c "ls /var/mmfs; ls /var/adm/ras; rmmod tracedev mmfs26 mmfslinux;"
   Starting pod/worker0examplecom-debug ...
   To use host binaries, run `chroot /host`
   ls: cannot access '/var/mmfs': No such file or directory
   ls: cannot access '/var/adm/ras': No such file or directory
   rmmod: ERROR: Module tracedev is not currently loaded
   rmmod: ERROR: Module mmfs26 is not currently loaded
   rmmod: ERROR: Module mmfslinux is not currently loaded
   Removing debug pod ...
   error: non-zero exit code from debug container
   

5. Remove any node labels that are created by the IBM Storage Scale container native operator:

   # clean up all 'scale.spectrum.ibm.com' labels on the nodes
   kubectl get nodes -ojson | jq -r '.items[].metadata.labels'  | \
   grep scale.spectrum.ibm.com | sort | cut -d: -f1 | \
   tr -d "\""| uniq | xargs -I{} kubectl label nodes --all {}\-
   
   # clean up 'scale' label on the nodes
   kubectl label node --all scale-