Shutting down and restarting IBM Fusion HCI rack with Data Foundation storage

Procedure to gracefully restart the IBM Fusion rack with Fusion Data Foundation storage.

1. Capture health check report before bringing down the rack. It helps to check for any preexisting issues post power-on.
   Note: The health report must be saved to a different system.

   Fusion Data Foundation

   For Fusion Data Foundation storage, run the following OC commands to get the system health status before shutdown:

   1. Check whether all cluster operators are Available and not in Degraded state. To verify, run the following commands:

      oc get co

   2. Check whether any updates are Progressing on the cluster. To verify, run the following command:

      oc get clusterversion

   3. Check that the Fusion Data Foundation cluster is in a healthy state.
      1. In the Red Hat® OpenShift web console, click Storage > Data Foundation.
      2. In the Status card of the Overview tab, clickStorage system.
      3. In the notification window, click the Storage system link.
      4. In the Status card of theBlock and File tab, verify that the storage cluster is in a healthy state.
      5. In the Details card, verify the cluster information.

2. Check whether there exists any active Backup & Restore jobs. If Backup & Restore or application synch is in progress, then wait for them to complete. Wait for in progress workload operations to complete. Before you proceed with the shutdown of the storage cluster, ensure that no data is in progress for any job or application.
3. Run the following storage commands to shut down storage cluster.

   Fusion Data Foundation

   For Fusion Data Foundation nodes with status Running, stop the applications that consume storage from the Fusion Data Foundation to stop the I/O. Hence, the shutdown sequence is important to avoid any data corruption and pod failures. The shutdown of the application components must to be done before you shut down the nodes. Scale down the application deployments so that the application pods do not get started on other nodes in case node selectors are not set. Also, this must stop storage I/O from the applications.

4. Place the IBM Data Cataloging service in an idle state on the Red Hat OpenShift environment. For more information about the procedure, see Graceful shutdown.
5. Shut down the Red Hat OpenShift Container Platform cluster.
   1. If the cluster-wide proxy is enabled, be sure to export the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables. To check if the proxy is enabled, run oc get proxy cluster -o yaml.
   2. Take etcd backup.

      oc debug node/<node_name> (any one control node)

      sh-4.15# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

   3. Copy the etcd backup to external system.

      snapshot_.db and static_kuberesources_.tar.gz

      You can use the oc rsync command to copy the files to an external system. You need two terminals for this operation.

      1. Open terminal one.
      2. Run the following commands for etcd backup:

         
         oc debug node/<node_name> 
         sh-4.15# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

         In oc debug node/<node_name> command, use any one control node.
      3. Run the following command and record the new pod name:

         It is the source pod, and the backup files reside inside the pod.

         oc debug

         Do not close the terminal 1.
      4. Open terminal two and run the following command to copy the file to the local folder:

         oc -n <namespace_of_debug_pod> rsync <source_podname_in_above_step>:/home/core/assets/backup/snapshot_.db <local_folder_path> 

         If required, add the namespace of the debug node pod location.

      5. Repeat the step ii to copy another backup file to the external system.
      6. Close the terminal windows after all the files are copied.

      For more information about the procedure, see Copy local files to or from a remote directory.
   4. Ensure that you take off the workloads before you shut down the nodes.
   5. Run the following commands to shut down the nodes:
      The node with OpenShift Container Platform user interface and IBM Fusion user interface must be the last node you must power off.
      For Fusion Data Foundation, you must first shut down the application nodes and then shutdown the Fusion Data Foundation nodes. Finally, shutdown the OpenShift control plane nodes.

      
      for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}');
      do oc debug node/${node} -- chroot /host shutdown -h 1;
      done

      After 3 to 5 minutes, the Red Hat OpenShift Container Platform becomes inaccessible.

      This step brings down all the software on the rack. The rack is ready to be powered off.

6. Physically press the power off button of the nodes and the rack.
   Note: The switches do not have the option to shutdown, and they can only be rebooted. When you power off the entire rack (unplugged), the switches shut down automatically. Similarly, when the power is restored to the rack, the switches comes up automatically.
7. Power on the rack.
   1. Power on the rack.
   2. Go to the physical node and click the power button to power on all the nodes.
      Power on all control nodes. After all control nodes are up, power on compute nodes.

      For Fusion Data Foundation, power on the nodes in this sequence:

      1. Start the Control Plane nodes.
      2. Start the Fusion Data Foundation nodes.
      3. Verify that these pods are in a running state:

         oc get pods -n openshift-storage

         It is possible that all Fusion Data Foundation pods may not be Running by now, but it is expected to have rook-ceph-{mds-*,mon-*,mgr-*,osd-*} pods to be running to serve storage. If the Ceph cluster is not in a good shape, it results in a FailedMounts with infra and application pods.
      4. Start the infra nodes.
      5. Start the application nodes.
   3. After all the nodes are up and cluster operators are up (except image registry), run the following commands to ensure that the OpenShift cluster is up along with the IBM Fusion operators.

      oc get po -A | grep -v Running | grep -v Completed
      oc get co  
      oc get nodes

   4. For Fusion Data Foundation, scale up the application deployments.
      Give it a few minutes and check the cluster or storage dashboard.
   5. Run the following commands to ensure that the storage pods are up:

      Fusion Data Foundation

      Run the following command to validate all pods are up in the openshift-{storage,monitoring,logging,image-registry} and the application namespaces:

      oc get pods -n <NAMESPACE>

8. Bring back IBM Data Cataloging to a running state.
   For the procedure, see Returning IBM Data Cataloging to a running state.