Shutting down and restarting IBM Fusion HCI rack with Data Foundation storage

Procedure to gracefully restart the IBM Fusion rack with Fusion Data Foundation storage.

1. Capture system health check report before bringing down the rack. It helps to check for any preexisting issues post power-on.
   1. Ensure no machine config or update is in progress or no node is not ready
   2. To verify, run the following commands:

      oc get co

      oc get clusterversion

   3. Run the following commands to list the pods, cluster operators, and nodes.

      oc get po -A | grep -v Running | grep -v Completed
      oc get nodes

   4. Check that the Fusion Data Foundation cluster is in a healthy state.
      1. In the Red Hat® OpenShift web console, click Storage > Data Foundation.
      2. In the Status card of the Overview tab, clickStorage system.
      3. In the notification window, click the Storage system link.
      4. In the Status card of theBlock and File tab, verify that the storage cluster is in a healthy state.
      5. In the Details card, verify the cluster information.

   Note: The health check must be saved to a different system.
2. Check whether there exists any active Backup & Restore jobs. If Backup & Restore or application synch is in progress, then wait for them to complete. Wait for in progress workload operations to complete. Before you proceed with the shutdown of the storage cluster, ensure that no data is in progress for any job or application.
3. Stop the applications that consume storage from the Fusion Data Foundation to stop the I/O.
   The shutdown sequence is important to avoid any data corruption and pod failures. The shutdown of the application components must to be done before you shut down the nodes. Scale down the application deployments so that the application pods do not get started on other nodes in case node selectors are not set. Also, this must stop storage I/O from the applications.
4. If you have enabled IBM Data Cataloging, then place the service in an idle state on the Red Hat OpenShift environment. For more information about the shut down procedure in IBM Data Cataloging, see Graceful shutdown.
5. Shut down the Red Hat OpenShift Container Platform cluster.
   1. If the cluster-wide proxy is enabled, be sure to export the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables, on bastion node from where you intend to run oc commands. To check whether the proxy is enabled run below command:

      oc get proxy cluster -o yaml

   2. Take etcd backup.

      oc debug node/<node_name> (any one control node)

      sh-4.15# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

   3. Copy the etcd backup to external system.

      snapshot_.db and static_kuberesources_.tar.gz

      You can use the oc rsync command to copy the files to an external system. You need two terminals for this operation.

      1. Open terminal one.
      2. Run the following commands for etcd backup:

         
         oc debug node/<node_name> 
         sh-4.15# /usr/local/bin/cluster-backup.sh /home/core/assets/backup

         In oc debug node/<node_name> command, use any one control node.
      3. Run the following command and record the new pod name:

         It is the source pod, and the backup files reside inside the pod.

         oc debug

         Do not close the terminal 1.
      4. Open terminal two and run the following command to copy the file to the local folder:

         oc -n <namespace_of_debug_pod> rsync <source_podname_in_above_step>:/home/core/assets/backup/snapshot_.db <local_folder_path> 

         If required, add the namespace of the debug node pod location.

      5. Repeat the step ii to copy another backup file to the external system.
      6. Close the terminal windows after all the files are copied.

      For more information about the procedure, see Copy local files to or from a remote directory.
   4. Ensure that you take off the workloads before you shut down the nodes.
   5. Run the following commands to shut down the nodes:

      Ensure that the control node hosting the IBM Fusion operators is powered off at the end. Shutting down this node prematurely results in loss of access to both the IBM Fusion and OpenShift Container Platform user interfaces.

      For Fusion Data Foundation, you must first shut down the application nodes and then shutdown the Fusion Data Foundation nodes.

      for i in `oc get nodes  |grep  compute|awk '{print $1}'`; do oc debug nodes/$i -- chroot /host shutdown -h 1 ; done

      Finally, shutdown the OpenShift control plane nodes.

      for i in `oc get nodes  |grep -v compute|grep -v NAME|awk '{print $1}'`; do oc debug nodes/$i -- chroot /host shutdown -h 1 ; done

      After 3 to 5 minutes, the Red Hat OpenShift Container Platform becomes inaccessible.

      This step brings down all the software on the rack. The rack is ready to be powered off.

6. After all the nodes are powered off, physically press the power off button of the nodes.
   Note:

   • This physical power off indicates to the Baseboard Management Controller (BMC) that you intend to keep the node powered down and prevents automatic restart.
   • The switches do not have the option to shutdown, and they can only be rebooted. When you power off the entire rack (unplugged), the switches shut down automatically. Similarly, when the power is restored to the rack, the switches comes up automatically.
7. Power on the rack.
   1. Power on the rack.
   2. Go to the physical node and click the power button to power on all the nodes.
      Power on all control nodes. After all control nodes are up, power on compute nodes.

      For Fusion Data Foundation, power on the nodes in this sequence:

      1. Start the Control Plane nodes.
      2. Start the Fusion Data Foundation nodes.
      3. Verify that these pods are in a running state:

         oc get pods -n openshift-storage

         It is possible that all Fusion Data Foundation pods may not be Running by now, but it is expected to have rook-ceph-{mds-*,mon-*,mgr-*,osd-*} pods to be running to serve storage. If the Ceph cluster is not in a good shape, it results in a FailedMounts with infra and application pods.
      4. Start the application nodes.
   3. After all the nodes are up and cluster operators are up (except image registry), run the following commands to ensure that the OpenShift cluster is up along with the IBM Fusion operators.

      oc get po -A | grep -v Running | grep -v Completed
      oc get co  
      oc get nodes

   4. For Fusion Data Foundation, scale up the application deployments.
      Give it a few minutes and check the cluster or storage dashboard.
   5. Run the following commands to ensure that the storage pods are up:

      Fusion Data Foundation

      Run the following command to validate all pods are up in the openshift-{storage,monitoring,logging,image-registry} and the application namespaces:

      oc get pods -n <NAMESPACE>

8. Bring back IBM Data Cataloging to a running state.
   For the procedure, see Returning IBM Data Cataloging to a running state.