Upgrading IBM Storage Scale container native

The following section describes how to upgrade the IBM Storage Scale container native cluster.

While an upgrade is in progress, do not perform the following:

Do not make changes to the Cluster custom resource.
Do not attempt to add a node to the cluster.

During an upgrade, the IBM Storage Scale operator orchestrates the upgrade procedure in a rolling node-by-node fashion. Each node will be:

Cordoned (tainted unschedulable)
Drained (pods are safely evicted and rescheduled to other available nodes)
Rebooted, if necessary
Uncordoned (returning it to normal service)

After the node is schedulable, IBM Storage Scale container native and IBM Storage Scale Container Storage Interface (CSI) pods will start. Applications may fail to attach storage until the system is started.

Prerequisites

All the core pods need to be in running status.

Use the following command to check the status of the core pods:
```
 oc get daemons ibm-spectrum-scale -n ibm-spectrum-scale -ojson | jq -r '.status.podsStatus'
```
Help ensure that there is no pods in any of the following states:
- starting
- terminating
- unknown
- waitingForDelete
  
  In the following example, the output shows 1 pod in "waitingForDelete", so the upgrade should not be done at this time.
```
$ oc get daemons ibm-spectrum-scale -n ibm-spectrum-scale -ojson | jq -r '.status.podsStatus'
{
"running": "4",
"starting": "0",
"terminating": "0",
"unknown": "0",
"waitingForDelete": "1"
}
```

Upgrade steps

Complete the following steps to upgrade:

Stop the running operator pod by setting the replicas in the deployment to 0.

oc scale deployment ibm-spectrum-scale-controller-manager -n ibm-spectrum-scale-operator --replicas=0

Delete the old security context constraint.

oc delete scc ibm-spectrum-scale-privileged

Delete the old role binding for privilege.

oc delete rolebinding -n ibm-spectrum-scale ibm-spectrum-scale-privileged --ignore-not-found

Delete the MutatingWebhookConfiguration and ValidatingWebhookConfiguration. These will be created in later steps.

oc delete MutatingWebhookConfiguration ibm-spectrum-scale-mutating-webhook-configuration
oc delete ValidatingWebhookConfiguration ibm-spectrum-scale-validating-webhook-configuration

The imagePullSecret ibm-spectrum-scale-registrykey in the service accounts have been deprecated. Delete the service accounts before applying the install.yaml as they will be created again.
```
for namespace in ibm-spectrum-scale ibm-spectrum-scale-operator ibm-spectrum-scale-dns ibm-spectrum-scale-csi; do
   oc delete sa --all -n ${namespace}
done
```
Configure namespace pull secrets.

Starting with Red Hat OpenShift 4.15 (kubernetes 1.28), warning messages appear when secrets are referenced in service account but are not created in the namespace. Configuring ICR entitlement using the Red Hat OpenShift global pull secret result in these messages appearing in the events: Unable to retrieve some image pull secrets.

For instructions on creating the namespace pull secrets, see Namespace Pull Secrets.

For major version upgrades, consider taking this opportunity to switch to namespace pull secrets before intializing the upgrade. This allows the operator to orchestrate all pod restarts.

After namespace secrets are created, the pods need to restart for the warning messages to quiesce. For core pods only, use the following annotation to have the operator orchestrate the pod restarts, preserving quorum: oc annotate pod -lapp.kubernetes.io/name=core scale.spectrum.ibm.com/pending=delete. For other pods, you must delete them manually.

Apply the new manifests.

oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.2.0.x/generated/scale/install.yaml

Verification

After the new IBM Storage Scale container native operator is deployed, the upgrade process will begin. It takes some time to complete as the new container images are rolled out into the cluster.

Validate the app.kubernetes.io/version on the operator deployment:

oc get deployment ibm-spectrum-scale-controller-manager \
-n ibm-spectrum-scale-operator  -ojson | jq -r .metadata.labels

To check the progress of the pod restarts and node reboots, query the daemon CR using the following command:

oc describe daemon ibm-spectrum-scale  -n ibm-spectrum-scale

Information is available under the "Status Details" and "Events" sections.

Code version updated

The version details are listed under .status.versions in the Daemon CR and is updated as the pods roll. The following command shows the versions that core pods currently have on them. Wait until all the pods are reporting the same new version.

oc get daemon ibm-spectrum-scale -n ibm-spectrum-scale -ojson  | jq -r .status.versions