Post upgrade tasks

The following section lists optional tasks for the administrator to consider when upgrading the IBM Storage Scale container native product to the new code levels.

Enable Network Policies (Red Hat Openshift cluster ONLY)

Network Policies is not supported on Kubernetes and GKE cluster

Starting at version v5.1.9, the operator can create network policies rules across the managed namespaces. If upgrading from an earlier version, the network policy configuration is not modified on upgrade. Use the following command to check the network policy configuration:

kubectl get cluster ibm-spectrum-scale -ojson | jq -r .spec.networkPolicy

If the response is {}, network policy is enabled, you are done.

If the response is null and you want to enable the operator to create network policy rules, patch the cluster by using the following command:

kubectl patch cluster.scale.spectrum.ibm.com ibm-spectrum-scale --type='json' \
-p='[{"op": "add", "path": "/spec/networkPolicy", "value":{}}]'

For more information, see Network policies.

Approve the new release level

It is suggested to first use the IBM Storage Scale container native cluster with the upgrade code before permanently setting the cluster to the new level.

For more information, see File system format changes between versions of IBM Storage Scale in the IBM Storage Scale documentation.

When a release level change is detected after an upgrade, the operator will create an UpgradeApproval resource. When you are ready to enable the new functions and lock in the new level, complete the following steps:

Upgrades within the .x version of a release typically do not have an upgrade approval.

Check to see whether any cluster upgrade approvals are present that require action.

An upgrade approval that shows nothing under the COMPLETED field are ones that require some action.
```
kubectl get upgradeapproval -n ibm-spectrum-scale
```
If an upgrade approval does not appear, check the Daemon CR status to validate that all pods are on the new version by using: kubectl get daemon -n ibm-spectrum-scale -ojson | jq -r .items[].status.versions

Check the minReleaseLevel of the cluster:

kubectl exec $(kubectl get pods -lapp.kubernetes.io/name=core -ojsonpath="{.items[0].metadata.name}" \
-n ibm-spectrum-scale) -c gpfs -n ibm-spectrum-scale -- mmlsconfig release

To approve the upgrade approval job, run the following command:

kubectl patch upgradeapproval <upgradeapproval-name> -n ibm-spectrum-scale --type='json' \
-p='[{"op": "replace", "path": "/spec/approved", "value":true}]'

Full Example:

# Check for any upgrade approvals for TYPE=cluster
$ kubectl get upgradeapproval -n ibm-spectrum-scale
NAME            TYPE      FILESYSTEM   LAST SCHEDULE TIME   LAST SUCCESSFUL TIME   RUNNING   COMPLETED
upgrade-rmlp4   cluster

# check the daemon status for the versions deployed on each core pod
$ kubectl get daemon -n ibm-spectrum-scale -ojson | jq  .items[].status.versions
[
  {
    "count": "3",
    "version": "5.1.9.1"
  }
]

# check the current cluster release version
$ kubectl exec $(kubectl get pods -lapp.kubernetes.io/name=core -ojsonpath="{.items[0].metadata.name}" -n ibm-spectrum-scale) -c gpfs -n ibm-spectrum-scale -- mmlsconfig release
minReleaseLevel 5.1.5.0

# patch the upgrade approval
$ kubectl patch upgradeapproval upgrade-rmlp4  -n ibm-spectrum-scale \
> --type='json'  -p='[{"op": "replace", "path": "/spec/approved", "value":true}]'
upgradeapproval.scale.spectrum.ibm.com/upgrade-rmlp4 patched

# query the upgrade approval to see it running
$ kubectl get upgradeapproval -n ibm-spectrum-scale
NAME            TYPE      FILESYSTEM   LAST SCHEDULE TIME   LAST SUCCESSFUL TIME   RUNNING                                                 COMPLETED
upgrade-rmlp4   cluster                23s                                         ibm-spectrum-scale/worker2/gpfs/upgradeCluster_p6Jhz9

# upgrade approval job completed
$ kubectl get upgradeapproval -n ibm-spectrum-scale
NAME            TYPE      FILESYSTEM   LAST SCHEDULE TIME   LAST SUCCESSFUL TIME   RUNNING   COMPLETED
upgrade-rmlp4   cluster                38s                  6s                               Successful

# verify that the cluster release level has been updated
$ kubectl exec $(kubectl get pods -lapp.kubernetes.io/name=core -ojsonpath="{.items[0].metadata.name}" -n ibm-spectrum-scale) -c gpfs -n ibm-spectrum-scale -- mmlsconfig release
minReleaseLevel 5.1.9.1

For more information, see Completing the upgrade to a new level of IBM Storage Scale in IBM Storage Scale documentation.

Remote storage cluster considerations

If the storage cluster is being mounted by other GPFS client clusters that are running a prior version of code, performing the following step results in these client clusters being unable to mount file systems from this storage cluster. See Software requirements for minimum storage cluster level support.

The storage cluster supports earlier versions than IBM Storage Scale container native client cluster. CSI functions are highly dependent upon the release, file system level, and version of IBM Storage Scale installed on the storage cluster. If the storage cluster is at an earlier version, some functions may not be available. For more information about the minimum levels required for specific CSI functions, see Hardware and Software Requirements in the IBM Storage Scale CSI documentation. For more information about compatibility and software matrix, see Section 17.3 in the IBM Storage Scale FAQ documentation.

Run the following command for each file system to upgrade to the latest metadata format available on the storage cluster:

This step is optional but recommended for enabling the functions that are provided at the latest levels of code.
```
mmchfs <Filesystem> -V full
```

Auto inode limit

Enable auto-inode-limit of the file system.

The --auto-inode-limit option is available only at file system format level of 28.00 or later. Enable this option when the file system is updated to 28.00 or later.
```
mmchfs <Filesystem> --auto-inode-limit
```

Remove deprecated resources

The following section provides steps to clean up deprecated resources.

Run the following commands to clean up deprecated resources, if they exist.

kubectl delete ClusterRoleBinding ibm-spectrum-scale-dns
kubectl delete ClusterRole ibm-spectrum-scale-dns

Remove deprecated CSI user secrets

The CSI user secrets are no longer needed if the storage clusters are running IBM Storage Scale 5.2.3.0 or newer.

Run the following steps to delete the deprecated CSI user secrets if the storage clusters are running IBM Storage Scale 5.2.3.0 or newer:

To list the names of CSI user secrets, run this command:

kubectl get remotecluster -n ibm-spectrum-scale -o json | grep csiSecretName

Delete the listed secrets in the ibm-spectrum-scale-csi namespace, for example:

kubectl delete secret csi-remote-mount-storage-cluster-1 -n ibm-spectrum-scale-csi

Delete the csiSecretName parameter from the spec of the RemoteCluster custom resources.
```
kubectl edit remotecluster -n ibm-spectrum-scale
```

For more information, see Storage cluster v5.2.3.0 and newer.