Red Hat OpenShift Container Platform

IBM Cloud Container Registry (ICR) pull secrets

IBM Storage Scale Container Native images are hosted in the IBM Cloud Container Registry. For images to be properly pulled at the pod level, the Red Hat OpenShift global pull secrets must be modified to contain credentials to access the IBM Cloud Container Registry. Obtain an entitlement key from IBM Container software library . Entitlement keys determine whether the IBM Storage Scale operator can automatically pull the required IBM Storage Scale Container Native images. During installation, image pull failures may occur due to an invalid entitlement key or a key belonging to an account that does not have entitlement to either IBM Storage Scale Data Access Edition or IBM Storage Scale Data Management Edition. It is therefore important to generate a key from an account that already has entitlement to the desired edition of IBM Storage Scale software.

For using IBM Cloud Container Registry, the credentials must use the cp user along with the entitlement key.
```
$ echo -n "cp:<ADD_YOUR_ENTITLEMENT_KEY>" | base64 -w0
```
Create an authority.json to include the base64 encoded string of your credentials, the fixed username cp, and the entitlement key for the IBM Cloud Container Registry.
```
{
  "auth": "ADD_BASE64_ENCODED_ENTITLEMENT_KEY_FROM_PREVIOUS_STEP",
  "username":"cp",
  "password":"ADD_YOUR_ENTITLEMENT_KEY"
}
```

Enter the following command to include the authority.json as a new authority in your .dockerconfigjson and store it as temp_config.json:

$ oc get secret/pull-secret -n openshift-config -ojson | jq -r '.data[".dockerconfigjson"]' | \
base64 -d - | jq '.[]."cp.icr.io" += input' - authority.json > temp_config.json

Verify that your authority credentials are appended at the end of the file:

$ cat temp_config.json
   {
      "auths": {
         ...
         ...
         ...
         "cp.icr.io": {
            "auth": "ADD_BASE64_ENCODED_ENTITLEMENT_KEY",
            "username": "cp",
            "password": "ADD_YOUR_ENTITLEMENT_KEY"
         }
      }
   }

Use the contents of the temp_config.json file and apply the updated config to the Red Hat OpenShift cluster:
```
$ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=temp_config.json
```
To verify that your pull-secret is updated with your new authority, issue the following command and confirm that your authority is present:
```
$ oc get secret/pull-secret -n openshift-config -ojson | jq -r '.data[".dockerconfigjson"]' | base64 -d -
```
This update is rolled out to all nodes. To track the progress of the roll out, run the following command:
```
$ oc get mcp
```
Remove the temporary files:
```
rm authority.json temp_config.json
```

Red Hat Openshift configuration (on-premise)

Applying Machine Config Operator (MCO) settings

Before you can start with the IBM Storage Scale CNSA installation, you must make changes to the Red Hat OpenShift compute nodes to ensure that IBM Storage Scale CNSA can operate properly.

Install kernel-development and kernel- header packages for IBM Storage Scale to successfully build its portability layer.
Set the vmalloc kernel parameter to operate properly with Red Hat CoreOS.

Note: Applying Machine Configs to update the Red Hat OpenShift Container Platform cluster can take around 30 minutes depending on the size of the compute node pool, as the compute nodes will be rebooted. If you have previously applied these Machine Configs, you do not need to apply them again.

Determine the level of the Red Hat OpenShift Container Platform cluster and apply the following steps.

$ oc apply -f https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.2.1.x/generated/scale/mco/ocp4.<version>/mco_s390x.yaml

Note: This process can take over 30 minutes as the compute node will be rebooted.

Verify the status of the nodes:

 $ oc get nodes

 NAME                    STATUS                        ROLES    AGE   VERSION
 master0.subdomain.com   NotReady,SchedulingDisabled   master   99d   v1.21.6+935ba91
 master1.subdomain.com   Ready                         master   99d   v1.21.6+935ba91
 master2.subdomain.com   Ready                         master   99d   v1.21.6+935ba91
 worker0.subdomain.com   NotReady,SchedulingDisabled   worker   99d   v1.21.6+935ba91
 worker1.subdomain.com   Ready                         worker   99d   v1.21.6+935ba91
 worker2.subdomain.com   Ready                         worker   99d   v1.21.6+935ba91
 worker3.subdomain.com   Ready                         worker   99d   v1.21.6+935ba91

As long as one of the nodes has the status SchedulingDisabled the changes have not applied completely.

Validate that the kernel-devel package is successfully applied onto the Red Hat OpenShift Container Platform compute nodes.

$ oc get nodes -lnode-role.kubernetes.io/worker= \-ojsonpath="{range.items[*]}{.metadata.name}{'\n'}" |\xargs -I{} oc debug node/{} -T -- chroot /host sh -c "rpm -q kernel-devel"
Starting pod/worker-001a311lp37fr25gslnxperfboe-debug-f7bq7 ...
To use host binaries, run `chroot /host`
kernel-devel-5.14.0-427.28.1.el9_4.s390x

Removing debug pod ...
Starting pod/worker-002a311lp37fr25gslnxperfboe-debug-tlsff ...

You will see the kernel-devel-<kernel version> package in the above output for each compute node.

The vmalloc kernel parameter can be verified in the same way:

$ oc get nodes -lnode-role.kubernetes.io/worker= \-ojsonpath="{range.items[*]}{.metadata.name}{'\n'}" |\xargs -I{} oc debug node/{} -T -- chroot /host sh -c "cat /proc/cmdline"
Starting pod/worker-001a311lp37fr25gslnxperfboe-debug-z22sd ...
To use host binaries, run `chroot /host`
rw  ostree=/ostree/boot.1/rhcos/8c1cf6f5cec2274dcf672ded1e7a27a73da84c847d78d1d4aae3c18c2a4bdb53/0 ignition.platform.id=metal root=UUID=d1dc517b-0fe3-447c-924a-fe3f9dc75457 rw rootflags=prjquota boot=UUID=86626c5e-039e-44eb-be6f-e35d0f1d14e4 vmalloc=4096G systemd.unified_cgroup_hierarchy=1 cgroup_no_v1="all" psi=1

vmalloc=4096G is displayed in the above output for each compute node.

Similar to the configuration tasks on the workers nodes, Machine Configuration Operator (MCO) settings must be applied to the infrastructure nodes if you want to run IBM Storage Scale core pods.

$ curl https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.2.1.x/generated/scale/mco/ocp4.16/mco_s390x.yaml | sed 's/worker/infra/g' | oc apply -f -

Infrastructure node configuration

Similar to the configuration tasks on the workers nodes, Machine Configuration Operator (MCO) settings must be applied to the infrastructure nodes if you want to run IBM Storage Scale core pods.

$ curl https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.2.1.x/generated/scale/mco/ocp4.16/mco_s390x.yaml | sed 's/worker/infra/g' | oc apply -f -