User affected:

All IBM Fusion HCI or IBM Fusion users who upgrade to Red Hat OpenShift z-stream across OpenShift 4.16 (4.16.48+), 4.17 (4.17.40+), and 4.18 (4.18.24+). 

Recommendations:

All IBM Fusion HCI or IBM Fusion users running upon OpenShift 4.16, 4.17, 4.18 should apply this fix before updating any OpenShift z-stream levels that install 5.14.0-427.87.1.el9_4 or later.

Resolution:

1. For offline installation : Mirror the required images by using the following steps:

   1. Set TARGET_PATH variable as follows:
      
      Note: Make sure you log into the local registry and IBM registry from the mirror host where you run the image copy skopeo command.

      export LOCAL_ISF_REGISTRY="<Your container registry host>:<port>"

      export LOCAL_ISF_REPOSITORY="<Your image path>"

      export TARGET_PATH="$LOCAL_ISF_REGISTRY/$LOCAL_ISF_REPOSITORY"-p

   2. From the mirroring host, run the following commands to copy the required images to the artifactory: 
      
      skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/data-management/ibm-spectrum-scale-daemon@sha256:ffb7a3ee5435283ae0f9d691dd02cbdf840621d742d41db3d74a7282a77deb7e docker://$TARGET_PATH/data-management/ibm-spectrum-scale-daemon@sha256:ffb7a3ee5435283ae0f9d691dd02cbdf840621d742d41db3d74a7282a77deb7e 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-core-init@sha256:39c162c536befc9e8388cf93c6bf35ff745772f20713a76a17a5fc453283b316 docker://$TARGET_PATH/ibm-spectrum-scale-core-init@sha256:39c162c536befc9e8388cf93c6bf35ff745772f20713a76a17a5fc453283b316 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-gui@sha256:e28be0799e62625504c0a7514e22f0946c943b39acebee63970f002112c87867 docker://$TARGET_PATH/ibm-spectrum-scale-gui@sha256:e28be0799e62625504c0a7514e22f0946c943b39acebee63970f002112c87867 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-logs@sha256:43ff86e15a214223fbe8d9ebfdbb4b3a4affd76dc37febeae665cf0c8e69e2d8 docker://$TARGET_PATH/ibm-spectrum-scale-logs@sha256:43ff86e15a214223fbe8d9ebfdbb4b3a4affd76dc37febeae665cf0c8e69e2d8 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-pmcollector@sha256:f64e2e6f0faf709093ec535b1c790ada8fd5ff88c96e7e793160271d55a53421 docker://$TARGET_PATH/ibm-spectrum-scale-pmcollector@sha256:f64e2e6f0faf709093ec535b1c790ada8fd5ff88c96e7e793160271d55a53421 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-monitor@sha256:58035894d2f16b3f198eaab41a4f4031c3c6afea4efa46a857de3fdc59a858a3 docker://$TARGET_PATH/ibm-spectrum-scale-monitor@sha256:58035894d2f16b3f198eaab41a4f4031c3c6afea4efa46a857de3fdc59a858a3 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-grafana-bridge@sha256:ce0de4057f5980183ca82f67d241d50e5c63b7b0c43bceb1ca847525f9fd58c5 docker://$TARGET_PATH/ibm-spectrum-scale-grafana-bridge@sha256:ce0de4057f5980183ca82f67d241d50e5c63b7b0c43bceb1ca847525f9fd58c5 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-must-gather@sha256:4364bae5e90691466e1c01780359ca152f8cfe8e118d4c1ec0f929ab641c64df docker://$TARGET_PATH/ibm-spectrum-scale-must-gather@sha256:4364bae5e90691466e1c01780359ca152f8cfe8e118d4c1ec0f929ab641c64df 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-csi-operator@sha256:3f358e4d2895867198ee114b90ce5923325115ef75a2b49751280901423e3464 docker://$TARGET_PATH/ibm-spectrum-scale-csi-operator@sha256:3f358e4d2895867198ee114b90ce5923325115ef75a2b49751280901423e3464 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-operator@sha256:8b1793ef7c7858cdcd1a85ecffbbec9e981615dba8a8c11e9edc3f1035e88494 docker://$TARGET_PATH/ibm-spectrum-scale-operator@sha256:8b1793ef7c7858cdcd1a85ecffbbec9e981615dba8a8c11e9edc3f1035e88494 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/csi/ibm-spectrum-scale-csi-driver@sha256:9fbd6cb9b434faa7431e0b7cb8b7423baeb0d9cea4db365b76af60b35d460054 docker://$TARGET_PATH/csi/ibm-spectrum-scale-csi-driver@sha256:9fbd6cb9b434faa7431e0b7cb8b7423baeb0d9cea4db365b76af60b35d460054
   3. After all the skopeo commands complete successfully, update the existing ICSP for scale and provide the new $TARGET_PATH for both sources cp.icr.io/cp/gpfs and icr.io/copen.
      
      - mirrors:
- old mirror path
- $TARGET_PATH. <---provide new path
source: cp.icr.io/cp/gpfs
- mirrors:
- old mirror path
- $TARGET_PATH. <---provide new path
source: icr.io/copen
   4. Wait until the MCP rollout is completed. You can check MCP rollout status using the following command:
      
      oc get mcp
   5. After the MCO rollout is completed, mirror the isf-cns-operator, isf-storage-operator, and isf-storage-services image:
      
      For HCI:
      
      skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-hci/isf-cns-operator@sha256:1d87e6a2d8325f6cfdc3deecd9ceff17656ad64b7b7aa1b9e0d4f6705066bd22 docker://$TARGET_PATH/cp/fusion-hci/isf-cns-operator@sha256:1d87e6a2d8325f6cfdc3deecd9ceff17656ad64b7b7aa1b9e0d4f6705066bd22

skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-hci/isf-storage-services@sha256:f7389e21b6c8046d873a0936689c023ece4deec41f4429628549492a59408cd5 docker://$TARGET_PATH/cp/fusion-hci/isf-storage-services@sha256:f7389e21b6c8046d873a0936689c023ece4deec41f4429628549492a59408cd5

skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-hci/isf-storage-operator@sha256:e0b2ff6a2cdb73bfb2052a7eea18e1a2b0e7a1ad5c15c35dda20d6cdac4c311a docker://$TARGET_PATH/cp/fusion-hci/isf-storage-operator@sha256:e0b2ff6a2cdb73bfb2052a7eea18e1a2b0e7a1ad5c15c35dda20d6cdac4c311a
      
      For SDS:
      
      skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-sds/isf-cns-operator@sha256:8076047c848c711e6958edd4a816f560ce2c1befb6e67068418dfc4315655bbe docker://$TARGET_PATH/cp/fusion-sds/isf-cns-operator@sha256:8076047c848c711e6958edd4a816f560ce2c1befb6e67068418dfc4315655bbe 
       
   6. HCI only - For the isf-storage-operator-image, update the ICSP for IBM Storage Fusion images and provide the new $TARGET_PATH to the source cp.icr.io/cp. 
      
      For example:
      - mirrors:
- old mirror path
- $TARGET_PATH. <---provide new path
source: cp.icr.io/cp
   7. After completing all the steps, wait until the MCP rollout is completed. You can check the MCP rollout status using the following command:
      
      oc get mcp

Note:MCP might not always roll out.

2. For online installation : Update the operator images as follows:

   For HCI:

   1. Replace the isf-storage-operator-controller-manager image with a new image in the installed operator CSV of the ibm-spectrum-fusion-ns.
      
      isf-storage-operator
      cp.icr.io/cp/fusion-hci/isf-storage-operator@sha256:e0b2ff6a2cdb73bfb2052a7eea18e1a2b0e7a1ad5c15c35dda20d6cdac4c311a
   2. Replace the if-cns-operator and isf-storage-service image in the cr-version-cm configmap in the ibm-spectrum-fusion-ns namespace.
      
      isf-cns-operator
      cp.icr.io/cp/fusion-hci/isf-cns-operator@sha256:1d87e6a2d8325f6cfdc3deecd9ceff17656ad64b7b7aa1b9e0d4f6705066bd22
      
      isf-storage-services
      cp.icr.io/cp/fusion-hci/isf-storage-services@sha256:f7389e21b6c8046d873a0936689c023ece4deec41f4429628549492a59408cd5

   For SDS:

   1. Go to Operators > Installed Operators.
   2. For isf-cns-operator, edit the IBM Fusion CSV in a YAML view.
   3. Update the image for isf-cns-operator.

   4. Search for cp.icr.io/cp/fusion-sds/isf-cns-operator@sha256 and replace all the images with the following image.

      cp.icr.io/cp/fusion-sds/isf-cns-operator@sha256:8076047c848c711e6958edd4a816f560ce2c1befb6e67068418dfc4315655bbe

3. Run the following commands to verify the running pods: 
   
   oc get pods -n ibm-spectrum-fusion-ns | grep isf-storage-service-dep (HCI only)
   oc get pods -n ibm-spectrum-fusion-ns | grep isf-storage-operator-controller-manager (HCI only)
   oc get pods -n ibm-spectrum-fusion-ns | grep isf-cns-operator-controller-manager (HCI and SDS)
   
   Example output:

   isf-storage-service-dep-xxxx-xxxx                     1/1     Running   0   86m
   isf-storage-operator-controller-manager-xxxx-xxxx     2/2     Running   0   87m
   isf-cns-operator-controller-manager-xxxx-xxxx         2/2     Running   0   87m

4.  For HCI, run the following command to trigger the update in Scale CR:
   
   oc patch Scale storagemanager -n ibm-spectrum-fusion-ns --type='json' -p='[{"op": "replace", "path": "/spec/triggerUpdate", "value": true}]' 
   
   After the new isf-storage-service-dep pod starts with the latest image:
   
   oc project ibm-spectrum-fusion-ns 
oc rsh <isf-storage-service-dep pod name> 
curl -k https://isf-scale-svc/api/v1/upgradeWithOperator 
   
   Expected output: 
   
   {"status":"Deployed ECE and CSI upgrade yaml files on OCP cluster successfully"}

5. For SDS, click upgrade button on the IBM Fusion user interface to upgrade the Global Data Platform service.

6. Follow the steps to verify the update in the Scale CR:

   1. Check if coreECE images are updated in the ibm-spectrum-scale-manager-config ConfigMap under the ibm-spectrum-scale-operator namespace.

   2. Verify new pods are running in the ibm-spectrum-scale-operator namespace

   3. Ensure all pods are in Running state under the ibm-spectrum-scale-csi namespace. 

7. Follow the steps to monitor the upgrade:
   1. After patching, the nodes restart one at a time.
   2. Once all nodes are rebooted, new Scale pods will come up in the ibm-spectrum-scale namespace.

   3. Run the following commands for verification:

      # Get all Scale pods
      oc get pods -n ibm-spectrum-scale
      # Login to any Scale core pod
      oc rsh <podname>
      # Inside the pod
      mmdiag --version       # Check scale version
      mmgetstate -a          # Check node state
      mmlsmount all          # Verify filesystems are mounted

8. For HCI GDP Metro-DR environments, following the same resolutions steps 1 to 7 above to apply the hotfix on both racks one after another.
9. Follow the steps to upgrade the tiebreaker to 5.2.3.3:
   1. SSH into the tiebreaker virtual machine.
   2. Download the IBM Storage Scale Data Management 5.2.3.3 from Fix Central as follows:
      1. Go to https://www.ibm.com/support/fixcentral.
      2. Navigate as follows:
         1. Enter the text "system storage" in the Product selector field.
         2. Select the System Storage from the search results.
         3. Select the Storage software in the first Product drop-down.
         4. Select the Storage defined storage in the second Product drop-down.
         5. Select the IBM Storage Scale in the third Product drop-down.
         6. Select 5.2.3.0 version in the Installed Version drop-down.
         7. Select the platform that you want from the Platform drop-down.
         8. Click Continue.

         9. In the Select Fixes page, select the Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux in the Data Management section.

         10. Download the Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux-install file.

   3. Run the following command to extract the IBM Storage Scale package:

      chmod +x Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux-install ./Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux-install

   4. Go to directory /usr/lpp/mmfs/5.2.3.3/ansible-toolkit.

   5. Run setup command to install ansible. 
      ./spectrumscale setup -s <Tiebreaker IP>

   6. Run the following command to add the tiebreaker node: 
      ./spectrumscale node add <Tiebreaker IP> -a

   7. Run the following command for upgrade precheck: 
      ./spectrumscale upgrade precheck --skip ssh

   8. Run the following command to upgrade tiebreaker: 
      ./spectrumscale upgrade run --skip ssh

   9. Reboot the tiebreaker.