User affected:

All IBM Fusion HCI users who upgrade to Red Hat OpenShift z-stream across OpenShift 4.16 (4.16.48+), 4.17 (4.17.40+), and 4.18 (4.18.24+). 

Recommendations:

All IBM Fusion HCI users running upon OpenShift 4.16, 4.17, 4.18 should apply this fix before updating any OpenShift z-stream levels that install 5.14.0-427.87.1.el9_4 or later.

Prerequisites:

Note: Ensure that you met the following prerequisites for the offline environment.

Mirror the required images by using the following steps:

1. Set TARGET_PATH variable as follows:
   
   Note: Make sure you log into the local registry and IBM registry from the mirror host where you run the image copy skopeo command.

   export LOCAL_ISF_REGISTRY="<Your container registry host>:<port>"

   export LOCAL_ISF_REPOSITORY="<Your image path>"

   export TARGET_PATH="$LOCAL_ISF_REGISTRY/$LOCAL_ISF_REPOSITORY"

2. From the mirroring host, run the following commands to copy the required images to the artifactory:
   
   skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/erasure-code/ibm-spectrum-scale-daemon@sha256:6b0fb12a6669e6b80008176e2ea951b7ce9915338fe3c642429d755ef2ca1537 docker://$TARGET_PATH/erasure-code/ibm-spectrum-scale-daemon@sha256:6b0fb12a6669e6b80008176e2ea951b7ce9915338fe3c642429d755ef2ca1537 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/data-management/ibm-spectrum-scale-daemon@sha256:ffb7a3ee5435283ae0f9d691dd02cbdf840621d742d41db3d74a7282a77deb7e docker://$TARGET_PATH/data-management/ibm-spectrum-scale-daemon@sha256:ffb7a3ee5435283ae0f9d691dd02cbdf840621d742d41db3d74a7282a77deb7e 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-core-init@sha256:39c162c536befc9e8388cf93c6bf35ff745772f20713a76a17a5fc453283b316 docker://$TARGET_PATH/ibm-spectrum-scale-core-init@sha256:39c162c536befc9e8388cf93c6bf35ff745772f20713a76a17a5fc453283b316 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-gui@sha256:e28be0799e62625504c0a7514e22f0946c943b39acebee63970f002112c87867 docker://$TARGET_PATH/ibm-spectrum-scale-gui@sha256:e28be0799e62625504c0a7514e22f0946c943b39acebee63970f002112c87867 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-logs@sha256:43ff86e15a214223fbe8d9ebfdbb4b3a4affd76dc37febeae665cf0c8e69e2d8 docker://$TARGET_PATH/ibm-spectrum-scale-logs@sha256:43ff86e15a214223fbe8d9ebfdbb4b3a4affd76dc37febeae665cf0c8e69e2d8 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-pmcollector@sha256:f64e2e6f0faf709093ec535b1c790ada8fd5ff88c96e7e793160271d55a53421 docker://$TARGET_PATH/ibm-spectrum-scale-pmcollector@sha256:f64e2e6f0faf709093ec535b1c790ada8fd5ff88c96e7e793160271d55a53421 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-monitor@sha256:58035894d2f16b3f198eaab41a4f4031c3c6afea4efa46a857de3fdc59a858a3 docker://$TARGET_PATH/ibm-spectrum-scale-monitor@sha256:58035894d2f16b3f198eaab41a4f4031c3c6afea4efa46a857de3fdc59a858a3 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-grafana-bridge@sha256:ce0de4057f5980183ca82f67d241d50e5c63b7b0c43bceb1ca847525f9fd58c5 docker://$TARGET_PATH/ibm-spectrum-scale-grafana-bridge@sha256:ce0de4057f5980183ca82f67d241d50e5c63b7b0c43bceb1ca847525f9fd58c5 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-must-gather@sha256:4364bae5e90691466e1c01780359ca152f8cfe8e118d4c1ec0f929ab641c64df docker://$TARGET_PATH/ibm-spectrum-scale-must-gather@sha256:4364bae5e90691466e1c01780359ca152f8cfe8e118d4c1ec0f929ab641c64df 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-csi-operator@sha256:3f358e4d2895867198ee114b90ce5923325115ef75a2b49751280901423e3464 docker://$TARGET_PATH/ibm-spectrum-scale-csi-operator@sha256:3f358e4d2895867198ee114b90ce5923325115ef75a2b49751280901423e3464 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-operator@sha256:8b1793ef7c7858cdcd1a85ecffbbec9e981615dba8a8c11e9edc3f1035e88494 docker://$TARGET_PATH/ibm-spectrum-scale-operator@sha256:8b1793ef7c7858cdcd1a85ecffbbec9e981615dba8a8c11e9edc3f1035e88494 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/csi/ibm-spectrum-scale-csi-driver@sha256:9fbd6cb9b434faa7431e0b7cb8b7423baeb0d9cea4db365b76af60b35d460054 docker://$TARGET_PATH/csi/ibm-spectrum-scale-csi-driver@sha256:9fbd6cb9b434faa7431e0b7cb8b7423baeb0d9cea4db365b76af60b35d460054
    
3. After all the skopeo commands complete successfully, update the existing ICSP for scale and provide the new $TARGET_PATH for both sources cp.icr.io/cp/gpfs and icr.io/cpopen.
   
   - mirrors:
  - old mirror path
  - $TARGET_PATH. <---provide new path
source: cp.icr.io/cp/gpfs
- mirrors:
  - old mirror path
  - $TARGET_PATH. <---provide new path
source: icr.io/cpopen
    
4. Wait until the MCP rollout is completed. You can check MCP rollout status using the following command:
   
   oc get mcp
    
5. Mirror the isf-cns-operator, isf-storage-operator, and isf-storage-services images in the ibm-spectrum-fusion-ns namespace:
   
   skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-hci/isf-cns-operator@sha256:95d59f418f1d338d2a7bd892da8e5cd3e341e18ac7d137aa62ab5a4915cd203f docker://$TARGET_PATH/cp/fusion-hci/isf-cns-operator@sha256:95d59f418f1d338d2a7bd892da8e5cd3e341e18ac7d137aa62ab5a4915cd203f

skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-hci/isf-storage-services@sha256:c489e40c60a474e7a23d630b52efa34af210deda0c8c6350d9a2642c78d6c9a0 docker://$TARGET_PATH/cp/fusion-hci/isf-storage-services@sha256:c489e40c60a474e7a23d630b52efa34af210deda0c8c6350d9a2642c78d6c9a0

skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-hci/isf-storage-operator@sha256:e8431fd5baa6f2d8f0643cb0bda7ff7c984f90eed05e80c4ff4d9eebc79b0751 docker://$TARGET_PATH/cp/fusion-hci/isf-storage-operator@sha256:e8431fd5baa6f2d8f0643cb0bda7ff7c984f90eed05e80c4ff4d9eebc79b0751
    
6. For the images, update the ICSP for IBM Storage Fusion images and provide the new $TARGET_PATH to the source cp.icr.io/cp. 
   
   For example:
   - mirrors:
  - old mirror path
  - $TARGET_PATH. <---provide new path
source: cp.icr.io/cp
    
7. After completing all the steps, wait until the MCP rollout is completed. You can check the MCP rollout status using the following command:
   
   oc get mcp

Note:MCP might not always roll out.
    

Resolution:

1. Enable the manual upgrade as follows:
   1. Log in to the OpenShift web console.
   2. Go to Administration > CustomResourceDefinitions.
   3. Select ScaleManager CR and go to the Instances tab.
   4. Under the Instances tab, select scalemanager and go to the YAML tab.

   5. Add a field enableManualInstallation: true to the scalemanager CR as follows:

      apiVersion: cns.isf.ibm.com/v1 
kind: ScaleManager 
metadata: 
  name:scalemanager 
   namespace: ibm-spectrum-fusion-ns 
spec: 
  creator: Fusion 
  enableManualInstallation: true <---- add here 

2. Update the operator images as follows:
   1. Replace the isf-storage-operator-controller-manager image with a new image in the installed operator CSV of the ibm-spectrum-fusion-ns namespace.
      
      isf-storage-operator
      cp.icr.io/cp/fusion-hci/isf-storage-operator@sha256:e8431fd5baa6f2d8f0643cb0bda7ff7c984f90eed05e80c4ff4d9eebc79b0751
       

   2. Replace the isf-cns-operator and isf-storage-service image in the installed operator CSV and the cr-version-cm configmap in the ibm-spectrum-fusion-ns namespace.
      
      isf-cns-operator
      cp.icr.io/cp/fusion-hci/isf-cns-operator@sha256:95d59f418f1d338d2a7bd892da8e5cd3e341e18ac7d137aa62ab5a4915cd203f
      
      isf-storage-services

      cp.icr.io/cp/fusion-hci/isf-storage-services@sha256:c489e40c60a474e7a23d630b52efa34af210deda0c8c6350d9a2642c78d6c9a0

3. Run the following commands to verify the running pods: 
   
   oc get pods -n ibm-spectrum-fusion-ns | grep isf-storage-service-dep 
oc get pods -n ibm-spectrum-fusion-ns | grep isf-storage-operator-controller-manager 
oc get pods -n ibm-spectrum-fusion-ns | grep isf-cns-operator-controller-manager
   
   Example output:

   isf-storage-service-dep-xxxx-xxxx                     1/1     Running   0   2m
   isf-storage-operator-controller-manager-xxxx-xxxx     2/2     Running   0   1m
   isf-cns-operator-controller-manager-xxxx-xxxx         2/2     Running   0   2m

4.  Run the following command to trigger the service upgrade:

    For GDP service system:

   1. Patch the Scale CR using the following command:
      
      oc patch Scale storagemanager -n ibm-spectrum-fusion-ns --type='json' -p='[{"op": "replace", "path": "/spec/triggerUpdate", "value": true}]'
       

   2. Trigger the upgrade using the following commands:

      ```
oc project ibm-spectrum-fusion-ns 
oc rsh <isf-storage-service-dep pod name> 
curl -k https://isf-scale-svc/api/v1/upgradeWithOperator 

      ```
      
      Expected output: 
      
      {"status":"Deployed ECE and CSI upgrade yaml files on OCP cluster successfully"}
       

    For GDP Remote mount service system:

   1. Click upgrade button on the IBM Fusion user interface to upgrade the Global Data Platform Remote mount service.
       
5. Follow the steps to monitor the upgrade:
   1. Check if coreECE and coreDME images are updated in the ibm-spectrum-scale-manager-config configmap under the ibm-spectrum-scale-operator namespace.
   2. After triggering upgrade, the Scale pods restart one at a time.
      Note: In some cases, the OCP nodes may be restarted.
   3. After the upgrade is completed, the new Scale pods come up in the ibm-spectrum-scale namespace.
       

6. Run the following commands for verification:

   # Switch to ibm-spectrum-scale project
   oc project ibm-spectrum-scale
   
   # Get all Scale pods
   oc get pods 
   
   # Login to any Scale core pod
   oc rsh <podname>
   
   # Inside the pod
   mmdiag --version       # Check scale version
   mmgetstate -a          # Check node state
   mmlsmount all          # Verify filesystems are mounted

    

Metro-DR environments:

1. For GDP Metro-DR environments, following the same resolutions steps above to apply the hotfix on both racks one after the another.
2. Follow the steps to upgrade the tiebreaker to 5.2.3.3:
   1. SSH into the tiebreaker virtual machine.
   2. Download the IBM Storage Scale Data Management 5.2.3.3 from Fix Central as follows:
      1. Go to https://www.ibm.com/support/fixcentral.
      2. Navigate as follows:
         1. Enter the text "system storage" in the Product selector field.
         2. Select the System Storage from the search results.
         3. Select the Storage software in the first Product drop-down.
         4. Select the Storage defined storage in the second Product drop-down.
         5. Select the IBM Storage Scale in the third Product drop-down.
         6. Select 5.2.3.0 version in the Installed Version drop-down.
         7. Select the platform that you want from the Platform drop-down.
         8. Click Continue.

         9. In the Select Fixes page, select the Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux in the Data Management section.

         10. Download the Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux-install file.

   3. Run the following command to extract the IBM Storage Scale package:

      chmod +x Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux-install ./Storage_Scale_Data_Management-5.2.3.3-x86_64-Linux-install

   4. Go to directory /usr/lpp/mmfs/5.2.3.3/ansible-toolkit.

   5. Run setup command to install ansible. 
      ./spectrumscale setup -s <Tiebreaker IP>

   6. Run the following command to add the tiebreaker node: 
      ./spectrumscale node add <Tiebreaker IP> -a

   7. Run the following command for upgrade precheck: 
      ./spectrumscale upgrade precheck --skip ssh

   8. Run the following command to upgrade tiebreaker: 
      ./spectrumscale upgrade run --skip ssh

   9. Reboot the tiebreaker.