User affected:

All IBM Fusion users who upgrade to Red Hat OpenShift z-stream across OpenShift 4.16 (4.16.48+), 4.17 (4.17.40+), and 4.18 (4.18.24+). 

Recommendations:

All IBM Fusion users running upon OpenShift 4.16, 4.17, 4.18 should apply this fix before updating any OpenShift z-stream levels that install 5.14.0-427.87.1.el9_4 or later.

Prerequisites:

Note: Ensure that you met the following prerequisites for the offline environment.

Mirror the required images by using the following steps:

1. Set TARGET_PATH variable as follows:
   
   Note: Make sure you log into the local registry and IBM registry from the mirror host where you run the image copy skopeo command.

   export LOCAL_ISF_REGISTRY="<Your container registry host>:<port>"

   export LOCAL_ISF_REPOSITORY="<Your image path>"

   export TARGET_PATH="$LOCAL_ISF_REGISTRY/$LOCAL_ISF_REPOSITORY"

2. From the mirroring host, run the following commands to copy the required images to the artifactory:
   
   skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/data-management/ibm-spectrum-scale-daemon@sha256:ffb7a3ee5435283ae0f9d691dd02cbdf840621d742d41db3d74a7282a77deb7e docker://$TARGET_PATH/data-management/ibm-spectrum-scale-daemon@sha256:ffb7a3ee5435283ae0f9d691dd02cbdf840621d742d41db3d74a7282a77deb7e 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-core-init@sha256:39c162c536befc9e8388cf93c6bf35ff745772f20713a76a17a5fc453283b316 docker://$TARGET_PATH/ibm-spectrum-scale-core-init@sha256:39c162c536befc9e8388cf93c6bf35ff745772f20713a76a17a5fc453283b316 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-gui@sha256:e28be0799e62625504c0a7514e22f0946c943b39acebee63970f002112c87867 docker://$TARGET_PATH/ibm-spectrum-scale-gui@sha256:e28be0799e62625504c0a7514e22f0946c943b39acebee63970f002112c87867 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-logs@sha256:43ff86e15a214223fbe8d9ebfdbb4b3a4affd76dc37febeae665cf0c8e69e2d8 docker://$TARGET_PATH/ibm-spectrum-scale-logs@sha256:43ff86e15a214223fbe8d9ebfdbb4b3a4affd76dc37febeae665cf0c8e69e2d8 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-pmcollector@sha256:f64e2e6f0faf709093ec535b1c790ada8fd5ff88c96e7e793160271d55a53421 docker://$TARGET_PATH/ibm-spectrum-scale-pmcollector@sha256:f64e2e6f0faf709093ec535b1c790ada8fd5ff88c96e7e793160271d55a53421 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-monitor@sha256:58035894d2f16b3f198eaab41a4f4031c3c6afea4efa46a857de3fdc59a858a3 docker://$TARGET_PATH/ibm-spectrum-scale-monitor@sha256:58035894d2f16b3f198eaab41a4f4031c3c6afea4efa46a857de3fdc59a858a3 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/ibm-spectrum-scale-grafana-bridge@sha256:ce0de4057f5980183ca82f67d241d50e5c63b7b0c43bceb1ca847525f9fd58c5 docker://$TARGET_PATH/ibm-spectrum-scale-grafana-bridge@sha256:ce0de4057f5980183ca82f67d241d50e5c63b7b0c43bceb1ca847525f9fd58c5 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-must-gather@sha256:4364bae5e90691466e1c01780359ca152f8cfe8e118d4c1ec0f929ab641c64df docker://$TARGET_PATH/ibm-spectrum-scale-must-gather@sha256:4364bae5e90691466e1c01780359ca152f8cfe8e118d4c1ec0f929ab641c64df 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-csi-operator@sha256:3f358e4d2895867198ee114b90ce5923325115ef75a2b49751280901423e3464 docker://$TARGET_PATH/ibm-spectrum-scale-csi-operator@sha256:3f358e4d2895867198ee114b90ce5923325115ef75a2b49751280901423e3464 

skopeo copy --all --preserve-digests docker://icr.io/cpopen/ibm-spectrum-scale-operator@sha256:8b1793ef7c7858cdcd1a85ecffbbec9e981615dba8a8c11e9edc3f1035e88494 docker://$TARGET_PATH/ibm-spectrum-scale-operator@sha256:8b1793ef7c7858cdcd1a85ecffbbec9e981615dba8a8c11e9edc3f1035e88494 

skopeo copy --all --preserve-digests docker://cp.icr.io/cp/gpfs/csi/ibm-spectrum-scale-csi-driver@sha256:9fbd6cb9b434faa7431e0b7cb8b7423baeb0d9cea4db365b76af60b35d460054 docker://$TARGET_PATH/csi/ibm-spectrum-scale-csi-driver@sha256:9fbd6cb9b434faa7431e0b7cb8b7423baeb0d9cea4db365b76af60b35d460054
    
3. After all the skopeo commands complete successfully, update the existing ICSP for scale and provide the new $TARGET_PATH for both sources cp.icr.io/cp/gpfs and icr.io/cpopen.
   
   - mirrors:
  - old mirror path
  - $TARGET_PATH. <---provide new path
source: cp.icr.io/cp/gpfs
- mirrors:
  - old mirror path
  - $TARGET_PATH. <---provide new path
source: icr.io/cpopen
    
4. Wait until the MCP rollout is completed. You can check MCP rollout status using the following command:
   
   oc get mcp
    
5. Mirror the isf-cns-operator image in the ibm-spectrum-fusion-ns namespace:
   
   skopeo copy --insecure-policy --preserve-digests --all docker://cp.icr.io/cp/fusion-sds/isf-cns-operator@sha256:efbacc098a023523436c778b65539837e8922b514842507874ed6ca278526aa6 docker://$TARGET_PATH/cp/fusion-sds/isf-cns-operator@sha256:efbacc098a023523436c778b65539837e8922b514842507874ed6ca278526aa6
    
6. Update the ICSP for IBM Storage Fusion images and provide the new $TARGET_PATH to the source cp.icr.io/cp. 
   
   For example:
   - mirrors:
  - old mirror path
  - $TARGET_PATH. <---provide new path
source: cp.icr.io/cp
    
7. After completing all the steps, wait until the MCP rollout is completed. You can check the MCP rollout status using the following command:
   
   oc get mcp

Note:MCP might not always roll out.
    

Resolution:

1. Enable the manual upgrade as follows:
   1. Log in to the OpenShift web console.
   2. Go to Administration > CustomResourceDefinitions.
   3. Select ScaleManager CR and go to the Instances tab.
   4. Under the Instances tab, select scalemanager and go to the YAML tab.

   5. Add a field enableManualInstallation: true to the scalemanager CR as follows:

      apiVersion: cns.isf.ibm.com/v1 
kind: ScaleManager 
metadata: 
  name:scalemanager 
   namespace: ibm-spectrum-fusion-ns 
spec: 
  creator: Fusion 
  enableManualInstallation: true <---- add here 

2. Replace the isf-cns-operator image with a new image in the installed operator CSV of the ibm-spectrum-fusion-ns namespace.
   
   isf-cns-operator
   cp.icr.io/cp/fusion-sds/isf-cns-operator@sha256:efbacc098a023523436c778b65539837e8922b514842507874ed6ca278526aa6
    

3. Run the following commands to verify the running pods: 
   
   oc get pods -n ibm-spectrum-fusion-ns | grep isf-cns-operator-controller-manager
   
   Example output:

   isf-cns-operator-controller-manager-xxxx-xxxx         2/2     Running   0   1m

4. Click upgrade button on the IBM Fusion user interface to upgrade the Global Data Platform Remote mount service.
    
5. Follow the steps to monitor the upgrade:
   1. Check if coreECE and coreDME images are updated in the ibm-spectrum-scale-manager-config configmap under the ibm-spectrum-scale-operator namespace.
   2. After triggering upgrade, the Scale pods restart one at a time.
      Note: In some cases, the OCP nodes may be restarted.
   3. After the upgrade is completed, the new Scale pods come up in the ibm-spectrum-scale namespace.
       

6. Run the following commands for verification:

   # Switch to ibm-spectrum-scale project
   oc project ibm-spectrum-scale
   
   # Get all Scale pods
   oc get pods 
   
   # Login to any Scale core pod
   oc rsh <podname>
   
   # Inside the pod
   mmdiag --version       # Check scale version
   mmgetstate -a          # Check node state
   mmlsmount all          # Verify filesystems are mounted