IBM Support

Operator upgrade can get stuck for IBM Fusion HCI and IBM Fusion

Troubleshooting


Problem

The Fusion operator upgrade can get stuck during upgrade due to a known Red Hat OLM issue. The issue is specific to Red Hat OpenShift Container Platform versions 4.12, 4.13, 4.14, 4.15, and 4.16. In IBM Fusion 2.9.0, this issue is also observed with OpenShift Container Platform version 4.17.
Note: The IBM Fusion HCI 2.9.0 does not include support for OpenShift Container Platform 4.17.

Diagnosing The Problem

In certain OpenShift Container Platform patch versions, there are chances that the Fusion operator can get stuck with a similar error:

error validating existing CRs against new CRD's schema for "fusionserviceinstances.service.isf.ibm.com": error validating service.isf.ibm.com/v1, Kind=FusionServiceInstance "ibm-spectrum-fusion-ns/odfmanager": updated validation is too restrictive: [].status.triggerCatSrcCreateStartTime: Invalid value: "number": status.triggerCatSrcCreateStartTime in body must be of type integer: "number"

If you encounter this error, you must contact the IBM support.

To prevent this error from occurring, follow the steps mentioned in the Resolving the problem section. The steps patch the FusionServiceInstance CRs and FusionServiceDefinition CRs to reset the fields that can block the upgrade.

Resolving The Problem

Do the following steps before you upgrade the Fusion operator:

If the OC commands do not work, then from the OpenShift console, upgrade your OC binary to the latest version. If the patch commands mentioned in the following steps do not work with the older versions, then the recommended OC version is 4.1.6.x.
  1. Run the following OC commands specific to FusionServiceInstance CRs:
     

    Note: If there are any FusionServiceInstance CRs present in the cluster, then run the commands mentioned in this step.

    oc project $(oc get spectrumfusion -A -o custom-columns=NS:metadata.namespace --no-headers)

    oc get fusionserviceinstance -o jsonpath='{.items[*].metadata.name}' |tr ' ' '\n' | xargs -I {} oc patch fusionserviceinstance {} --type=merge --subresource=status -p '{"status": {"triggerCatSrcCreateStartTime": 0}}'

    oc get fusionserviceinstance -o jsonpath='{.items[*].metadata.name}' |tr ' ' '\n' | xargs -I {} oc patch fusionserviceinstance {} --type=merge --subresource=status -p '{"status": {"currentInstallStartTime": 0}}'

    oc get fusionserviceinstance -o jsonpath='{.items[*].metadata.name}' |tr ' ' '\n' | xargs -I {} oc patch fusionserviceinstance {} --type=merge --subresource=status -p '{"status": {"operatorLastUpdateTime": 0}}'

    oc get fusionserviceinstance -o jsonpath='{.items[*].metadata.name}' |tr ' ' '\n' | xargs -I {} oc patch fusionserviceinstance {} --type=merge --subresource=status -p '{"status": {"operatorUpgradeStartTime": 0}}'
  2. Run the OC command specific to FusionServiceDefinition CR: 
    oc get fusionservicedefinition -o jsonpath='{.items[*].metadata.name}' |tr ' ' '\n' | xargs -I {} oc patch fusionservicedefinition {} --type=merge -p '{"spec": {"serviceInformation": {"lastUpdated": 0}}}'

Note: If there are any FusionServiceDefinition CRs present in the cluster, then run the commands mentioned in this step.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB69","label":"Storage TPS"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSFETU","label":"IBM Fusion"},"ARM Category":[{"code":"a8m3p0000000rXCAAY","label":"SW"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB69","label":"Storage TPS"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSXEFDS","label":"IBM Fusion HCI Appliance Software"},"ARM Category":[{"code":"a8m3p0000000rXCAAY","label":"SW"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
24 January 2025

UID

ibm17173499

Page Feedback