ImageContentSourcePolicy creation fails

Cloud Pak Operand deployment fails in an air-gapped or disconnected environment with an ImagePullBackOff error when a Quay registry is being used with defined organizations.

Symptom

The Operand deployment fails with an error similar to:

Failed to pull image "cp.icr.io/cp/iaf-ui-customization@sha256:d9a3bf86182e977a15700d948ede61a00390dff88666615b394f9b74b193d2a4": rpc error: code = Unknown desc = (Mirrors also failed: [my-quay-registry.com/myOrg/iaf-ui-customization@sha256:d9a3bf86182e977a15700d948ede61a00390dff88666615b394f9b74b193d2a4: reading manifest sha256:d9a3bf86182e977a15700d948ede61a00390dff88666615b394f9b74b193d2a4 in my-quay-registry.com/myOrg/iaf-ui-customization: name unknown: repository not found]): cp.icr.io/cp/iaf-ui-customization@sha256:d9a3bf86182e977a15700d948ede61a00390dff88666615b394f9b74b193d2a4: Requesting bear token: invalid status code from registry 400 (Bad Request)

Cause

When you run the oc ibm-pak command with --action configure-cluster-airgap and use the --nsPrefix parameter to specify the organization in Quay where the images are mirrored, the generated ImageContentSourcePolicy is invalid.

For example, the ImageContentSourcePolicy creation fails when you run the following command:

# oc ibm-pak case launch --case ${CASE_LOCAL_PATH} --inventory ${CASE_INVENTORY_SETUP} --action configure-cluster-airgap --namespace ${NAMESPACE} --args "--registry ${LOCAL_REGISTRY} --user ${LOCAL_REGISTRY_USER} --pass ${LOCAL_REGISTRY_PASS} --inputDir ${OFFLINEDIR} --nsPrefix ${NSPREFIX}"

Resolving the problem

To resolve this issue, you must manually correct the generated ImageContentSourcePolicy.

1. Locate the generated ImageContentSourcePolicy.

    oc get imageContentSourcePolicy
   

2. Edit the policy.

    oc edit imageContentSourcePolicy <image-content-source-policy>
   

3. Append the image subpaths after the organization. For example, change the following specification:

    Spec:
      Repository Digest Mirrors:
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg
        Source:  cp.icr.io/cp
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg
        Source:  docker.io/ibmcom
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg
        Source:  icr.io/cpopen
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg
        Source:  icr.io/db2u
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg
        Source:  icr.io/ibm-messaging
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg
        Source:  quay.io/opencloudio
   

   to the following specification:

    Spec:
      Repository Digest Mirrors:
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg/cp
        Source:  cp.icr.io/cp
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg/ibmcom
        Source:  docker.io/ibmcom
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg/cpopen
        Source:  icr.io/cpopen
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg/db2u
        Source:  icr.io/db2u
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg/ibm-messaging
        Source:  icr.io/ibm-messaging
        Mirrors:
          us-west-2-registry.cloudpak-bringup.com/myOrg/opencloudio
        Source:  quay.io/opencloudio
   

You might need to wait for the cluster nodes to reboot after editing the ImageContentSourcePolicy in order for it to take effect.