Troubleshooting issues when mirroring images for an air-gapped OpenShift cluster

Troubleshoot errors you may get when mirroring images.

In the final procedure for mirroring images, the command fails with an error

Symptom: In the final procedure, "Mirror images to final location and configure the OpenShift cluster", the command fails with this error:

error: unable to upload blob sha256:** no basic auth credentials

Cause: The registry credentials that you provided are incorrect.

Solution: In section 3.2, step 1, confirm that the value of <TARGET_REGISTRY> includes the hostname and port for your registry server, and ensure that the username and password used for docker or podman login are correct.

Authorization error received while images are pulled from entitled registry

Symptom: Authorization error, unauthorized, while images are pulled from the entitled registry.

For example:

unauthorized: authentication required
 Unable to copy layer ******

Solution:

  • Confirm that your credentials are correct by logging in to the cp.icr.io registry with either docker or podman. In this example, the username is cp and the password is your assigned entitlement key:

    docker login cp.icr.io -u cp -p <your_entitlement_key>

  • Try to pull the image from the registry by using either docker or podman. For example:

    podman pull cp.icr.io/cp/***/***/***

    If you can pull the image, that confirmation that the image exists in the entitled registry.

  • Retry the failing command.

  • If the error still exists, see Finding and applying your entitlement key by using the UI (online installation) to confirm your entitlement credentials, or reach out to IBM Support.

"Server misbehaving" error when pulling an image

Symptom: You receive an error when pulling an image: server misbehaving.

Solution:

  • On one of the nodes, confirm that /host/etc/containers/registries.conf contains all the rules defined in ImageContentSourcePolicy (search for that string in the OpenShift web console). If /host/etc/containers/registries.conf does not contain those rules, copy them into the file.

  • Confirm that a pullSecret has been created in the openshift-config namespace. This is a global pull secret and contains a docker/podman login credential for the applicable registries, which gets copied onto the nodes. Credentials can be found in $HOME/.docker/config.json or <user_path>/kubelet/config.json. If the pullSecret does not exist, you may need to contact IBM Support.

  • On the same node, confirm that the mirroring and the credentials are working. Pull the image manually with docker or podman, using the credentials from the preceding config.json file . For example:

    docker pull <image-name>

"manifest invalid" error when pushing images to Red Hat Quay

Symptom: You receive the following error when pushing images to a Red Hat Quay image registry as part of Mirroring images for an air-gapped OpenShift cluster:

manifest invalid: manifest invalid info: Mirroring completed in 1m35.45s (0B/s) error: one or more errors occurred while uploading images

Cause: Some images in the ibm-integration-platform-navigator and ibm-integration-asset-repository operator packages contain uncompressed layers. These layers are not supported on Red Hat Quay.

Solution: The issue has been fixed in the IBM Cloud Pak for Integration operator v. 7.3.6 and above, and IBM Cloud Pak foundational services operator v. 1.7.5 and above. Use an operator package version with the fix to mirror the images.

Unable to mirror images because the image name contains too many path separators

Cause: The local registry (such as GitLab) does not support enough path separators in the image name.

Solution: Compress the path segments by adding one of the following parameters to the oc ibm-pak generate command.

  • If you are using a bastion host, add the following parameter:

    --max-components <integer> indicates the maximum number of path components allowed on the target registry. Enter an integer value of 2 or more (paths compressed from right to left per the integer provided).

    For example:

    oc ibm-pak generate mirror-manifests $OPERATOR_PACKAGE_NAME --version $OPERATOR_VERSION --max-components 3 $TARGET_REGISTRY

  • If you are using a portable compute or storage device, add one or more of the following parameters:

    --max-components <integer> indicates the maximum number of path components allowed on the portable compute or storage device. Enter an integer value of 2 or more (paths compressed from right to left per the integer provided). This parameter is only required if the portable compute or storage device also has a restriction on the number of path separators supported.

    --final-max-components <integer> indicates the maximum number of path components allowed on the final registry. Enter an integer value of 2 or more (paths are compressed from right to left per the integer provided).

    For example:

    oc ibm-pak generate mirror-manifests $OPERATOR_PACKAGE_NAME --version $OPERATOR_VERSION file://integration --final-max-components 3 --final-registry $TARGET_REGISTRY