Troubleshooting issues when mirroring images for an air-gapped cluster

Troubleshoot errors you may get when mirroring images.

In the final procedure for mirroring images, the command fails with an error

Symptom: In the final procedure, "Mirror images to final location and configure the cluster", the command fails with this error:

error: unable to upload blob sha256:** no basic auth credentials

Cause: The registry credentials that you provided are incorrect.

Solution: In section 3.2, step 1, confirm that the value of <TARGET_REGISTRY> includes the hostname and port for your registry server, and ensure that the username and password used for docker or podman login are correct.

Authorization error received while images are pulled from entitled registry

Symptom: Authorization error, unauthorized, while images are pulled from the entitled registry.

For example:

unauthorized: authentication required
 Unable to copy layer ******

Solution:

  • Confirm that your credentials are correct by logging in to the cp.icr.io registry with either docker or podman. In this example, the username is cp and the password is your assigned entitlement key:

    docker login cp.icr.io -u cp -p <your_entitlement_key>

  • Try to pull the image from the registry by using either docker or podman. For example:

    podman pull cp.icr.io/cp/***/***/***

    If you can pull the image, that confirmation that the image exists in the entitled registry.

  • Retry the failing command.

  • If the error still exists, see Finding and applying your entitlement key by using the UI (online installation) to confirm your entitlement credentials, or reach out to IBM Support.

Server misbehaving error when pulling an image

Symptom: You receive an error when pulling an image: server misbehaving.

Solution:

  • On one of the nodes, confirm that /host/etc/containers/registries.conf contains all the rules defined in ImageContentSourcePolicy (search for that string in the OpenShift web console). If /host/etc/containers/registries.conf does not contain those rules, copy them into the file.

  • Confirm that a pullSecret has been created in the openshift-config namespace. This is a global pull secret and contains a docker/podman login credential for the applicable registries, which gets copied onto the nodes. Credentials can be found in $HOME/.docker/config.json or <user_path>/kubelet/config.json. If the pullSecret does not exist, you may need to contact IBM Support.

  • On the same node, confirm that the mirroring and the credentials are working. Pull the image manually with docker or podman, using the credentials from the preceding config.json file . For example:

    docker pull <image-name>

Authorization error in the pod logs of the product for a capability (for example, API Connect or App Connect)

Symptom: Unauthorized: authentication required error in the pod logs of the product for a capability (for example, API Connect or App Connect) and not all pods are starting while installing.

Solution:

  • If the the operator group was created in a restricted namespace such as openshift-marketplace, create the operator group in a non-restricted namespace.

  • If you selected a restricted namespace to install the product, select a different namespace.