Troubleshooting and known issues in offline mirroring

Issues can occur during the offline mirroring of images. This topic outlines known issues and offers troubleshooting guidance.

Use the latest versions of the oc and oc-mirror tools

Problem statement
Issues may occur when you do not use the latest version of the oc CLI or oc-mirror plug in.
Diagnosis
Red Hat® has identified several library compatibility issues in the older versions of these tools, which are resolved in the latest releases. When you run older versions of oc or oc-mirror commands, you may encounter messages similar to the following examples.
oc: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by oc) 
oc: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by oc) 
oc: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by oc)
Resolution
Download the latest available oc CLI tool, even if it is a newer than your Red Hat OpenShift® version. The oc CLI is backward compatible with older OpenShift versions. For example, your oc CLI can be on version 4.17 while your OpenShift cluster remains on 4.14.

In addition, ensure you obtain the latest version of the oc-mirror plugin. The oc-mirror plugin need not match the oc CLI version and can even be newer. For example, you can use an oc-mirror plugin 4.17 with an oc CLI 4.16, even if your cluster is on 4.14.

Login to both source and target registries before mirroring images

Problem statement
You may encounter issues where some or all images fail to copy, resulting in errors related to either authorization or authentication requirements.
Diagnosis
If you encounter messages similar to the following examples, then it indicates that you must login to the specified registries:
  • error: unable to retrieve source image cp.icr.io/cp/isf/isf-prereq-operator manifest 
sha256:1d062784dd6818297bd8275d139d0dd7ac6cd2ae6a4188bb3258bcdad8ccedf4: 
Get "https://cp.icr.io/v2/cp/isf/isf-prereq-operator/manifests/sha256:1d062784dd6818297bd8275d139d0dd7ac6cd2ae6a4188bb3258bcdad8ccedf4": 
unauthorized: Authorization required. See https://cloud.ibm.com/docs/Registry?topic=Registry-troubleshoot-auth-req
  • Checking push permissions for docker-na-public.artifactory.swg-devops.com
error: error checking push permissions for docker-na-public.artifactory.swg-devops.com: 
creating push check transport for docker-na-public.artifactory.swg-devops.com failed: 
GET https://docker-na-public.artifactory.swg-devops.com/artifactory/api/docker/null/v2/token?scope=repository%3Aoffline%2Foc-mirror%3Apush%2Cpull&service=docker-na-public.artifactory.swg-devops.com: 
Authentication is required
Resolution
As a resolution, log in to all the registries:
  • registry.redhat.io
  • icr.io
  • quay.io
  • Your target registry host

If authorization errors persist even after logging, it indicates that your user does not have the right permissions to pull from the source registries or push to target registries. In such a case, contact your registry administrator regarding your user permissions

Wrong product version

Problem statement
If you use ibm-pak with a IBM Fusion version earlier than 2.9.0, services like Backup & Restore and IBM Data Cataloging do not get mirrored correctly although ibm-pak works and the base IBM Fusion gets mirrored correctly.
Resolution
For previous IBM Fusion versions, see the appropriate official offline mirroring documentation. Use ibm-pak only from IBM Fusion 2.9.0 or higher.

ibm-pak generate command does not generate an oc mirror command

Problem statement
If you encounter messages similar to the following example after you run the oc ibm-pak generate, it indicates that you have not configured ibm-pak to use the oc-mirror tool: 
- To mirror the images: oc image mirror -f /root/.ibm-pak/data/mirror/ibm-spectrum-fusion-sds/2.9.0+20241211.110649/images-mapping.txt --filter-by-os '.*' -a $REGISTRY_AUTH_FILE --insecure --skip-multiple-scopes --max-per-registry=1
Resolution
  1. Run the following command before you run any other oc ibm-pak command:
    oc ibm-pak config mirror-tools –e oc-mirror
  2. Run the oc ibm-pak generate command again to get the proper oc-mirror command.

Invalid mirror sequence error message

Problem statement
You may observe invalid mirror sequence error message.
Diagnosis
You may observe error messages similar to the following examples: 
oc adm catalog mirror 
file://cpopen/idp-server-operator-catalog@sha256:0b5de81594fb12075c3d3617a2ae938a12f8c2070bf95686b5df4b64e876a972 
REGISTRY/REPOSITORY error image list [] error: invalid mirror sequence order, want 1, got 2

or

oc adm catalog mirror 
file://cpopen/isf-operator-catalog@sha256:f34ee5806ac6a8c5fbcb50b6ccd979a245a0781022a8f8bfb2df9a4ba8658d41 
REGISTRY/REPOSITORY error image list [] error: invalid mirror sequence order, want 1, got 4
These error messages indicate a corrupted backend storage. When oc-mirror runs, it stores metadata related to the mirror operation in a backend storage directory. With ibm-pak, this backend storage directory is in ~/.ibm-pak/oc-mirror-storage.

This directory is useful when you want to resume a previous oc-mirror job that may have failed due to transient errors. However, if you use the same backend storage directory for mirroring an entirely different ImageSetConfig, you may get the invalid mirror sequence error message.

Resolution
Delete ~/.ibm-pak/oc-mirror-storage directory and retry the oc-mirror command.

Check available disk space

Problem statement
If repeated retries of oc-mirror continue to fail, it is possible that the target registry has run out of disk space. Additionally, if you mirror to a local directory for offline install media, your local disk may be full.
Resolution
Check disk space usage and free up disk space if necessary.

Error rebuilding catalog images error

Problem statement
The oc-mirror operation may fail with an error message similar to the following example:
Rendering catalog image 
"our.offline-registry.net/fusion-hci-2.9.0/cpopen/isf-operator-catalog:f34ee5" with file-based catalog 
error: error rebuilding catalog images from file-based catalogs: 
error regenerating the cache for our.offline-registry.net /fusion-hci-2.9.0/cpopen/isf-operator-catalog:f34ee5: exit status 1
Resolution
This is a known issue in the oc-mirror tool. For more information about how to resolve the issue, see the Red Hat article.

Error with HTTP status: 502 Bad Gateway

Problem statement
The oc-mirror operation may fail with an error message similar to the following example:
error: unable to retrieve source image registry.redhat.io/amq-streams/kafka-34-rhel8 manifest sha256:e4b2073b3001502b3bd02275eecaf1bfd6bf688c3f138da777eff57865337e85: received unexpected HTTP status: 502 Bad Gateway
error: unable to retrieve source image registry.access.redhat.com/ubi8/openjdk-11 manifest sha256:d9739b266415be7c915daf3c61f8f1cc4e14d0b237a0ab664693bc2e0d3577e4: name unknown: Repo not found
error: an error occurred during planning
Resolution
As a resolution, retry the oc-mirror command.