Troubleshooting common air gap problems

You might encounter an issue when you install or run WebSphere Automation in an airgap environment.

Signature validation error during CASE download

Downloading a CASE via oc ibm-pak get fails with the following error:

Error: Unable to validate a signature object for the CASE ...

or

Error: Unable to validate a signature object for the CASE XXXX: unable to validate the signature against any known public keys. This may mean that the CASE has been modified or tampered with or that it was signed with a newer version of ibm-pak

This problem occurs because the CASE has been signed with a new code signing certificate.

To resolve the issue, download and install the most recent version of IBM Catalog Management Plug-in for IBM Cloud Paks from IBM/ibm-pak External link icon:
oc ibm-pak get

Communication issue due to too many requests error (HTTP 429) while image mirroring

The oc image mirror command run fails with the general error message one or more errors occurred.

It happens for the mirror installation using portable device scenario that mirrors images to local filesystem command and uses the --continue-on-error=true.

Example of oc image mirror command which standard error output is redirected to my-mirror-progress.txt file:

nohup oc image mirror \
-f $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-to-filesystem.txt \
-a $REGISTRY_AUTH_FILE \
--filter-by-os '.*' \
--insecure \
--skip-multiple-scopes \
--max-per-registry=1 \
--continue-on-error=true > my-mirror-progress.txt 2>&1 &

Look for the following message at the end of output printed to the console: 

info: Mirroring completed in 2h33m43.97s (16.81MB/s)
error: one or more errors occurred

Mirror errors are caused by the communication issues with the repository, as indicated by the HTTP 429 error.

The issue is with Red Hat oc image mirror function to mirror to filesystem (OCP 4.8). What can be seen in logs, when 429 error is received from registry, the layer is incorrectly flagged as having been mirrored and therefore not picked up by subsequent mirror requests. This was seen when using --continue-on-error which if an error occurs, continues work and attempts to mirror as much as possible.

To resolve this problem, do the following steps:

  1. Check if OCP version is 4.12 or less. Starting from OCP 4.13 this problem should not occur. If the user has noticed the similar behavior, but has OCP 4.13 or higher, should search for a different cause of those errors.
  2. Confirm the cause by searching for HTTP 429 error or toomanyrequests statements in the standard error output. If you redirected output to the my-mirror-progress.txt file by including > my-mirror-progress.txt 2>&1 & at the end of the command, open the my-mirror-progress.txt file and search for HTTP 429 or toomanyrequests statement. Check if the output contains one of the following lines:
    error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: toomanyrequests

error: unable to open source layer sha256:... to copy to file://cpfs/cp/ai/image-name: toomanyrequests

    If there are one or more lines containing these statements, proceed to the next step. If nothing is found, then the problem is not due to too many requests, and another cause should be considered.

  3. Create a list of images that failed, by searching in returned output lines for different image-name values. Depending on what was found in the previous step. Lines should be searched according to one of the two patterns:
    error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: error parsing HTTP 429 response body: unexpected end of JSON input: ""

    or

    error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: toomanyrequests

    Rerun the oc image mirror command with the --force option for all images that failed. Then, check if images-mapping-to-filesystem.txt file contains missing images. This file might be found in ./ibm-pak/data/mirror/case/version/images-mapping-to-filesystem.txt.

    Analogously, this process can be carried out in case of mirroring from file system. Run the oc image mirror command and validate the images-mapping-from-filesystem.txt file.

Change docker.io image references during mirror images

Generate mirror manifests for a CASE with oc ibm-pak generate mirror-manifests command gives the following warning:

One or more images will be mirrored from Docker Hub (docker.io). If you want to mirror those images from the IBM Container Registry instead, see https://ibm.biz/docker-ref-to-icr

This problem occurs because the CASE references images from docker.io registry.

To resolve the problem, update references to docker.io/ibmcom to icr.io/cpopen in the generated image mapping file. Run the following command to create a backup file for the original image mapping file (for example, images-mapping.txt.backup for connected mirroring scenario) and update references to docker.io/ibmcom to icr.io/cpopen.

Use the original file with replaced contents in the oc image mirror command.

For connected mirroring
cd $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION
sed -i.backup  's/docker.io\/ibmcom/icr.io\/cpopen/g' images-mapping.txt
For disconnected mirroring
cd $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION
sed -i.backup  's/docker.io\/ibmcom/icr.io\/cpopen/g' images-mapping-to-filesystem.txt
For intermediate registry mirroring
cd $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION
sed -i.backup  's/docker.io\/ibmcom/icr.io\/cpopen/g' images-mapping-to-registry.txt