Communication issue due to too many requests error (HTTP 429) while image mirroring

Symptom

The oc image mirror command run fails with the general error message one or more errors occurred. It happens for the mirror installation using portable device scenario that mirrors images to local filesystem command and uses the --continue-on-error=true.

Example of oc image mirror command which standard error output is redirected to my-mirror-progress.txt file:

  nohup oc image mirror \
  -f $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-to-filesystem.txt \
  -a $REGISTRY_AUTH_FILE \
  --filter-by-os '.*' \
  --insecure \
  --skip-multiple-scopes \
  --max-per-registry=1 \
  --continue-on-error=true > my-mirror-progress.txt 2>&1 &

After running the preceding command, the user receives a following message at the end of output printed to the console:

  info: Mirroring completed in 2h33m43.97s (16.81MB/s)
  error: one or more errors occurred

Cause

Mirror errors are caused by the communication issues with the repo - that is indicated by the HTTP 429 error.

The issue is with Red Hat oc image mirror function to mirror to filesystem (OCP 4.8). What can be seen in logs, when 429 error is received from registry, the layer is incorrectly flagged as having been mirrored & therefore NOT picked up by subsequent mirror requests. This was seen when using --continue-on-error which if an error occurs, continues work and attempts to mirror as much as possible.

Solution

Check if OCP version is 4.12 or less. Starting from OCP 4.13 this problem should not occur. If the user has noticed the similar behavior, but has OCP 4.13 or higher, should search for a different cause of those errors.
Confirm the cause by searching for HTTP 429 error or toomanyrequests statements in the standard error output. If the command output was redirected to my-mirror-progress.txt file by adding > my-mirror-progress.txt 2>&1 & at the end of command, open the my-mirror-progress.txt file and search for HTTP 429 or toomanyrequests statement. Check if output contains one of the following lines:
```
error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: error parsing HTTP 429 response body: unexpected end of JSON input: ""

error: unable to open source layer sha256:... to copy to file://cpfs/cp/ai/image-name: error parsing HTTP 429 response body: unexpected end of JSON input: ""
```

or (for OCP 4.9 or higher):

  error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: toomanyrequests

  error: unable to open source layer sha256:... to copy to file://cpfs/cp/ai/image-name: toomanyrequests

If there are one or more lines containing the preceding statements, follow next step of this guide. However, if nothing is found, then the problem is not due to too many requests error and another cause should be considered.

Create a list of images that failed, by searching in returned output lines for different image-name values. Depending on what was found in the previous step. Lines should be searched according to one of the two patterns
```
error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: error parsing HTTP 429 response body: unexpected end of JSON input: ""
```

  error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: toomanyrequests

At the end rerun the oc image mirror command with --force option for all images that failed, then check if images-mapping-to-filesystem.txt file (could be found in ./ibm-pak/data/mirror/case/version/images-mapping-to-filesystem.txt) contains missing images.

Analogously, the preceding process can be carried out in case of mirroring from filesystem, then at the end the images-mapping-from-filesystem.txt should be given as oc image mirror argument and validated at the end.