Communication issue due to too many requests error (HTTP 429) while image mirroring
oc image mirror command run fails with the general error message
one or more errors occurred. It happens for the mirror installation using portable device scenario that mirrors images to local filesystem command and
oc image mirror command which standard error output is redirected to my-mirror-progress.txt file:
nohup oc image mirror \ -f $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-to-filesystem.txt \ -a $REGISTRY_AUTH_FILE \ --filter-by-os '.*' \ --insecure \ --skip-multiple-scopes \ --max-per-registry=1 \ --continue-on-error=true > my-mirror-progress.txt 2>&1 &
After running the preceding command, the user receives a following message at the end of output printed to the console:
info: Mirroring completed in 2h33m43.97s (16.81MB/s) error: one or more errors occurred
Mirror errors are caused by the communication issues with the repo - that is indicated by the HTTP 429 error.
The issue is with Red Hat oc image mirror function to mirror to filesystem (OCP 4.8). What can be seen in logs, when 429 error is received from registry, the layer is incorrectly flagged as having been mirrored & therefore NOT picked up by subsequent mirror requests. This was seen when using --continue-on-error which if an error occurs, continues work and attempts to mirror as much as possible.
Check if OCP version is 4.12 or less. Starting from OCP 4.13 this problem should not occur. If the user has noticed the similar behavior, but has OCP 4.13 or higher, should search for a different cause of those errors.
Confirm the cause by searching for
HTTP 429error or
toomanyrequestsstatements in the standard error output. If the command output was redirected to my-mirror-progress.txt file by adding
> my-mirror-progress.txt 2>&1 &at the end of command, open the my-mirror-progress.txt file and search for
toomanyrequestsstatement. Check if output contains one of the following lines:
error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: error parsing HTTP 429 response body: unexpected end of JSON input: "" error: unable to open source layer sha256:... to copy to file://cpfs/cp/ai/image-name: error parsing HTTP 429 response body: unexpected end of JSON input: ""
or (for OCP 4.9 or higher):
error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: toomanyrequests error: unable to open source layer sha256:... to copy to file://cpfs/cp/ai/image-name: toomanyrequests
If there are one or more lines containing the preceding statements, follow next step of this guide. However, if nothing is found, then the problem is not due to too many requests error and another cause should be considered.
Create a list of images that failed, by searching in returned output lines for different image-name values. Depending on what was found in the previous step. Lines should be searched according to one of the two patterns
error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: error parsing HTTP 429 response body: unexpected end of JSON input: ""
error: unable to push icr.io/cpopen/image-name: failed to retrieve blob sha256:...: toomanyrequests
At the end rerun the
oc image mirror command with
--force option for all images that failed, then check if
images-mapping-to-filesystem.txt file (could be found in
contains missing images.
Analogously, the preceding process can be carried out in case of mirroring from filessytem, then at the end the
images-mapping-from-filesystem.txt should be given as
oc image mirror argument and validated at the end.