IBM Support

How to check images available on IBM Cloud Container Registry (icr.io)?

Question & Answer


Question

I am getting an image pull error for multiple pods. How can I check if the images are available on the IBM Cloud Container Registry ( icr.io)? 
 

Cause

  • Firewall-related issues can cause access issues to the online registry. 
  • Old operator versions might not be available at the IBM Cloud Container Registry 

Symptoms:

NAMESPACE                  NAME                                                    READY   STATUS                  RESTARTS   AGE     IP          
ibm-common-services        auth-idp-5fd8445784-hzs5q                               0/4     Init:ImagePullBackOff   0          18h     x.x.x.x
ibm-common-services        auth-pap-7cf47b7bf-t2lzp                                0/2     ImagePullBackOff        4          90d     x.x.x.x
ibm-common-services        auth-pdp-7ff7fff6dd-rgkwq                               0/2     Init:ImagePullBackOff   4          86d     x.x.x.x
ibm-common-services        common-web-ui-7558d97c4d-qrj99                          0/1     ImagePullBackOff        0          12d     x.x.x.x
ibm-common-services        ibm-cert-manager-operator-6f8d8b66dd-lzt4b              0/1     ImagePullBackOff        4          86d     x.x.x.x 
ibm-common-services        ibm-ingress-nginx-operator-6558b9dc59-xvxq8             0/1     ImagePullBackOff        0          18h     x.x.x.x 
ibm-common-services        ibm-management-ingress-operator-7649bb756c-566d5        0/1     ImagePullBackOff        0          18h     x.x.x.x
In the description you find timeouts accessing the registry kubelet:
Failed to pull image "icr.io/cpopen/ibm-operator-catalog:latest": rpc error: code = Unknown desc = error pinging docker registry icr.io: Get "https://icr.io/v2/": dial tcp x.x.x.x:443: i/o timeout

Answer

Before you start
If skopeo is not on your operating system, install it by using the following command: 
yum install skopeo -y
Steps
Check whether the image is available in the IBM Cloud registry using the skopeo tool. 
  1. Use the error message to identify the image that is having the timeout errors.
  2. SSH to the Red Hat Server in the same network as the OpenShift Cluster cluster with the image pull issues.
  3. Test the connection to icr.io using an example like the following. Search for the image having the errors. In this example, the image is icr.io/cpopen/ibm-operator-catalog: 
    skopeo list-tags docker://icr.io/cpopen/ibm-operator-catalog
    Example output: 
    {
    "Repository": "icr.io/cpopen/ibm-operator-catalog",
    "Tags": [
        "fbc-latest",
        "latest",
        "sqlite-latest",
        "v1.20-20211014.235635-B855DDD56",
        "v1.20-20211015.154619-5A967C7ED",
        ...
    ]
}
    Important: Some old images might not be available at the icr.io registry; compare the available tags between the icr.io and quay.io registry. The following tags available for the following operator on the icr.io registry:
    skopeo list-tags docker://icr.io/cpopen/cpfs/icp-management-ingress
{
    "Repository": "icr.io/cpopen/cpfs/icp-management-ingress",
    "Tags": [
        "2.5.23",
        "2.5.24",
        "2.5.25",
        "2.6.0",
        "2.6.1"
    ]
}
    Compare the tags available at quay.io.
    skopeo list-tags docker://quay.io/opencloudio/icp-management-ingress
    Example output: 
    {
    "Repository": "quay.io/opencloudio/icp-management-ingress",
    "Tags": [
        "2.5.10",
        "2.5.11",
        "2.5.12",
        "2.5.13",
        "2.5.14",
        "2.5.15",
        "2.5.16",
        "2.5.17",
        "2.5.18",
        "2.5.19",
        "2.5.20",
        "2.5.21",
        "2.5.22",
        "2.5.23",
        "2.5.24"
    ]
}
Alternative steps:
You can also find the Digest for an image by using the "inspect" option. 
skopeo inspect  docker://icr.io/cpopen/ibm-common-service-catalog:3.19.3 | egrep -i "Digest"
        "Digest": "sha256:c8d461f5d9f933cbae94e84dd28643f0874a58a04d56de23181610c65c473314", 

skopeo inspect  docker://icr.io/cpopen/ibm-cpd-cloud-native-postgresql-operator-catalog@sha256:f3feb39b0dfb444a46d99a9cd160c616f8548d84d452cbc89e4985f0a2574486 | grep version
        "io.buildah.version": "1.24.2",
        "sample.ibm.com.catalog.version": "1.14.0",
        "version": "1.14.0"
Result:
A successful result looks similar to the following: 
{
    "Repository": "icr.io/cpopen/cpfs/icp-management-ingress",
    "Tags": [
        "2.5.23",
         ...
    ]
}
An unsuccessful result looks similar to the following: 
FATA[0000] Error listing repository tags: Error fetching tags list: invalid status code from registry 404 (Not Found)

[{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS8QTD","label":"IBM Cloud Pak for Integration"},"ARM Category":[{"code":"a8m0z0000001iM4AAI","label":"FAQ"}],"ARM Case Number":"TS010494525","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB45","label":"Automation"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBYVB","label":"IBM Cloud Pak for Business Automation"},"ARM Category":[{"code":"a8m3p000000PCINAA4","label":"Operate-\u003EBAW Install\\Upgrade\\Setup-\u003EPFS"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSTDPP","label":"IBM Cloud Pak for Security"},"ARM Category":[{"code":"a8m0z0000001h8uAAA","label":"Install \/ Upgrade"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
16 September 2022

UID

ibm16618197

Page Feedback