Mirroring images directly to the private container registry

If your client workstation can connect to the internet and to the private container registry, you can mirror the images directly to your private container registry.

If your client workstation cannot connect to the internet and to the private container registry, see Mirroring images using an intermediary container registry.

Installation phase
  • You are not here. Setting up a client workstation
  • You are not here. Setting up a cluster
  • You are not here. Collecting required information
  • You are not here. Preparing to run installs in a restricted network
  • You are here icon. Preparing to run installs from a private container registry
  • You are not here. Preparing the cluster for Cloud Pak for Data
  • You are not here. Preparing to install an instance of Cloud Pak for Data
  • You are not here. Installing an instance of Cloud Pak for Data
  • You are not here. Setting up the Cloud Pak for Data control plane
  • You are not here. Installing solutions and services
Who needs to complete this task?

Registry administrator A registry administrator or a user with permissions to push images to the private container registry must complete this task.

When do you need to complete this task?

If you want to pull images from a private container registry, you must complete this task before you install Cloud Pak for Data.

  • One-time setup With careful planning, you can identify all of the components that you plan to install on the cluster so that you can complete this task once.
  • Repeat as needed However, if you decide to install additional services and the images are not in your private container registry, you might need to complete this task multiple times.

Before you begin

Best practice: You can run the commands in this task exactly as written if you set up environment variables. For instructions, see Setting up installation environment variables.

Ensure that you source the environment variables before you run the commands in this task.

Before you mirror the images to the private container registry, ensure that you have sufficient space for the images that you plan to mirror. For information about approximate image sizes, see Hardware requirements.

About this task

Use the cpd-cli manage commands to mirror the images from the IBM Entitled Registry to the private container registry.

The following steps assume that you will mirror all of the components in a single step. The components that are mirrored are determined by the ${COMPONENTS} variable, from the installation environment variables script. If you want to mirror a specific component instead of multiple components, you can export COMPONENTS with the appropriate component ID.

Procedure

  1. Log in to the IBM Entitled Registry registry:
    cpd-cli manage login-entitled-registry \
    ${IBM_ENTITLEMENT_KEY}
  2. Log in to the private container registry.

    The following command assumes that you are using private container registry that is secured with credentials:

    cpd-cli manage login-private-registry \
    ${PRIVATE_REGISTRY_LOCATION} \
    ${PRIVATE_REGISTRY_PUSH_USER} \
    ${PRIVATE_REGISTRY_PUSH_PASSWORD}
    If your private registry is not secured omit the following arguments:
    • ${PRIVATE_REGISTRY_PUSH_USER}
    • ${PRIVATE_REGISTRY_PUSH_PASSWORD}
  3. Confirm that you have access to the images that you want to mirror from the IBM Entitled Registry:
    1. Inspect the IBM Entitled Registry.
      Tip: If you want to validate that you have access to the images for a specific component, you can run the following command before you run the list-images command:
      export COMPONENTS=<component-ID>

      The list-images command downloads the CASE packages for the specified components. By default, the packages are downloaded from github.com/IBM. If you cannot access GitHub, add the following option to the command to download the packages from the IBM Entitled Registry: --from_oci=true.

      cpd-cli manage list-images \
      --components=${COMPONENTS} \
      --release=${VERSION} \
      --inspect_source_registry=true

      The output is saved to the list_images.csv file in the work/offline/${VERSION} directory.

    2. Check the output for errors:
      grep "level=fatal" list_images.csv

      The command returns images that failed because of authorization errors or network errors.

  4. EDB Postgres Standard users only. If you purchased EDB Postgres Standard, run the following command to remove the EDB Postgres Enterprise images from the list of images that will be mirrored to the private container registry:
    Workstations that use the default cpd-cli-workspace/olm-utils-workspace/work directory
    sed -i -e '/edb-postgres-advanced/d' ./cpd-cli-workspace/olm-utils-workspace/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-cpd-edb/*/ibm-cpd-edb-*-images.csv
    Workstations that use the CPD_CLI_MANAGE_WORKSPACE environment variable
    sed -i -e '/edb-postgres-advanced/d' ${CPD_CLI_MANAGE_WORKSPACE}/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-cpd-edb/*/ibm-cpd-edb-*-images.csv
  5. Watson Machine Learning users only. If you are mirroring the images for Watson Machine Learning, you can choose which runtime images are mirrored.
    The default runtimes must be mirrored; however, you can decide whether you want to mirror the Runtime 23.1 images.

    Runtime 23.1 on 5.0.1 and later

    Starting in Version 5.0.1, the Runtime 23.1 images are not mirrored by default. If you plan to use the Runtime 23.1 images, you must explicitly mirror them by running the following command:

    cpd-cli manage mirror-images \
    --components=wml \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwmlRuntimes231

  6. Watson Studio users only. If you are mirroring the images for Watson Studio, you can choose which Watson Studio Runtimes images are mirrored.
    The default runtimes must be mirrored; however, you can decide whether you want to mirror the following images:
    GPU images

    The GPU images are mirrored by default. If you don't need GPU images, run the following command to remove them:

    Workstations that use the default cpd-cli-workspace/olm-utils-workspace/work directory
    sed -i -e '/gpu/d' ./cpd-cli-workspace/olm-utils-workspace/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv
    Workstations that use the CPD_CLI_MANAGE_WORKSPACE environment variable
    sed -i -e '/gpu/d' ${CPD_CLI_MANAGE_WORKSPACE}/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv

    Pre-trained NLP models

    The pre-trained NLP images are mirrored by default. If you don't need pre-trained natural language processing (NLP) models, run the following command to remove them:

    Workstations that use the default cpd-cli-workspace/olm-utils-workspace/work directory
    sed -i -e '/nlp/d' ./cpd-cli-workspace/olm-utils-workspace/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv
    Workstations that use the CPD_CLI_MANAGE_WORKSPACE environment variable
    sed -i -e '/nlp/d' ${CPD_CLI_MANAGE_WORKSPACE}/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv

    Runtime 23.1 on 5.0.1 and later

    Starting in Version 5.0.1, the Runtime 23.1 images are not mirrored by default. If you plan to use the Runtime 23.1 images, you must explicitly mirror them by running the following command:

    cpd-cli manage mirror-images \
    --components=ws_runtimes \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwsRuntimes231

  7. watsonx.ai users only. 5.0.1 or later If you are mirroring the images for watsonx.ai, you can choose which images are mirrored.

    For more information about the models that are provided with watsonx.ai, see Foundation models in watsonx.ai.

    Tip: The following commands enable you to download foundation model individually. Alternatively, you can specify a comma separated list of groups to download multiple foundation models at the same time.

    All models
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false

    allam-1-13b-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxAllam113bInstruct

    codellama-34b-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxCodellamaCodellama34bInstructHf

    elyza-japanese-llama-2-7b-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxElyzaJapaneseLlama27bInstruct

    flan-t5-xl-3b
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGoogleFlanT5xl

    flan-t5-xxl-11b
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGoogleFlanT5xxl

    flan-ul2-20b
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGoogleFlanul2

    granite-7b-lab
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite7bLab

    granite-8b-japanese
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite8bJapanese

    granite-13b-chat-v2
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite13bChatv2

    granite-13b-instruct-v2
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite13bInstructv2

    granite-20b-multilingual
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite20bMultilingual

    granite-3b-code-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite3bCodeInstruct

    granite-8b-code-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite8bCodeInstruct

    granite-20b-code-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite20bCodeInstruct

    granite-34b-code-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite34bCodeInstruct

    jais-13b-chat
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxCore42Jais13bChat

    llama-2-13b-chat
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMetaLlamaLlama213bChat

    llama-2-70b-chat
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMetaLlamaLlama370bChat

    llama2-13b-dpo-v7
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMncaiLlama213bDpov7

    llama3-8b-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMetaLlamaLlama38bInstruct

    llama3-70b-instruct
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMetaLlamaLlama370bInstruct

    merlinite-7b
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMistralaiMerlinite7b

    mixtral-8x7b-instruct-v01
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMistralaiMixtral8x7bInstructv01

    mixtral-8x7b-instruct-v01-q (Deprecated)
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxMistralaiMixtral8x7bInstructv01q

    mt0-xxl-13b
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxBigscienceMt0xxl

    ibm_slate_30m_english_rtrvr
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxSlate30mEnglishRtrvr

    ibm_slate_125m_english_rtrvr
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxSlate125mEnglishRtrvr

  8. watsonx Assistant users only. If you are mirroring the images for watsonx Assistant, you can choose which watsonx Assistant images are mirrored.

    Version 5.0.2
    The images for features that require GPUs, such as conversational skills and conversational search, are included by default. If you don't plan to enable features that require GPUs, run the following command to remove them.
    Important: Do not run this command if you plan to install other services with a dependency on the Inference foundation models (watsonx_ai_ifm) component. For information about which services have a dependency on this component, see Software requirements.
    Workstations that use the default cpd-cli-workspace/olm-utils-workspace/work directory
    sed -i -e '/operator\|image_name/!d' ./cpd-cli-workspace/olm-utils-workspace/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-watsonx-ai-ifm/*/ibm-watsonx-ai-ifm-*-images.csv
    Workstations that use the CPD_CLI_MANAGE_WORKSPACE environment variable
    sed -i -e '/operator\|image_name/!d' ${CPD_CLI_MANAGE_WORKSPACE}/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-watsonx-ai-ifm/*/ibm-watsonx-ai-ifm-*-images.csv

    Version 5.0.3 or later

    The images for features that require GPUs, such as conversational skills and conversational search, are not included by default. If you plan to enable features that require GPUs, you must run the following command to mirror the required model:

    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite13bChatv2

  9. watsonx Orchestrate users only. 5.0.1 or later You must explicitly mirror the image for features that require GPUs, such as such as conversational skills and conversational search.
    cpd-cli manage mirror-images \
    --components=watsonx_ai_ifm \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false \
    --groups=ibmwxGranite13bChatv2
  10. Mirror the images to the private container registry.
    Tip: Determine whether you need to modify the behavior of this command:
    • By default, this command mirrors only the images that are needed for your cluster architecture. If you want to mirror the images for all supported architectures, remove the --arch=${IMAGE_ARCH} option.
    • This command mirrors the images for all of the components that are specified in the ${COMPONENTS} environment variable. If you want to mirror images for a specific component, you can run export COMPONENTS=<component-ID> before you run the command.
    cpd-cli manage mirror-images \
    --components=${COMPONENTS} \
    --release=${VERSION} \
    --target_registry=${PRIVATE_REGISTRY_LOCATION} \
    --arch=${IMAGE_ARCH} \
    --case_download=false

    For each component, the command generates a log file in the work directory.

    Tip: Run the following command to print out any errors in the log files:
    grep "error" mirror_*.log
  11. Confirm that the images were mirrored to the private container registry:
    1. Inspect the contents of the private container registry:
      cpd-cli manage list-images \
      --components=${COMPONENTS} \
      --release=${VERSION} \
      --target_registry=${PRIVATE_REGISTRY_LOCATION} \
      --case_download=false

      The output is saved to the list_images.csv file in the work/offline/${VERSION} directory.

    2. Check the output for errors:
      grep "level=fatal" list_images.csv

      The command returns images that are missing or that cannot be inspected.

  12. watsonx Orchestrate users only. Run the following command to mirror the App Connect images to your private container registry:

    Docker
    cat <<EOF | docker exec --interactive olm-utils-v3 sh
    export IBMPAK_HOME="/tmp/work/offline/${VERSION}/"
    
    oc ibm-pak get ibm-appconnect \
    --version ${AC_CASE_VERSION} \
    --disable-top-level-images-mode
    
    oc ibm-pak generate mirror-manifests ibm-appconnect ${PRIVATE_REGISTRY_LOCATION} \
    --version ${AC_CASE_VERSION}
    
    oc image mirror -f /tmp/work/offline/${VERSION}/.ibm-pak/data/mirror/ibm-appconnect/${AC_CASE_VERSION}/images-mapping.txt \
    --max-per-registry=1 \
    --skip-multiple-scopes=true \
    --continue-on-error=true \
    --filter-by-os '.*' \
    -a /opt/ansible/.airgap/secrets/config.json
    
    EOF

    Podman
    cat <<EOF | podman exec --interactive olm-utils-v3 sh
    export IBMPAK_HOME="/tmp/work/offline/${VERSION}/"
    
    oc ibm-pak get ibm-appconnect \
    --version ${AC_CASE_VERSION} \
    --disable-top-level-images-mode
    
    oc ibm-pak generate mirror-manifests ibm-appconnect ${PRIVATE_REGISTRY_LOCATION} \
    --version ${AC_CASE_VERSION}
    
    oc image mirror -f /tmp/work/offline/${VERSION}/.ibm-pak/data/mirror/ibm-appconnect/${AC_CASE_VERSION}/images-mapping.txt \
    --max-per-registry=1 \
    --skip-multiple-scopes=true \
    --continue-on-error=true \
    --filter-by-os '.*' \
    -a /opt/ansible/.airgap/secrets/config.json
    
    EOF

Results

The images for your architecture are mirrored to the private container registry.
Note: Some components, such as the cpfs component, provide only multi-arch images. For components with multi-arch images, all of the images are mirrored to the private container registry.

What to do next

Now that you've mirrored the images to the private container registry, you are ready to complete Configuring an image content source policy for IBM Cloud Pak for Data software images.