Mirroring images using an intermediary container registry

If your client workstation cannot connect to the internet and to the private container registry, you must mirror images to an intermediary container registry before you can mirror the images to your private container registry.

If your client workstation can connect to the internet and to the private container registry, see Mirroring images directly to the private container registry.

Installation phase
You are not here. Setting up a client workstation
You are not here. Setting up a cluster
You are not here. Collecting required information
You are not here. Preparing to run installs from a private container registry
You are here icon. Preparing the cluster for Cloud Pak for Data
You are not here. Preparing to install an instance of Cloud Pak for Data
You are not here. Installing an instance of Cloud Pak for Data
Who needs to complete this task?

Registry administrator A registry administrator or a user with permissions to push images to the private container registry must complete this task.

When do you need to complete this task?

If you want to pull images from a private container registry, you must complete this task before you install Cloud Pak for Data.

  • One-time setup With careful planning, you can identify all of the components that you plan to install on the cluster so that you can complete this task once.
  • Repeat as needed However, if you decide to install additional services and the images are not in your private container registry, you might need to complete this task multiple times.

Before you begin

Best practice: You can run the commands in this task exactly as written if you set up environment variables. For instructions, see Setting up installation environment variables.

Ensure that you source the environment variables before you run the commands in this task.

About this task

Use the cpd-cli manage commands to:
  • Mirror the images from the IBM® Entitled Registry to an intermediary container registry on the client workstation.
  • Mirror the images from the intermediary container registry to the private container registry.

The cpd-cli manage mirror-images command automatically sets up an intermediary container registry on the client workstation. The address of the intermediary container registry is 127.0.0.1:12443.

You must be able to move the intermediary container registry behind your firewall.

Procedure

  1. From a client workstation that can connect to the internet:
    1. Log in to the IBM Entitled Registry registry:
      cpd-cli manage login-entitled-registry \
      ${IBM_ENTITLEMENT_KEY}
    2. Confirm that you have access to the images that you want to mirror from the IBM Entitled Registry:
      1. Inspect the IBM Entitled Registry:
        Tip: If you want to validate that you have access to the images for a specific component, you can run the following command before you run the list-images command:
        export COMPONENTS=<component-ID>
        cpd-cli manage list-images \
        --components=${COMPONENTS} \
        --release=${VERSION} \
        --inspect_source_registry=true

        The output is saved to the list_images.csv file in the work/offline/${VERSION} directory.

      2. Check the output for errors:
        grep "level=fatal" list_images.csv

        The command returns images that failed because of authorization errors or network errors.

    3. Watson™ Studio users only. If you are mirroring the images for Watson Studio, you can choose which Watson Studio Runtimes images are mirrored.
      The default runtimes must be mirrored; however, you can optionally remove the following images if you don't plan to use them:
      GPU images

      If you don't need GPU images, run the following command to remove them:

      Workstations that use the default cpd-cli-workspace/olm-utils-workspace/work directory
      sed -i -e '/gpu/d' ./cpd-cli-workspace/olm-utils-workspace/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv
      Workstations that use the CPD_CLI_MANAGE_WORKSPACE environment variable
      sed -i -e '/gpu/d' ${CPD_CLI_MANAGE_WORKSPACE}/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv

      Pre-trained NLP models

      If you don't need pre-trained natural language processing (NLP) models, run the following command to remove them:

      Workstations that use the default cpd-cli-workspace/olm-utils-workspace/work directory
      sed -i -e '/nlp/d' ./cpd-cli-workspace/olm-utils-workspace/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv
      Workstations that use the CPD_CLI_MANAGE_WORKSPACE environment variable
      sed -i -e '/nlp/d' ${CPD_CLI_MANAGE_WORKSPACE}/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-wsl-runtimes/*/ibm-wsl-runtimes-*-images.csv

    4. EDB Postgres Standard users only. If you purchased EDB Postgres Standard, run the following command to remove the EDB Postgres Enterprise images from the list of images that will be mirrored to the private container registry:
      Workstations that use the default cpd-cli-workspace/olm-utils-workspace/work directory
      sed -i -e '/edb-postgres-advanced/d' ./cpd-cli-workspace/olm-utils-workspace/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-cpd-edb/*/ibm-cpd-edb-*-images.csv
      Workstations that use the CPD_CLI_MANAGE_WORKSPACE environment variable
      sed -i -e '/edb-postgres-advanced/d' ${CPD_CLI_MANAGE_WORKSPACE}/work/offline/${VERSION}/.ibm-pak/data/cases/ibm-cpd-edb/*/ibm-cpd-edb-*-images.csv
    5. Mirror the images to the intermediary container registry.

      The command automatically sets up an intermediary container registry on the client workstation. The address of the intermediary container registry is 127.0.0.1:12443.

      Tip: Determine whether you need to modify the behavior of this command:
      • By default, this command mirrors only the images that are needed for your cluster architecture. If you want to mirror the images for all supported architectures, remove the --arch=${IMAGE_ARCH} option.
      • This command mirrors the images for all of the components that are specified in the ${COMPONENTS} environment variable. If you want to mirror images for a specific component, you can run export COMPONENTS=<component-ID> before you run the command.
      cpd-cli manage mirror-images \
      --components=${COMPONENTS} \
      --release=${VERSION} \
      --target_registry=127.0.0.1:12443 \
      --arch=${IMAGE_ARCH}

      For each component, the command generates a log file in the work directory.

    6. Confirm that the images were mirrored to the intermediary container registry:
      1. Inspect the contents of the intermediary container registry:
        cpd-cli manage list-images \
        --components=${COMPONENTS} \
        --release=${VERSION} \
        --target_registry=127.0.0.1:12443 \
        --case_download=false

        The output is saved to the list_images.csv file in the work/offline/${VERSION} directory.

      2. Check the output for errors:
        grep "level=fatal" list_images.csv

        The command returns images that are missing or that cannot be inspected.

  2. Move the intermediary container registry behind the firewall.

    Options for moving the intermediary container registry
    Option Details
    Use a portable compute device, such as a laptop, that you can move behind your firewall.
    You can use the same device to:
    • Mirror images from the IBM Entitled Registry to the intermediary container registry.
    • Mirror images from the intermediary container registry to the private container registry.
    Use a portable storage device, such as a USB drive, that you can move behind your firewall.
    You must set up two client workstations:
    • A workstation that can connect to the internet. From this workstation, you can mirror the images from the IBM Entitled Registry to the intermediary container registry on the portable storage device.
    • A workstation that can connect to the private container registry. After you move the portable storage device to this workstation, you can mirror the images from the intermediary container registry to the private container registry.
    Important: The workstation that can connect to the private container registry must have:
    • A copy of the cpd-cli
    • A copy of the work directory, which contains the CASE packages and intermediary container registry.

      For simplicity, you can copy the entire cpd-cli-workspace or CPD_CLI_MANAGE_WORKSPACE directory.

    Use a file transfer protocol, such as scp or sftp, to move images behind your firewall.
    You must set up two client workstations:
    • A workstation that can connect to the internet. From this workstation, you can mirror the images from the IBM Entitled Registry to the intermediary container registry.
    • A workstation that can connect to the private container registry. After you transfer the intermediary container registry to this workstation, you can mirror the images from the intermediary container registry to the private container registry.
    Important: The workstation that can connect to the private container registry must have:
    • A copy of the cpd-cli
    • A copy of the work directory, which contains the CASE packages and intermediary container registry.

      For simplicity, you can copy the entire cpd-cli-workspace or CPD_CLI_MANAGE_WORKSPACE directory.


  3. From a client workstation that connect to private container registry:
    1. Log in to the private container registry.

      The following command assumes that you are using private container registry that is secured with credentials:

      cpd-cli manage login-private-registry \
      ${PRIVATE_REGISTRY_LOCATION} \
      ${PRIVATE_REGISTRY_PUSH_USER} \
      ${PRIVATE_REGISTRY_PUSH_PASSWORD}
      If your private registry is not secured omit the following arguments:
      • ${PRIVATE_REGISTRY_PUSH_USER}
      • ${PRIVATE_REGISTRY_PUSH_PASSWORD}
    2. Mirror the images from the intermediary container registry to the private container registry.
      Tip: Determine whether you need to modify the behavior of this command:
      • By default, this command mirrors only the images that are needed for your cluster architecture. If you want to mirror the images for all supported architectures, remove the --arch=${IMAGE_ARCH} option.
      • This command mirrors the images for all of the components that are specified in the ${COMPONENTS} environment variable. If you want to mirror images for a specific component, you can run export COMPONENTS=<component-ID> before you run the command.
      cpd-cli manage mirror-images \
      --components=${COMPONENTS} \
      --release=${VERSION} \
      --source_registry=127.0.0.1:12443 \
      --target_registry=${PRIVATE_REGISTRY_LOCATION} \
      --arch=${IMAGE_ARCH} \
      --case_download=false

      For each component, the command generates a log file in the work directory.

    3. Confirm that the images were mirrored to the private container registry:
      1. Inspect the contents of the intermediary container registry:
        cpd-cli manage list-images \
        --components=${COMPONENTS} \
        --release=${VERSION} \
        --target_registry=${PRIVATE_REGISTRY_LOCATION} \
        --case_download=false

        The output is saved to the list_images.csv file in the work/offline/${VERSION} directory.

      2. Check the output for errors:
        grep "level=fatal" list_images.csv

        The command returns images that are missing or that cannot be inspected.

Results

The images for your architecture are mirrored to the private container registry.
Note: Some components, such as the cpfs component, provide only multi-arch images. For components with multi-arch images, all of the images are mirrored to the private container registry.

What to do next

Now that you've mirrored the images to the private container registry, you are ready to Configuring an image content source policy for IBM Cloud Pak for Data software images.