Mirroring images with a bastion host

OpenShift-only contentIf your cluster is not connected to the internet, you can mirror images to a registry in your restricted network environment by using a bastion host.

This task must be performed by a Red Hat® OpenShift® administrator.

A bastion host is a device that has access to both the public internet and the restricted network environment where a local registry and Red Hat OpenShift Container Platform clusters reside. Using the bastion host, you can mirror your images directly to the local registry.

The process for mirroring images involves the following steps:

Troubleshooting: For a list of possible errors and solutions when completing these instructions, see Troubleshooting installation issues in an air-gapped environment.

1. Set up your mirroring environment

Before you install IBM App Connect on an air-gapped environment, you must set up a host that can be connected to the internet to mirror your environment.

To set up your mirroring environment, complete the following steps:

1.1. Prerequisites

  • Your cluster must be set up and running Red Hat OpenShift Container Platform 4.12, 4.14, 4.15, 4.16, 4.17, or 4.18.
  • You must have a Docker V2 registry that is available and accessible from the Red Hat OpenShift cluster nodes. (For more information, see 1.3. Set up the local image registry and access.)
  • The following sites and ports must be accessible:
    • icr.io:443 for entitled registry
    • quay.io:443 for IBM App Connect
    • github.com for CASEs and tools
    • redhat.com for Red Hat OpenShift upgrades
  • Storage must be available and configured on your cluster.

1.2. Prepare a bastion host for mirroring the images

Your bastion host must have access to a set of resources and command-line tools.

  • You must be able to connect your bastion host to the internet (to download the CASE and images) and to the air-gapped environment at the same time. In the air-gapped environment, the host must be able to access the Red Hat OpenShift cluster (to install the Operator) and the local registry (where the images will be mirrored).
  • Your bastion host must be on a Linux® x86_64 or Mac platform with any operating system that the Red Hat OpenShift CLI supports.
    Important: If you are using Windows, execute the actions in a Linux x86_64 VM or from a Windows Subsystem for Linux (WSL) terminal.
  • The following command-line tools are required.

    Software Purpose
    Docker Container management
    Podman Container management
    OpenShift Container Platform CLI (oc) Red Hat OpenShift administration
    oc ibm-pak oc IBM Catalog Management Plug-in for IBM Cloud Paks

To download and install the command-line tools on your bastion host, complete the following steps:

  1. Install Docker or Podman.
    • To install Docker (for example, on Red Hat Enterprise Linux), run the following commands:
      yum check-update
yum install docker
    • To install Podman, see Podman Installation Instructions in the Podman documentation.
  2. Install the OpenShift Container Platform CLI (oc) tool.

    For more information, see Getting started with the OpenShift CLI in the Red Hat OpenShift documentation. Use the version of oc that is provided for the Red Hat OpenShift version on which IBM App Connect will be installed.

  3. Download IBM Catalog Management Plug-in for IBM Cloud Paks 1.1.0 or later from GitHub. This plug-in allows you to run oc ibm-pak commands against the cluster.

    To confirm that ibm-pak is installed, run the following command.

    oc ibm-pak --help

    The output should return the oc ibm-pak usage details.

1.3. Set up the local image registry and access

Use a local Docker registry to store your images in your restricted network environment. If you already have one or more centralized, corporate registry servers that store production container images, you can use one of them as your local registry. If a registry is not already available, install and configure a production-grade registry.

The local registry must meet the following requirements:

  • Supports multi-architecture images through Docker Manifest V2, Schema 2.
  • Has sufficient storage to hold all the software that is to be transferred to the local registry.
  • Is accessible from the Red Hat OpenShift cluster nodes.
  • Allows path separators in the image name.
  • Supports auto-repository creation.
Important: Do not use OpenShift image registry as your local registry. The OpenShift image registry does not support multi-architecture images or path separators in the image name.

The user access requirements for this registry are as follows:

  • To access your registries during the mirroring process, you need user credentials that can write to the target local registry and create repositories.
  • To access your registries during run time, you need user credentials that can read all repositories from the target local registry. (The Red Hat OpenShift cluster uses these credentials.)

2. Set environment variables and download your CASE files

Before you mirror your images, set the required environment variables on your mirroring device and then download the corresponding CASE files.

You need to connect to the internet to download the files. Versions of the IBM App Connect CASE are available in the IBM CASE repository in the public GitHub location https://github.com/IBM/cloud-pak/tree/master/repo/case/ibm-appconnect.

To set environment variables for IBM App Connect and download the CASE files to your bastion host, complete the following steps:

  1. Create the following environment variables with the installer image name and the image inventory on your bastion host.
    export CASE_NAME=ibm-appconnect
export CASE_VERSION=12.11.1
    Tip: You can identify the latest version of the IBM App Connect CASE by accessing the index.yaml file in the IBM CASE repository on GitHub, and then noting the value of latestVersion.
  2. Connect your bastion host to the internet. You do not need to connect your host to the restricted network environment at this time.
  3. Download the CASE to your host. If you omit the --version parameter, the latest CASE version is downloaded.
    oc ibm-pak get $CASE_NAME --version $CASE_VERSION

    By default, the plug-in uses a root directory of ~/.ibm-pak, and the preceding command downloads the CASE under ~/.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION. The default location of the logs files is ~/.ibm-pak/logs/oc-ibm_pak.log.

Your bastion host is now configured and you are ready to mirror your images.

3. Mirror the images

The process of mirroring images pulls the image from the internet and pushes it to your local (private) registry.

Some of the App Connect images are in the IBM Entitled Registry (which requires authentication), and other images are in IBM Container Registry (which does not need any special credentials). You need to log in to the IBM Entitled Registry for source images, and also log in to your local registry that will be the target when images are copied over. After you mirror your images, you can configure your cluster and complete the air-gapped installation.

To mirror the images from your bastion host to your air-gapped environment, complete the following steps:

3.1. Generate mirror manifests

A mirror manifest is a YAML file that directs the ibm-pak plug-in what images to mirror, and where to mirror them.

To generate mirror manifests, complete the following steps:

  1. Define an environment variable named $TARGET_REGISTRY by running the following command, where <target_registry> is the registry to mirror images to, for access by the Red Hat OpenShift cluster.

    Specify <target-registry> in the format hostname:port - for example, 172.16.0.10:5000. 

    export TARGET_REGISTRY=<target_registry>

    If you want the images to use a specific namespace in the target registry, you can add a registry namespace suffix. For example, setting TARGET_REGISTRY to myregistry.com:5000/mynamespace will create manifests that cause images to be mirrored to the top-level namespace mynamespace.

  2. Generate mirror manifests.
    oc ibm-pak generate mirror-manifests $CASE_NAME --version $CASE_VERSION $TARGET_REGISTRY

This process generates the following files at ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION, which you will use later.

  • catalog-sources.yaml (stores the CatalogSource manifest)
  • image-content-source-policy.yaml (stores the ImageContentSourcePolicy manifest)
  • images-mapping.txt (provides input to the oc image mirror command, which copies or mirrors the images to the local registry)

3.2. Authenticate the entitled registry

Before you mirror the required images, configure authentication for the IBM Entitled Registry, which acts as a source image registry.

To authenticate to the entitled registry and store its authentication credentials, complete the following steps:

  1. Obtain an entitlement key, which is required to pull App Connect images from the IBM Entitled Registry.
    1. Log in the IBM Container software library with the IBMid and password that are associated with the entitled software.
    2. To obtain a new key, click Get entitlement key or Add new key, and then click Copy.

      If you have an existing key to use, click Copy.

    3. Save the entitlement key for use later.
  2. Export the path to the authentication file that a podman login or docker login command generates when run. This file will store the authentication credentials that you specify in the next step.


    For a docker login command, the authentication file is typically located at $HOME/.docker/config.json on Linux or %USERPROFILE%/.docker/config.json on Windows. For a podman login command, the default location is ${XDG_RUNTIME_DIR}/containers/auth.json, which might be substituted with $HOME/.docker/config.json if the credentials are deemed invalid when read (as described here).

    export REGISTRY_AUTH_FILE=$HOME/.docker/config.json
  3. Log in to the cp.icr.io registry by using the podman login command (as shown in the following example), or the equivalent docker login command. 
    podman login cp.icr.io
    When prompted, specify cp as your username and your entitlement key (which you copied earlier) as the password. (You can optionally run the command with the -u and -p parameters, as shown in the following example.) 
    podman login cp.icr.io -u cp -p myEntitlementKey

    The output should indicate that your login succeeded.

3.3. Authenticate the local registry

Before you mirror the required images, also configure authentication for the local registry that is intended to be your target registry.

Log in to the local registry by using the podman login command (as shown in the following example), or the equivalent docker login command. 

podman login $TARGET_REGISTRY

When prompted, specify your username and password. Use an account that can write images to the local registry. (You can optionally run the command with the -u and -p parameters.) These credentials are similarly stored in the authentication file.

The output should indicate that your login succeeded.

3.4. Mirror the images to the local registry

To mirror the images, your bastion host must be connected to both the internet and the restricted network environment that contains the local registry.

Copy the images to the local registry by running the following command. If the local registry is not secured by TLS, or the certificate that the local registry presents is not trusted by your device, add the --insecure option to the command.

oc image mirror \
  -f ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping.txt \
  -a $REGISTRY_AUTH_FILE \
  --filter-by-os '.*' \
  --skip-multiple-scopes \
  --max-per-registry=1

The oc image mirror command starts by planning what images and layers need to be transferred. It can take a couple of minutes before you see output.

3.5. Configure the cluster

To enable your offline cluster to access images from your local registry, configure the cluster to use your local registry as a mirror of the images that are hosted in the online registries. Also ensure that the cluster can access those images.

To configure your cluster, complete the following steps:

  1. From the command line, log in to your Red Hat OpenShift cluster by using the oc login command.
  2. Update the global image pull secret for your Red Hat OpenShift cluster, to allow it to access the local registry.

    To do so, you need to add the username and password that you used to authenticate to the local registry earlier. Follow the steps in Updating the global cluster pull secret in the Red Hat OpenShift documentation. These steps enable your cluster to have proper authentication credentials in place to pull images from your TARGET_REGISTRY, as specified in the image-content-source-policy.yaml file, which you will apply to your cluster in the next step.

  3. Create an ImageContentSourcePolicy resource to enable listed images in the CASE to be loaded from the local registry.
    oc apply -f  ~/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml

    Applying ImageContentSourcePolicy causes all worker nodes in the cluster to restart, and might take over 10 minutes to complete.

    Verify that the ImageContentSourcePolicy resource is created.

    oc get imageContentSourcePolicy

    Verify your cluster node status and wait for all nodes to be updated before proceeding.

    oc get MachineConfigPool -w