Mirroring images takes the image from the internet to your mirroring device, then effectively copies that image on to your air-gapped environment. After you mirror your images, you can configure your cluster and complete the air-gapped installation. 

In these steps, your mirroring device is connected to the internet and disconnected from your air-gapped environment.

Procedure

Set the $TARGET_REGISTRY environment variable to the IP address or FQDN and the port for your target registry. The target registry is the registry where your images are mirrored to and accessed by the Red Hat OpenShift Container Platform cluster.

export TARGET_REGISTRY=<target_registry>

Verify access to the target registry.

podman login $TARGET_REGISTRY

Generate the mirror manifests to use when you mirror the images to the target registry.

oc ibm-pak generate mirror-manifests $CASE_NAME file://cp4s_images --version $CASE_VERSION --final-registry $TARGET_REGISTRY

The following files are generated in the $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION directory

• images-mapping-to-filesystem.txt
• images-mapping-from-filesystem.txt
• image-content-source-policy.yaml

These files are used when you mirror the images to your target registry.

ls -la  $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION

Tip: If you want to view the list of images to be mirrored, type the following command:

oc ibm-pak describe $CASE_NAME --version $CASE_VERSION --list-mirror-images | less

Store the authentication credentials for the IBM Entitled Registry, cp.icr.io.

• If you are using Podman, store authentication credentials for cp.icr.io by typing the following commands:

  export REGISTRY_AUTH_FILE=$IBMPAK_HOME/.ibm-pak/auth.json
  podman login cp.icr.io -u cp

• If you are using Docker, store authentication credentials for cp.icr.io by typing the following commands.

  export REGISTRY_AUTH_FILE=$IBMPAK_HOME/.docker/config.json
  docker login cp.icr.io -u cp

The password is your IBM Entitled Registry key.

The command stores and caches the registry credentials in the location that is specified for the $REGISTRY_AUTH_FILE environment variable.

Store the authentication credentials for your TARGET_REGISTRY

export REGISTRY_AUTH_FILE=$IBMPAK_HOME/.ibm-pak/auth.json
podman login $TARGET_REGISTRY

or if using Docker

export REGISTRY_AUTH_FILE=$IBMPAK_HOME/.docker/config.json
docker login $TARGET_REGISTRY

Set $IBMPAK_HOME to your current directory

cd $IBMPAK_HOME

Mirror images to your local file system

oc image mirror \
-f $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-to-filesystem.txt \
--filter-by-os '.*'  \
-a $REGISTRY_AUTH_FILE \
--insecure \
--skip-multiple-scopes \
--max-per-registry=1

The images are mirrored to a directory called v2/cp4s_images in the current directory. If you run into errors like error parsing HTTP 429 response body: unexpected end of JSON input: "", repeat the oc image mirror command until the whole process completed without errors. The existing images in the local image registry will not be downloaded again. Error HTTP 429 means too many requests to the registry server. The whole process takes about 90 minutes, depending on network speed.

Optionally, copy the following files and directories from your local file system to your storage device.

• The v2 directory that was generated in the previous step.
• The $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION directory.

Notes

1. You also might want to copy this document, the pull-secrets, the CLI installers, and a RHEL iso to the storage device as well as the SHA256 values for the installers and the iso. If required, the storage device can be scanned by the customer’s IT department. The Red Hat OpenShift documentation, see https://docs.openshift.com/container-platform/4.10/installing/installing_vsphere/installing-restricted-networks-vsphere.html, can be downloaded and saved as a pdf onto this storage device as well. If your pdf contains only blank pages, try a different browser. You can download the CLI installers with yum download. For example, the command

   yum download podman

   downloads the podman-xxx.rpm to the current directory.
2. Download Red Hat OpenShift CLI 4.10 or later from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/. The file to download is called openshift-client-<platform>-<version>.tar.gz.
3. To download the latest release version from the public GitHub repo on Linux®, type the following command

   curl -L https://github.com/IBM/ibm-pak-plugin/releases/latest/download/oc-ibm_pak-linux-amd64.tar.gz -o oc-ibm_pak-plugin.tar.gz

4. To save the Podman registry engine into a tar file, type the following command:

   podman save -o registry.tar registry:2 

5. Save the rhcos.ova file onto the storage device as well. You have to import it into the vSphere server.

In these steps, the mirroring device is disconnected from the internet and connected to the air-gapped environment.

1. Disconnect your mirroring device from the internet.
2. Check whether the CLI tools are installed on the mirroring device in the air-gapped environment.
   • WSL (if Windows)
   • Docker CLI or Podman CLI
   • Red Hat OpenShift CLI
   • IBM Catalog Management plug-in for Red Hat OpenShift CLI
3. Mount the removable storage device to the mirroring device in the air-gapped environment.
4. Install the CLI tools.

   tar -xvf openshift-client-linux-4.10.32.tar.gz
   cp oc /usr/local/sbin/oc
   tar -xvf oc-ibm_pak-plugin.tar.gz
   cp oc-ibm_pak /usr/local/sbin/ 

5. Create the SAN certificate

   openssl genrsa -out private/cakey.pem 4096
   openssl req -new -x509 -days 3650  -config openssl.cnf  -key private/cakey.pem -out certs/cacert.pem
   openssl genrsa -out server.key
   openssl req -new -key server.key -out server.csr -config server_cert.cnf
   openssl x509 -req -days 365 -in server.csr -CA /root/tls/certs/cacert.pem -CAkey /root/tls/private/cakey.pem -CAcreateserial -out server.crt -extensions req_ext -extfile server_cert.cnf

6. Trust the SAN certificate

   ln -s /etc/pki/tls/certs/server.crt /etc/pki/ca-trust/source/anchors
   update-ca-trust

7. Copy the trusted SAN certificate

   cp /root/tls/server.* /var/lib/registry/certs/

8. Load the Podman registry image with the following command:

   cat /store/registry.tar | podman load

9. Tag the Podman registry image with the following commands:

   podman images
   podman tag  <IMAGE ID> registry:air

10. Run the Podman registry with the following command:

    docker run –privileged -d \
    --restart=always \
    --name registry \
    -v /var/lib/registry/certs:/certs:Z \
    -v /var/lib/registry:/var/lib/registry:Z \
    -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/var/lib/registry/certs/server.crt \
    -e REGISTRY_HTTP_TLS_KEY=/var/lib/registry/certs/server.key \
    -p 5000:5000 \
    registry:air

11. Verify that you can log in to the registry.
12. Verify that Red Hat OpenShift is installed on the cluster. If needed, follow the Red Hat OpenShift documentation to install it, see https://docs.openshift.com/container-platform/4.10/installing/installing_vsphere/installing-restricted-networks-vsphere.html.
13. Here are the variables you might want to use:

    OCP_RELEASE=4.10.32
    LOCAL_REGISTRY='rhel8.ocp4.airgap:5000'
    LOCAL_REPOSITORY=ocp4/openshift410
    PRODUCT_REPO='openshift-release-dev'
    LOCAL_SECRET_JSON='/store/pullsecret.json'
    RELEASE_NAME='ocp-release'
    ARCHITECTURE='x86_64'
    REMOVABLE_MEDIA_PATH=/store

14. To mirror the Red Hat OpenShift flat files to the local registry, you can use:

    oc image mirror -a ${LOCAL_SECRET_JSON} \
    --from-dir=${REMOVABLE_MEDIA_PATH}/mirror \ "file://openshift/release:${OCP_RELEASE}*" \ ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}

15. To create the installation program that is based on the content that you mirrored, extract it and pin it to the release:

    oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install –from="${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"

16. If you run into errors like "emergency.target" it means the ignition failed. Check your ignition settings and reboot the node.
17. After you start the worker nodes, approve the certificate signing requests.

    oc get csr -A -o name | xargs oc adm certificate approve

18. Set the environment variable IBMPAK_HOME to the mount point of the storage device.

    export IBMPAK_HOME=<mount_point>

19. Set the $TARGET_REGISTRY environment variable to the IP address or FQDN and the port for your target registry. The target registry is the registry where your images are mirrored to and accessed by the Red Hat OpenShift Container Platform cluster.

    export TARGET_REGISTRY=<target_registry>

20. Create the following environment variables with the installer image name and the image inventory on your mirroring device by typing the following command. <path_to_V2> is the path to the V2 directory generated in step 4 of Mirroring images from the internet to your mirroring device.

    export CASE_NAME=ibm-cp-security
    export CASE_VERSION=1.0.28
    export V2_DIR=<path_to_V2>

21. Store the authentication credentials for your target registry.
    • If you are using Podman, store authentication credentials for your target registry.

      export REGISTRY_AUTH_FILE=$IBMPAK_HOME/.ibm-pak/auth.json
      podman login $TARGET_REGISTRY
      

    • If you are using Docker, store authentication credentials for your target registry.

      export REGISTRY_AUTH_FILE=$IBMPAK_HOME/.docker/config.json
      docker login $TARGET_REGISTRY

    • The command stores and caches the registry credentials in the location that is specified for the $REGISTRY_AUTH_FILE environment variable.
22. If your target registry is on a different host than in the insecure environment, you need to edit the file images-mapping from-filesystem.txt in the $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/ directory

    sed -i “s/<old_registry/new_registry>/g” images-mapping-from-filesystem.txt

23. Check whether you have enough space in your root directory for the registry.

    df -h

24. If you run out of disk space in your root volume, you need to add more space.

    vgextend
    lvextend
    xfs-growfs

25. Mirror images from your mirroring device to the target registry.

    oc image mirror \
    -f $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping-from-filesystem.txt \
    --from-dir=$IBMPAK_HOME \
    --filter-by-os '.*' \
    -a $REGISTRY_AUTH_FILE \
    --insecure \
    --skip-multiple-scopes \
    --max-per-registry=1

26. Log in to the Red Hat OpenShift Container Platform cluster as a cluster administrator.

    oc login <openshift_URL> --username=<cluster_admin_user> --password=<cluster_admin_password>

    If you are getting errors complaining about a self-signed certificate, you can log in to the UI, and then grab the command line token session login.
27. Update the global image pull secret for your Red Hat OpenShift Container Platform cluster and add the credentials for your target registry.

    oc registry login --registry="$TARGET_REGISTRY" --auth-basic="<username>:<password>" --to=<pull_secret_location>

    1. Retrieve the existing global pull secret by typing the following command, where <pull_secret_location> is the location of the file where you want to store the global pull secret configuration.

       oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' > <pull_secret_location>

    2. Add the new pull secret to the global pull secret file by typing the following command, where <username> and <password> are the username and password for your target registry.
    3. Update the global pull secret in the cluster

       oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location>

    4. Verify the status of the nodes

       oc get MachineConfigPool -w

28. After the global pull secret is updated, the configuration of your nodes is updated sequentially.
    Important: Wait until all MachineConfigPools are updated before you proceed to the next step.
29. Edit the ImageContentSourcePolicy resource

    vim $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml

30. Check whether the correct mirror registry is there, if not:

    :%s/<old_registry>/<new_registry>/g

31. Create the ImageContentSourcePolicy resource.

    oc apply -f $IBMPAK_HOME/.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml

32. Verify that the ImageContentSourcePolicy resource is created.

    oc get imageContentSourcePolicy ibm-cp-security -o yaml

33. Verify the status of the nodes.

    oc get MachineConfigPool -w

    After the ImageContentSourcePolicy resource is created, the configuration of your nodes is updated sequentially.
    Important: Wait until all MachineConfigPools are updated before you proceed to the next step.
34. If you are using an insecure registry, you must add the local registry to the cluster insecureRegistries 

    oc patch image.config.openshift.io/cluster --type=merge \
     -p '{"spec":{"registrySources":{"insecureRegistries":["'${TARGET_REGISTRY}'"]}}}'

    Important: Do not use insecure registries for production systems.
35. Verify the status of the nodes.

    oc get MachineConfigPool -w

    After the insecureRegistries list is updated, the configuration of your nodes is updated sequentially.
    Important: Wait until all MachineConfigPools are updated before you proceed to the next step.