Upgrading remote workers in an air-gapped environment

• To install Docker (for example, on Red Hat Enterprise Linux®), run these commands:

  yum check-update
  yum install docker
  

• To install Podman, see Podman Installation Instructions.

1. Download the latest version from this location: IBM Catalog Management plug-in.
2. Install the plug-in by running these commands:

   tar -xf oc-ibm_pak-linux-amd64.tar.gz
   mv oc-ibm_pak-linux-amd64 /usr/local/bin/oc-ibm_pak
   

3. To confirm that the plug-in is installed, connect to the air-gapped cluster and run the following command:

   oc ibm-pak

export TARGET_REGISTRY=<target-registry>
export TARGET_REGISTRY_REPO=${TARGET_REGISTRY}$"/cp/pliant"
export TARGET_REGISTRY_USER=<registry-username>
export TARGET_REGISTRY_PASSWD=<registry-password>

• Replace <target-registry> with the name of the private container registry (hostname and port). For example:

  export TARGET_REGISTRY=myregistry.com:5000

• Replace <registry-username> and <registry-password> with the username and password of a user account that has write permissions to the private registry.

oc ibm-pak generate mirror-manifests $CASE_NAME $TARGET_REGISTRY --version $CASE_VERSION

Generating mirror manifests of CASE: ibm-rapid-infrastructure-automation, version: 1.1.5 is complete

1. Edit the /.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping.txt file.
2. If required, update the list of images and save your changes.

oc image mirror \
 -f ./.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/images-mapping.txt \
 --filter-by-os '.*' \
 -a $REGISTRY_AUTH_FILE \
 --insecure \
 --skip-multiple-scopes \
 --max-per-registry=1 \
 --continue-on-error=true > my-mirror-progress.txt  2>&1 &

• Depending on the speed of your network connection, it might take several hours to mirror all the images. The time can vary depending on conditions in the registry's network.
• The --continue-on-error parameter ensures that the command continues to mirror the images even when an error occurs.
• Optional: Use the --help parameter to see all the command options.

tail -f my-mirror-progress.txt

If you see errors that indicate a network or bandwidth problem, you can run the image mirroring command again. When you run it again, the command attempts to copy only those images that are not already copied.

• For Red Hat OpenShift clusters, to configure your cluster with the authentication credentials to pull images from the private container registry, follow the steps in Updating the global cluster pull secret.
• For Kubernetes clusters, follow the steps in Pull an Image from a Private Registry.
• The registry authentication credentials are specified in the /.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml file. Verify that this file contains credentials for each of the mirrors that are specified in the following example configuration. In this example, myregistry.com:5000 is the hostname and port of the private container registry.

  apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy 
  metadata:
    name: ibm-rapid-infrastructure-automation 
  spec:   
    repositoryDigestMirrors:
    - mirrors:     
      - myregistry.com:5000/cp     
      source: cp.icr.io/cp
    - mirrors:     
      - myregistry.com:5000/cpopen     
      source: icr.io/cpopen   

• Go to the parent directory of the .ibm-pak directory.
• For Red Hat OpenShift clusters, run this command:

  oc apply -f ./.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml

• For Kubernetes clusters, run this command:

  kubectl apply -f ./.ibm-pak/data/mirror/$CASE_NAME/$CASE_VERSION/image-content-source-policy.yaml

• For Red Hat OpenShift clusters, run this command:

  oc get imageContentSourcePolicy

• For Kubernetes clusters, run this command:

  kubectl get imageContentSourcePolicy

• For Red Hat OpenShift clusters, run this command:

  oc get MachineConfigPool -w

• For Kubernetes clusters, run this command:

  kubectl get MachineConfigPool -w

• After the ImageContentsourcePolicy and the global image pull secret are applied, the configuration of your nodes is updated sequentially. Wait until all MachineConfigPools resources are in the UPDATED=True status before proceeding.

• For Red Hat OpenShift clusters, use commands like the following example commands that create a namespace that is called ria.

  export NAMESPACE="ria"
  oc new-project ${NAMESPACE}

• For Kubernetes clusters, use commands like the following example commands that create a namespace that is called ria.

  export NAMESPACE="ria"
  kubectl new-project ${NAMESPACE}

oc patch image.config.openshift.io/cluster --type=merge \
   -p '{"spec":{"registrySources":{"insecureRegistries":["'${TARGET_REGISTRY}'"]}}}'

1. Create the certificate by using a certificate generation tool, such as OpenSSL. For example:

   openssl req -x509 -newkey rsa:4096 -keyout airgap.key -out airgap.crt -days 365 -nodes -subj "/CN=$TARGET_REGISTRY_REPO"
   

2. Creaete a ConfigMap from the contents of the generated certificate by running this command:

   oc create configmap airgap-ca \
     --from-file=ca-bundle.crt=airgap.crt \
     -n openshift-config
   

3. Add the certificate to the trusted certificate list for the cluster by running this command:

   oc patch image.config.openshift.io/cluster \
     --type='merge' \
     -p '{"spec":{"additionalTrustedCA":{"name":"airgap-ca"}}}'
   

oc get image.config.openshift.io -o yaml

apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: Image
  metadata:
    annotations:
      include.release.openshift.io/ibm-cloud-managed: "true"
      include.release.openshift.io/self-managed-high-availability: "true"
      release.openshift.io/create-only: "true"
    creationTimestamp: "2025-03-03T16:15:00Z"
    generation: 3
    name: cluster
    ownerReferences:
    - apiVersion: config.openshift.io/v1
      kind: ClusterVersion
      name: version
      uid: 35b75984-af3a-42e1-b79a-9eb9cf64a139
    resourceVersion: "1460593"
    uid: bb1c6aaa-a098-40bb-8c28-830c08e3e2e2
  spec:
    additionalTrustedCA:
      name: airgap-ca
    registrySources:
      insecureRegistries:
      - pliant-airgap1.fyre.ibm.com:5000
  status:
    internalRegistryHostname: image-registry.openshift-image-registry.svc:5000
kind: List
metadata:
  resourceVersion: ""

helm repo add ibm-helm https://raw.githubusercontent.com/IBM/charts/master/repo/ibm-helm

This repository contains the required Helm charts that the upgrade script uses to deploy the remote worker pods, services, and volumes on the cluster. If the following message is shown, the repository is successfully added:

"ibm-helm" has been added to your repositories

export RIA_IP=<IP-or-domain-name>

export IBM_REG_PASS=<entitlement-key>

Replace <entitlement-key> with your entitlement key.

export WORKER_GROUP_NAME=<group-name>
export WORKER_GROUP_SECRET=<group-secret>

Replace <group_name> and <group_secret> with the name and secret of the worker group that you want the remote worker to be a member of.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

export INSTALL_SCRIPTS=$(pwd)
tar -xvzf ./.ibm-pak/data/cases/$CASE_NAME/$CASE_VERSION/$CASE_NAME-$CASE_VERSION.tgz 
$INSTALL_SCRIPTS/ibm-rapid-infrastructure-automation/inventory/ria/files/install_rw.sh --license-acceptance=y --namespace=$NAMESPACE --server-hostname=$RIA_IP --registry=$TARGET_REGISTRY --registry-user=$TARGET_REGISTRY_USER --registry-password=$TARGET_REGISTRY_PASSWD --worker-group-name=$WORKER_GROUP_NAME --worker-group-secret-key=$WORKER_GROUP_SECRET

• For Red Hat OpenShift clusters, run this command:

  oc get pods -n rna-rw

• For Kubernetes clusters, run this command:

  kubectl get pods -n rna-rw

• When all pods are ready and are in a status of Running or Completed, the upgrade is successful.