Air-gapped installs

In production, it is common to have a cluster that does not have internet access. If your Kubernetes cluster is air-gapped (otherwise known as offline or disconnected), you can install an IBM Storage Scale container native by mirroring images. These tasks must be ran by a Kubernetes administrator.

A bastion host is a device that has access to both the public internet and the network-restricted environment where a local registry and Kubernetes clusters reside. Using the bastion host, you can mirror your images directly to the local registry.

You need to do the Air gap setup if the worker nodes are not able to access the repository due to network and firewall restrictions.

Prerequisites

A production-grade Docker V2 registry that is available and accessible from the Kubernetes cluster nodes such as Quay Enterprise, JFrog Artifactory, or Docker Registry. The Red Hat OpenShift Internal Registry is not supported.
A bastion node that:
- Access to the public internet.
- Access to the network-restricted environment where the local registry exists.
- Access to the entitled registry: icr.io:443, cp.icr.io:443.
- skopeo command is installed.
Access to the Kubernetes cluster as a user with the cluster-admin role.

Do not use Red Hat OpenShift internal image registry as your local registry. The Red Hat OpenShift registry does not support multi-architecture images or path separators in the image name.

Before starting, export the following variables. They are used in the subsequent steps.

export the internal registry domain
```
  export INTERNAL_REGISTRY=example.io
```

export the internal registry target path

  export INTERNAL_REGISTRY_PATH=path/to/images

Mirror the images

Mirror the images by using skopeo to copy images from the IBM Container Repository, ICR, to an internal container image registry.

Complete the following steps from the bastion node:

 #
 # assumes you have exported INTERNAL_REGISTRY
 #
 skopeo login $INTERNAL_REGISTRY

Use skopeo copy to copy the images from ICR to the internal image registry.

The image listing for each release is available on GitHub. See the Container image list for IBM Storage Scale container native in the GitHub repository.

Starting with IBM Storage Scale container native v5.2.3.0, the ICR path for the container images have been changed from spectrum/scale to gpfs.

Copy the air-gapped images from GitHub to a file called images.txt. You can then use the following helper script to create the skopeo copy commands to run:
```
 #
 # assumes you have exported INTERNAL_REGISTRY
 #
 for image in `cat images.txt | grep -v \#`; do
     if [[ $image == *"cpopen"* ]]; then
         echo "skopeo copy --all docker://${image} docker://$INTERNAL_REGISTRY/${image#*icr.io/}"
     else
         echo "skopeo copy --all docker://${image} docker://$INTERNAL_REGISTRY/${image#*cp.icr.io/}"
     fi
 done
```
The helper script is provided as an example. Use at your own risk.

For more information, see Skopeo Copy to the Rescue.
Log out of the IBM Container Registry:
```
 skopeo logout cp.icr.io
```

Log out of your internal image registry:

 #
 # assumes you have exported INTERNAL_REGISTRY
 #
 skopeo logout ${INTERNAL_REGISTRY}

Modify the installer yamls

Download installation manifest file

For Red Hat OpenShift:

 curl -fs https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.2.3.x/generated/scale/install.yaml > install.yaml || echo "Failed to download install yaml"

For Kubernetes:

 curl -fs https://raw.githubusercontent.com/IBM/ibm-spectrum-scale-container-native/v5.2.3.x/generated/scale/install-k8s.yaml > install.yaml || echo "Failed to download install yaml"

Ensure that you pick the correct curl command based on your environment. The source file is different but the downloaded files are renamed to install.yaml.

Use sed to replace production path to the internal image registry path

 #
 # assumes you have exported INTERNAL_REGISTRY
 #
 sed -i s#cp.icr.io/cp#$INTERNAL_REGISTRY#g install.yaml
 sed -i s#icr.io/cpopen#$INTERNAL_REGISTRY#g install.yaml

Apply the installation manifest file to the cluster
```
 kubectl apply -f install.yaml
```

Create namespace docker pull secrets

Create ibm-entitlement-key pull secret in the following namespaces with credentials to your internal registry.

ibm-spectrum-scale
ibm-spectrum-scale-operator
ibm-spectrum-scale-dns
ibm-spectrum-scale-csi

This is required to allow the images to pull successfully.

Export the entitlement key and user from your internal image registry.

 export INTERNAL_REGISTRY=example.io
 export ENTITLEMENT_USER=<REPLACE WITH USER OF INTERNAL IMAGE REGISTRY>
 export ENTITLEMENT_KEY=<REPLACE WITH PASSWORD OF INTERNAL IMAGE REGISTRY>

Use the following helper script to create a docker-registry secret for each namespace:

 for namespace in ibm-spectrum-scale ibm-spectrum-scale-operator ibm-spectrum-scale-dns ibm-spectrum-scale-csi; do
     kubectl create secret docker-registry ibm-entitlement-key -n ${namespace} \
     --docker-server=${INTERNAL_REGISTRY} \
     --docker-username=${ENTITLEMENT_USER} \
     --docker-password=${ENTITLEMENT_KEY}
 done

Unset the export:

 unset INTERNAL_REGISTRY
 unset ENTITLEMENT_USER
 unset ENTITLEMENT_KEY

Verify the deployment

After deploying the installation manifest files and creating the image pull secrets, the images should pull successfully and the operator pods will be in a running state.

# kubectl get pods -n ibm-spectrum-scale-operator
NAME                                                     READY   STATUS    RESTARTS   AGE
ibm-spectrum-scale-controller-manager-64d548f576-znrgc   1/1     Running   0          37m
# kubectl get pods -n ibm-spectrum-scale-csi
NAME                                               READY   STATUS    RESTARTS   AGE
ibm-spectrum-scale-csi-operator-67d5bbb486-8zwq5   1/1     Running   0          56m