Deprecated: Upgrading with a bastion host using cloudctl

You can use a bastion host to perform an air-gapped upgrade of IBM® API Connect to the latest version of 10.0.1.x-eus on OpenShift Container Platform (OCP) when your cluster has no internet connectivity.

Before you begin

Your deployment must be at 10.0.1.7-eus or later and running on OpenShift 4.10 before you can upgrade to the latest version of API Connect 10.0.1.x-eus.
Restriction: Cloud Pak for Integration 2020.4 is now End of Support and the API Management component cannot be upgraded to a version later than API Connect 10.0.1.7-eus.
If you are upgrading to a version of API Connect that supports a newer version of Red Hat OpenShift, complete the API Connect upgrade before upgrading Red Hat OpenShift.
The upgrade procedure requires you to use Red Hat Skopeo for moving container images. Skopeo is not available for Microsoft Windows, so you cannot perform this task using a Windows host.
The Gateway subsystem remains available during the upgrade of the Management, Portal, and Analytics subsystems.
Don't use the tilde ~ within double quotation marks in any command because the tilde doesn’t expand and your commands might fail.

Procedure

Ensure that you have completed all of the steps in Preparing to upgrade on OpenShift and Cloud Pak for Integration, including reviewing the Upgrade considerations on OpenShift and Cloud Pak for Integration.

Do not attempt an upgrade until you have reviewed the considerations and prepared your deployment.
Prepare a host that can be connected to the internet.
Note: If you are using the same host that you used for installing API Connect, skip this step.

The host must satisfy the following requirements:
- The host must be on a Linux x86_64 platform, or any operating system that the IBM Cloud Pak CLI, the OpenShift CLI. and RedHat Skopeo support.
- The host locale must be set to English.
- The host must have sufficient storage to hold all of the software that is to be transferred to the local Docker registry.
Complete the following steps to set up your external host:
1. Install OpenSSL version 1.1.1 or higher.
2. Install Docker or Podman:
  - To install Docker (for example, on Red Hat® Enterprise Linux®), run the following commands:
```
yum check-update
yum install docker
```
  - To install Podman, see Podman Installation Instructions.
3. Install httpd-tools by running the following commands:
```
yum install httpd-tools
```
4. Install the IBM Cloud Pak CLI by completing the following steps:
  Install the latest version of the binary file for your platform. For more information, see cloud-pak-cli.
  1. Download the binary file by running the following command:
```
wget https://github.com/IBM/cloud-pak-cli/releases/latest/download/<binary_file_name
```
    For example:
    wget https://github.com/IBM/cloud-pak-cli/releases/latest/download/cloudctl-linux-amd64.tar.gz
  2. Extract the binary file by running the following command:
```
tar -xf <binary_file_name>
```
  3. Run the following commands to modify and move the file:
```
chmod 755 <file_name
mv <file_name> /usr/local/bin/cloudctl
```
  4. Confirm that cloudctl is installed by running the following command:
```
cloudctl --help
```
    The cloudctl usage is displayed.
5. Install the oc OpenShift Container Platform CLI tool.
  
  For more information, see Getting started with the CLI in the Red Hat OpenShift documentation.
6. Install RedHat Skopeo CLI version 1.0.0 or higher.
  
  For more information, see Installing Skopeo from packages.
7. Run the following command to create a directory that serves as the offline store.
  The following example creates a directory called "upgrade_offline", which is used in the subsequent steps.
```
mkdir $HOME/upgrade_offline
```
  Notes:
  - The $HOME/upgrade_offline store must be persistent to avoid transferring data more than once. The persistence also helps to run the mirroring process multiple times or on a schedule.
  - The $HOME/upgrade_offline store must not use the same name that you used for the previous installation. If you use the same name as original directory ($HOME/upgrade_offline), then the api-connect-catalog tag will not get updated correctly, the catalog source pods will not pick up the new image, and as a result, the operator will not upgrade. The environment variable should be updated in all the steps.
On the bastion host, create environment variables for the installer and image inventory.
Note: If you are using the same bastion host that you used for installing API Connect, you can re-use the environment variables from the installation, but you must update the following variables for the new release:
- CASE_VERSION - update to the newest CASE.
- OFFLINEDIR - update to reflect new folder created for upgrade; for example, $HOME/upgrade_offline.
Create the following environment variables with the installer image name and the image inventory. Set the CASE_VERSION to the value for the new API Connect release. The CASE version shown in the example might not be correct for your deployment – refer to Operator, operand, and CASE version for the correct CASE version.
```
export CASE_NAME=ibm-apiconnect
export CASE_VERSION=2.1.17
export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
export CASE_INVENTORY_SETUP=apiconnectOperatorSetup
export OFFLINEDIR=$HOME/upgrade_offline
export OFFLINEDIR_ARCHIVE=offline.tgz
export CASE_REMOTE_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case/$CASE_NAME/$CASE_VERSION/$CASE_ARCHIVE
export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE

export BASTION_DOCKER_REGISTRY_HOST=localhost
export BASTION_DOCKER_REGISTRY_PORT=443
export BASTION_DOCKER_REGISTRY=$BASTION_DOCKER_REGISTRY_HOST:$BASTION_DOCKER_REGISTRY_PORT
export BASTION_DOCKER_REGISTRY_USER=username
export BASTION_DOCKER_REGISTRY_PASSWORD=password
export BASTION_DOCKER_REGISTRY_PATH=$OFFLINEDIR/imageregistry
```
Download the API Connect installer and image inventory by running the following command:
```
cloudctl case save \
  --case $CASE_REMOTE_PATH \
  --outputdir $OFFLINEDIR
```

Mirror the images from the ICR (source) registry to the bastion (destination) registry.

Store the credentials for the ICR (source) registry.

The following command stores and caches the IBM Entitled Registry credentials in a file on your file system in the $HOME/.airgap/secrets location.

cloudctl case launch \
   --case $OFFLINEDIR/$CASE_ARCHIVE \
   --inventory $CASE_INVENTORY_SETUP \
   --action configure-creds-airgap \
   --namespace $NAMESPACE \
   --args "--registry cp.icr.io --user cp --pass <entitlement-key> --inputDir $OFFLINEDIR"

Store the credentials for the bastion (destination) registry.

The following command stores and caches the Docker registry credentials in a file on your file system in the $HOME/.airgap/secrets location:

cloudctl case launch \
  --case $CASE_LOCAL_PATH \
  --inventory $CASE_INVENTORY_SETUP \
  --action configure-creds-airgap \
  --args "--registry $BASTION_DOCKER_REGISTRY --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD"

If needed, start the Docker registry service on the bastion host.

If you are using the same bastion host that you used for installing API Connect, the Docker registry service might already be running.

Initialize the Docker registry by running the following command:

cloudctl case launch \
  --case $CASE_LOCAL_PATH \
  --inventory $CASE_INVENTORY_SETUP \
  --action init-registry \
  --args "--registry $BASTION_DOCKER_REGISTRY_HOST --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD --dir $BASTION_DOCKER_REGISTRY_PATH"

Start the Docker registry by running the following command:

cloudctl case launch \
  --case $CASE_LOCAL_PATH \
  --inventory $CASE_INVENTORY_SETUP \
  --action start-registry \
  --args "--registry $BASTION_DOCKER_REGISTRY_HOST --port $BASTION_DOCKER_REGISTRY_PORT --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD --dir $BASTION_DOCKER_REGISTRY_PATH"

Mirror the images to the registry on the bastion host.

cloudctl case launch \
  --case $CASE_LOCAL_PATH \
  --inventory $CASE_INVENTORY_SETUP \
  --action mirror-images \
  --args "--registry $BASTION_DOCKER_REGISTRY --inputDir $OFFLINEDIR"

Optional: Save the Docker registry image that you stored on the bastion host.
If your air-gapped network doesn’t have a Docker registry image, you can save the image on the bastion host and copy it later to the host in your air-gapped environment.
```
docker save docker.io/library/registry:2.6 -o $BASTION_DOCKER_REGISTRY_PATH/registry-image.tar
```

Configure access to the local registry for installation.

Create environment variables with the local Docker registry connection information.

Note: If you are using the same bastion host that you used for installing API Connect, you can re-use the environment variables from the installation, but you must update the CASE_VERSION variable for the new release.

Create the following environment variables with the installer image name and the image inventory. Set the CASE_VERSION to the same value for the new API Connect release that you used in step #tapic_upgrade_OpenShift_bastion__upgrade_airgap_env_vars.

For example:

export CASE_NAME=ibm-apiconnect
export CASE_VERSION=2.1.17
export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
export CASE_INVENTORY_SETUP=apiconnectOperatorSetup
export OFFLINEDIR=$HOME/upgrade_offline
export OFFLINEDIR_ARCHIVE=offline.tgz
export CASE_REMOTE_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case/$CASE_NAME/$CASE_VERSION/$CASE_ARCHIVE
export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE

export LOCAL_DOCKER_REGISTRY_HOST=<IP_or_FQDN_of_local_docker_registry>
export LOCAL_DOCKER_REGISTRY_PORT=443
export LOCAL_DOCKER_REGISTRY=$LOCAL_DOCKER_REGISTRY_HOST:$LOCAL_DOCKER_REGISTRY_PORT
export LOCAL_DOCKER_REGISTRY_USER=username
export LOCAL_DOCKER_REGISTRY_PASSWORD=password

Set up local registry credentials for mirroring.

Store the credentials of the registry that is running on the internal host.

cloudctl case launch \
  --case $CASE_LOCAL_PATH \
  --inventory $CASE_INVENTORY_SETUP \
  --action configure-creds-airgap \
  --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD"

If you use an insecure registry, add the local registry to the cluster's insecureRegistries list.

oc patch image.config.openshift.io/cluster --type=merge -p '{"spec":{"registrySources":{"insecureRegistries":["'$LOCAL_DOCKER_REGISTRY'"]}}}'

If needed, update the operator channel and the operator for Cloud Pak common services:
1. Open the OpenShift web console and click Operators > Installed Operators > IBM Foundational Services > Subscriptions.
2. Change the channel to v3, and update the operator to 3.19.
Immediately update the operator channel for API Connect:
1. Confirm that the pod ibm-apiconnect-catalog-xxxx in the openshift-marketplace namespace was updated.
2. Open the OpenShift web console and click Operators > Installed Operators > IBM API connect > Subscriptions.
3. Change the channel to the new version (v2.1.11-eus), which triggers an upgrade of the API Connect operator.
Known issue: If you are upgrading to API Connect version 10.0.1.6-eus, 10.0.1.6-ifix1-eus, 10.0.1.7-eus, or 10.0.1.8-eus, you might encounter the following error while updating the operator:
```
Message: unpack job not completed: Unpack pod(openshift-marketplace/e9f169cee8bffacf9ab35d276a48b7207d9606e2b7a0a8087bc58b4ff7tx22l) container(pull) is pending. Reason: ImagePullBackOff, Message: Back-off pulling image "ibmcom/ibm-apiconnect-operator-bundle@sha256:ef0ce455270189c37a5dc0500219061959c041f88110f601f6e7bf8072df4943" Reason: JobIncomplete
```
Resolve the error by completing the following steps to update the ImageContentSourcePolicy for your deployment:
1. Log in to the OpenShift cluster UI as an administrator of your cluster.
2. Click Search > Resources and search for ICSP.
3. In the list of ICSPs, click ibm-apiconnect to edit it.
4. In the ibm-apiconnect ICSP, click the YAML tab.
5. In the spec.repositoryDigestMirrors section, locate the -mirrors: subsection containing source: docker.io/ibmcom).
6. Add a new mirror ending with /ibmcom to the section as in the following example:
```
- mirrors:
        - <AIRGAP_REGISTRY_ADDRESS>/ibmcom
        - <AIRGAP_REGISTRY_ADDRESS>/cpopen
      source: docker.io/ibmcom
```
7. If the job does not automatically continue, uninstall and reinstall the API Connect operator.
When the APIC Connect operator is updated, the new pod starts automatically.
Verify that the API Connect operator was updated by completing the following steps:
1. Get the name of the pod that hosts the operator by running the following command:
```
oc get po -n <APIC_namespace> | grep apiconnect
```
  The response looks like the following example:
```
ibm-apiconnect-7bdb795465-8f7rm            1/1     Running     0      4m23s
```
2. Get the API Connect version deployed on that pod by running the following command:
```
oc describe po <ibm-apiconnect-operator-podname> -n <APIC_namespace> | grep -i productversion
```
  The response looks like the following example:
```
productVersion: 10.0.1.12-eus
```
Use the latest version of apicops to validate the certificates.
1. Run the following command:
```
apicops upgrade:stale-certs -n <APIC_namespace>
```
2. Delete any stale certificates that are managed by cert-manager.
  If a certificate failed the validation and it is managed by cert-manager, you can delete the stale certificate secret, and let cert-manager regenerate it. Run the following command:
```
oc delete secret <stale-secret> -n <APIC_namespace>
```
3. Restart the corresponding so that it can pick up the new secret.
  To determine which pod to restart, see the following topics:
  - List of external certificates
  - List of internal certificates
For information on the apicops tool, see The API Connect operations tool: apicops.
Ensure that the operators and operands are healthy before proceeding.
- Operators: The OpenShift web console indicates that all operators are in Succeeded state without any warnings.
- Operands: To verify whether operands are healthy, run the following command: oc get apic
  Check the status of the apiconnectcluster custom resource. All subsystems should report as READY
Upgrade the API Connect operand:
1. Update the version field in the top-level CR; for example:
```
apiVersion: apiconnect.ibm.com/v1beta1
kind: APIConnectCluster
metadata:
  labels:
    app.kubernetes.io/instance: apiconnect
    app.kubernetes.io/managed-by: ibm-apiconnect
    app.kubernetes.io/name: apiconnect-production
  name: prod
  namespace: APIC_namespace
spec:
  license:
    accept: true
    use: production
  profile: n12xc4.m12
  version: 10.0.1.12-eus
  storageClassName: rook-ceph-block
```
  Specify the currently deployed profile and use values, which might not match the example. If you want to change to a different profile, you can do it after completing the upgrade (for instructions, see Changing deployment profiles on OpenShift.)
2. In the spec.gateway section of the top-level CR, remove the template override section, if it exists.
  
  You cannot perform an upgrade if the CR contains an override.
3. Apply the updated top-level CR to upgrade the API Connect operand by running the following command:
```
oc apply -f CR_file_name.yaml
```
  The response looks like the following example:
```
apiconnectcluster.apiconnect.ibm.com/prod configured
```

Verify that the upgraded subsystems report as Running.

Run the following command:

oc get apic --all-namespaces

All subsystems should report as Running.

Example response:

NAME                                                READY   STATUS    VERSION              RECONCILED VERSION      AGE
analyticscluster.analytics.apiconnect.ibm.com/analytics      8/8     Running   10.0.1.12-eus   10.0.1.12-eus-1074   121m

NAME                                     PHASE     READY   SUMMARY                           VERSION    AGE
datapowerservice.datapower.ibm.com/gw1   Running   True    StatefulSet replicas ready: 1/1   10.0.1.12-eus   100m

NAME                                     PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
datapowermonitor.datapower.ibm.com/gw1   Running                false          false              100m

NAME                                            READY   STATUS    VERSION              RECONCILED VERSION      AGE
gatewaycluster.gateway.apiconnect.ibm.com/gw1   2/2     Running   10.0.1.12-eus   10.0.1.12-eus-1074  100m

NAME                                                 READY   STATUS    VERSION              RECONCILED VERSION      AGE
managementcluster.management.apiconnect.ibm.com/m1   16/16   Running   10.0.1.12-eus   110.0.1.12-eus-1074   162m


NAME                                             READY   STATUS    VERSION              RECONCILED VERSION      AGE
portalcluster.portal.apiconnect.ibm.com/portal   3/3     Running   10.0.1.12-eus   10.0.1.12-eus-1074   139m

Validate that the upgrade was successfully deployed by running the following command:

oc get apic -n APIC_namespace

The response looks like the following example and should show the new product version:

NAME                                                     READY   STATUS    VERSION              RECONCILED VERSION       AGE
analyticscluster.analytics.apiconnect.ibm.com/prod-a7s              8/8     Running   10.0.1.12-eus		   10.0.1.12-eus   21h

NAME                                        READY   STATUS   VERSION              RECONCILED VERSION       AGE
apiconnectcluster.apiconnect.ibm.com/prod   4/4     Ready    10.0.1.12-eus	      10.0.1.12-eus             22h

NAME                                         PHASE     READY   SUMMARY                           VERSION        AGE
datapowerservice.datapower.ibm.com/prod-gw   Running   True    StatefulSet replicas ready: 3/3   10.0.1.12-eus   21h

NAME                                         PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
datapowermonitor.datapower.ibm.com/prod-gw   Running                false          false              21h

NAME                                                READY   STATUS    VERSION              RECONCILED VERSION       AGE
gatewaycluster.gateway.apiconnect.ibm.com/prod-gw   2/2     Running   10.0.1.12-eus		   10.0.1.12-eus             21h

NAME                                                        READY   STATUS    VERSION              RECONCILED VERSION       AGE
managementcluster.management.apiconnect.ibm.com/prod-mgmt   16/16   Running   10.0.1.12-eus         10.0.1.12-eus             22h

NAME                                                                STATUS   ID                                  CLUSTER     TYPE   CR TYPE   AGE
managementbackup.management.apiconnect.ibm.com/prod-mgmt-0f583bd9   Ready    20210505-141020F_20210506-011830I   prod-mgmt   incr   record    11h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-10af02ee   Ready    20210505-141020F                    prod-mgmt   full   record    21h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-148f0cfa   Ready    20210505-141020F_20210506-012856I   prod-mgmt   incr   record    11h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-20bd6dae   Ready    20210505-141020F_20210506-090753I   prod-mgmt   incr   record    3h28m
managementbackup.management.apiconnect.ibm.com/prod-mgmt-40efdb38   Ready    20210505-141020F_20210505-195838I   prod-mgmt   incr   record    16h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-681aa239   Ready    20210505-141020F_20210505-220302I   prod-mgmt   incr   record    14h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-7f7150dd   Ready    20210505-141020F_20210505-160732I   prod-mgmt   incr   record    20h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-806f8de6   Ready    20210505-141020F_20210505-214657I   prod-mgmt   incr   record    14h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-868a066a   Ready    20210505-141020F_20210506-090140I   prod-mgmt   incr   record    3h34m
managementbackup.management.apiconnect.ibm.com/prod-mgmt-cf9a85dc   Ready    20210505-141020F_20210505-210119I   prod-mgmt   incr   record    15h
managementbackup.management.apiconnect.ibm.com/prod-mgmt-ef63b789   Ready    20210506-103241F                    prod-mgmt   full   record    83m

NAME                                                                   STATUS     MESSAGE                                                     AGE
managementdbupgrade.management.apiconnect.ibm.com/prod-mgmt-up-649mc   Complete   Upgrade is Complete (DB Schema/data are up-to-date)         142m
managementdbupgrade.management.apiconnect.ibm.com/prod-mgmt-up-9mjhk   Complete   Fresh install is Complete (DB Schema/data are up-to-date)   22h

NAME                                               READY   STATUS    VERSION              RECONCILED VERSION       AGE
portalcluster.portal.apiconnect.ibm.com/prod-ptl   3/3     Running   10.0.1.12-eus         10.0.1.12-eus             21h

Upgrade to Red Hat OpenShift Container Platform 4.12 if you have not already done so.
Red Hat OpenShift requires you to upgrade in stages, so that you install every version between your starting point and your ending point. For example, to upgrade from 4.10 to 4.12, you must complete 2 upgrades:
1. Upgrade to 4.11
2. Upgrade to 4.12
For upgrade instructions, see the Red Hat OpenShift documentation.
Optional: Install the latest version of the API Connect Toolkit and the API Connect Local Test Environment.