Deprecated: Upgrading with a bastion host using cloudctl

You can use a bastion host to perform an air-gapped upgrade of IBM® API Connect to the latest version of 10.0.1.x-eus on OpenShift Container Platform (OCP) when your cluster has no internet connectivity.

Before you begin

  • Your deployment must be at 10.0.1.7-eus or later and running on OpenShift 4.10 before you can upgrade to the latest version of API Connect 10.0.1.x-eus.
    Restriction: Cloud Pak for Integration 2020.4 is now End of Support and the API Management component cannot be upgraded to a version later than API Connect 10.0.1.7-eus.
  • If you are upgrading to a version of API Connect that supports a newer version of Red Hat OpenShift, complete the API Connect upgrade before upgrading Red Hat OpenShift.
  • The upgrade procedure requires you to use Red Hat Skopeo for moving container images. Skopeo is not available for Microsoft Windows, so you cannot perform this task using a Windows host.
  • The Gateway subsystem remains available during the upgrade of the Management, Portal, and Analytics subsystems.
  • Don't use the tilde ~ within double quotation marks in any command because the tilde doesn’t expand and your commands might fail.

Procedure

  1. Ensure that you have completed all of the steps in Preparing to upgrade on OpenShift and Cloud Pak for Integration, including reviewing the Upgrade considerations on OpenShift and Cloud Pak for Integration.

    Do not attempt an upgrade until you have reviewed the considerations and prepared your deployment.

  2. Prepare a host that can be connected to the internet.
    Note: If you are using the same host that you used for installing API Connect, skip this step.

    The host must satisfy the following requirements:

    • The host must be on a Linux x86_64 platform, or any operating system that the IBM Cloud Pak CLI, the OpenShift CLI. and RedHat Skopeo support.
    • The host locale must be set to English.
    • The host must have sufficient storage to hold all of the software that is to be transferred to the local Docker registry.

    Complete the following steps to set up your external host:

    1. Install OpenSSL version 1.1.1 or higher.
    2. Install Docker or Podman:
      • To install Docker (for example, on Red Hat® Enterprise Linux®), run the following commands:
        yum check-update
        yum install docker
        
      • To install Podman, see Podman Installation Instructions.
    3. Install httpd-tools by running the following commands:
      yum install httpd-tools
      
    4. Install the IBM Cloud Pak CLI by completing the following steps:
      Install the latest version of the binary file for your platform. For more information, see cloud-pak-cli.
      1. Download the binary file by running the following command:
        wget https://github.com/IBM/cloud-pak-cli/releases/latest/download/<binary_file_name
        
        For example:
        wget https://github.com/IBM/cloud-pak-cli/releases/latest/download/cloudctl-linux-amd64.tar.gz
      2. Extract the binary file by running the following command:
        tar -xf <binary_file_name>
      3. Run the following commands to modify and move the file:
        chmod 755 <file_name
        mv <file_name> /usr/local/bin/cloudctl
      4. Confirm that cloudctl is installed by running the following command:
        cloudctl --help

        The cloudctl usage is displayed.

    5. Install the oc OpenShift Container Platform CLI tool.

      For more information, see Getting started with the CLI in the Red Hat OpenShift documentation.

    6. Install RedHat Skopeo CLI version 1.0.0 or higher.

      For more information, see Installing Skopeo from packages.

    7. Run the following command to create a directory that serves as the offline store.

      The following example creates a directory called "upgrade_offline", which is used in the subsequent steps.

      mkdir $HOME/upgrade_offline
      Notes:
      • The $HOME/upgrade_offline store must be persistent to avoid transferring data more than once. The persistence also helps to run the mirroring process multiple times or on a schedule.
      • The $HOME/upgrade_offline store must not use the same name that you used for the previous installation. If you use the same name as original directory ($HOME/upgrade_offline), then the api-connect-catalog tag will not get updated correctly, the catalog source pods will not pick up the new image, and as a result, the operator will not upgrade. The environment variable should be updated in all the steps.
  3. On the bastion host, create environment variables for the installer and image inventory.
    Note: If you are using the same bastion host that you used for installing API Connect, you can re-use the environment variables from the installation, but you must update the following variables for the new release:
    • CASE_VERSION - update to the newest CASE.
    • OFFLINEDIR - update to reflect new folder created for upgrade; for example, $HOME/upgrade_offline.

    Create the following environment variables with the installer image name and the image inventory. Set the CASE_VERSION to the value for the new API Connect release. The CASE version shown in the example might not be correct for your deployment – refer to Operator, operand, and CASE version for the correct CASE version.

    export CASE_NAME=ibm-apiconnect
    export CASE_VERSION=2.1.17
    export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
    export CASE_INVENTORY_SETUP=apiconnectOperatorSetup
    export OFFLINEDIR=$HOME/upgrade_offline
    export OFFLINEDIR_ARCHIVE=offline.tgz
    export CASE_REMOTE_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case/$CASE_NAME/$CASE_VERSION/$CASE_ARCHIVE
    export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE
    
    export BASTION_DOCKER_REGISTRY_HOST=localhost
    export BASTION_DOCKER_REGISTRY_PORT=443
    export BASTION_DOCKER_REGISTRY=$BASTION_DOCKER_REGISTRY_HOST:$BASTION_DOCKER_REGISTRY_PORT
    export BASTION_DOCKER_REGISTRY_USER=username
    export BASTION_DOCKER_REGISTRY_PASSWORD=password
    export BASTION_DOCKER_REGISTRY_PATH=$OFFLINEDIR/imageregistry
    
  4. Download the API Connect installer and image inventory by running the following command:
    cloudctl case save \
      --case $CASE_REMOTE_PATH \
      --outputdir $OFFLINEDIR
    
  5. Mirror the images from the ICR (source) registry to the bastion (destination) registry.
    1. Store the credentials for the ICR (source) registry.

      The following command stores and caches the IBM Entitled Registry credentials in a file on your file system in the $HOME/.airgap/secrets location.

      cloudctl case launch \
         --case $OFFLINEDIR/$CASE_ARCHIVE \
         --inventory $CASE_INVENTORY_SETUP \
         --action configure-creds-airgap \
         --namespace $NAMESPACE \
         --args "--registry cp.icr.io --user cp --pass <entitlement-key> --inputDir $OFFLINEDIR"
      
    2. Store the credentials for the bastion (destination) registry.

      The following command stores and caches the Docker registry credentials in a file on your file system in the $HOME/.airgap/secrets location:

      cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --inventory $CASE_INVENTORY_SETUP \
        --action configure-creds-airgap \
        --args "--registry $BASTION_DOCKER_REGISTRY --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD"
      
    3. If needed, start the Docker registry service on the bastion host.

      If you are using the same bastion host that you used for installing API Connect, the Docker registry service might already be running.

      1. Initialize the Docker registry by running the following command:
        cloudctl case launch \
          --case $CASE_LOCAL_PATH \
          --inventory $CASE_INVENTORY_SETUP \
          --action init-registry \
          --args "--registry $BASTION_DOCKER_REGISTRY_HOST --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD --dir $BASTION_DOCKER_REGISTRY_PATH"
        
      2. Start the Docker registry by running the following command:
        cloudctl case launch \
          --case $CASE_LOCAL_PATH \
          --inventory $CASE_INVENTORY_SETUP \
          --action start-registry \
          --args "--registry $BASTION_DOCKER_REGISTRY_HOST --port $BASTION_DOCKER_REGISTRY_PORT --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD --dir $BASTION_DOCKER_REGISTRY_PATH"
        
    4. Mirror the images to the registry on the bastion host.
      cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --inventory $CASE_INVENTORY_SETUP \
        --action mirror-images \
        --args "--registry $BASTION_DOCKER_REGISTRY --inputDir $OFFLINEDIR"
      
  6. Optional: Save the Docker registry image that you stored on the bastion host.

    If your air-gapped network doesn’t have a Docker registry image, you can save the image on the bastion host and copy it later to the host in your air-gapped environment.

    docker save docker.io/library/registry:2.6 -o $BASTION_DOCKER_REGISTRY_PATH/registry-image.tar
    
  7. Configure access to the local registry for installation.
    1. Create environment variables with the local Docker registry connection information.
      Note: If you are using the same bastion host that you used for installing API Connect, you can re-use the environment variables from the installation, but you must update the CASE_VERSION variable for the new release.

      Create the following environment variables with the installer image name and the image inventory. Set the CASE_VERSION to the same value for the new API Connect release that you used in step #tapic_upgrade_OpenShift_bastion__upgrade_airgap_env_vars.

      For example:

      export CASE_NAME=ibm-apiconnect
      export CASE_VERSION=2.1.17
      export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz
      export CASE_INVENTORY_SETUP=apiconnectOperatorSetup
      export OFFLINEDIR=$HOME/upgrade_offline
      export OFFLINEDIR_ARCHIVE=offline.tgz
      export CASE_REMOTE_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case/$CASE_NAME/$CASE_VERSION/$CASE_ARCHIVE
      export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE
      
      export LOCAL_DOCKER_REGISTRY_HOST=<IP_or_FQDN_of_local_docker_registry>
      export LOCAL_DOCKER_REGISTRY_PORT=443
      export LOCAL_DOCKER_REGISTRY=$LOCAL_DOCKER_REGISTRY_HOST:$LOCAL_DOCKER_REGISTRY_PORT
      export LOCAL_DOCKER_REGISTRY_USER=username
      export LOCAL_DOCKER_REGISTRY_PASSWORD=password
      
    2. Set up local registry credentials for mirroring.

      Store the credentials of the registry that is running on the internal host.

      cloudctl case launch \
        --case $CASE_LOCAL_PATH \
        --inventory $CASE_INVENTORY_SETUP \
        --action configure-creds-airgap \
        --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD"
      
    3. If you use an insecure registry, add the local registry to the cluster's insecureRegistries list.
      oc patch image.config.openshift.io/cluster --type=merge -p '{"spec":{"registrySources":{"insecureRegistries":["'$LOCAL_DOCKER_REGISTRY'"]}}}'
      
  8. If needed, update the operator channel and the operator for Cloud Pak common services:
    1. Open the OpenShift web console and click Operators > Installed Operators > IBM Foundational Services > Subscriptions.
    2. Change the channel to v3, and update the operator to 3.19.
  9. Immediately update the operator channel for API Connect:
    1. Confirm that the pod ibm-apiconnect-catalog-xxxx in the openshift-marketplace namespace was updated.
    2. Open the OpenShift web console and click Operators > Installed Operators > IBM API connect > Subscriptions.
    3. Change the channel to the new version (v2.1.11-eus), which triggers an upgrade of the API Connect operator.
    Known issue: If you are upgrading to API Connect version 10.0.1.6-eus, 10.0.1.6-ifix1-eus, 10.0.1.7-eus, or 10.0.1.8-eus, you might encounter the following error while updating the operator:
    Message: unpack job not completed: Unpack pod(openshift-marketplace/e9f169cee8bffacf9ab35d276a48b7207d9606e2b7a0a8087bc58b4ff7tx22l) container(pull) is pending. Reason: ImagePullBackOff, Message: Back-off pulling image "ibmcom/ibm-apiconnect-operator-bundle@sha256:ef0ce455270189c37a5dc0500219061959c041f88110f601f6e7bf8072df4943" Reason: JobIncomplete
    Resolve the error by completing the following steps to update the ImageContentSourcePolicy for your deployment:
    1. Log in to the OpenShift cluster UI as an administrator of your cluster.
    2. Click Search > Resources and search for ICSP.
    3. In the list of ICSPs, click ibm-apiconnect to edit it.
    4. In the ibm-apiconnect ICSP, click the YAML tab.
    5. In the spec.repositoryDigestMirrors section, locate the -mirrors: subsection containing source: docker.io/ibmcom).
    6. Add a new mirror ending with /ibmcom to the section as in the following example:
      - mirrors:
              - <AIRGAP_REGISTRY_ADDRESS>/ibmcom
              - <AIRGAP_REGISTRY_ADDRESS>/cpopen
            source: docker.io/ibmcom
    7. If the job does not automatically continue, uninstall and reinstall the API Connect operator.
    When the APIC Connect operator is updated, the new pod starts automatically.
  10. Verify that the API Connect operator was updated by completing the following steps:
    1. Get the name of the pod that hosts the operator by running the following command:
      oc get po -n <APIC_namespace> | grep apiconnect
      The response looks like the following example:
      ibm-apiconnect-7bdb795465-8f7rm            1/1     Running     0      4m23s
    2. Get the API Connect version deployed on that pod by running the following command:
      oc describe po <ibm-apiconnect-operator-podname> -n <APIC_namespace> | grep -i productversion
      The response looks like the following example:
      productVersion: 10.0.1.12-eus
  11. Use the latest version of apicops to validate the certificates.
    1. Run the following command:
      apicops upgrade:stale-certs -n <APIC_namespace>
    2. Delete any stale certificates that are managed by cert-manager.
      If a certificate failed the validation and it is managed by cert-manager, you can delete the stale certificate secret, and let cert-manager regenerate it. Run the following command:
      oc delete secret <stale-secret> -n <APIC_namespace>
    3. Restart the corresponding so that it can pick up the new secret.
      To determine which pod to restart, see the following topics:

    For information on the apicops tool, see The API Connect operations tool: apicops.

  12. Ensure that the operators and operands are healthy before proceeding.
    • Operators: The OpenShift web console indicates that all operators are in Succeeded state without any warnings.

    • Operands: To verify whether operands are healthy, run the following command: oc get apic

      Check the status of the apiconnectcluster custom resource. All subsystems should report as READY

  13. Upgrade the API Connect operand:
    1. Update the version field in the top-level CR; for example:
      apiVersion: apiconnect.ibm.com/v1beta1
      kind: APIConnectCluster
      metadata:
        labels:
          app.kubernetes.io/instance: apiconnect
          app.kubernetes.io/managed-by: ibm-apiconnect
          app.kubernetes.io/name: apiconnect-production
        name: prod
        namespace: APIC_namespace
      spec:
        license:
          accept: true
          use: production
        profile: n12xc4.m12
        version: 10.0.1.12-eus
        storageClassName: rook-ceph-block

      Specify the currently deployed profile and use values, which might not match the example. If you want to change to a different profile, you can do it after completing the upgrade (for instructions, see Changing deployment profiles on OpenShift.)

    2. In the spec.gateway section of the top-level CR, remove the template override section, if it exists.

      You cannot perform an upgrade if the CR contains an override.

    3. Apply the updated top-level CR to upgrade the API Connect operand by running the following command:
      oc apply -f CR_file_name.yaml
      The response looks like the following example:
      apiconnectcluster.apiconnect.ibm.com/prod configured
  14. Verify that the upgraded subsystems report as Running.

    Run the following command:

    oc get apic --all-namespaces

    All subsystems should report as Running.

    Example response:

    NAME                                                READY   STATUS    VERSION              RECONCILED VERSION      AGE
    analyticscluster.analytics.apiconnect.ibm.com/analytics      8/8     Running   10.0.1.12-eus   10.0.1.12-eus-1074   121m
    
    NAME                                     PHASE     READY   SUMMARY                           VERSION    AGE
    datapowerservice.datapower.ibm.com/gw1   Running   True    StatefulSet replicas ready: 1/1   10.0.1.12-eus   100m
    
    NAME                                     PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
    datapowermonitor.datapower.ibm.com/gw1   Running                false          false              100m
    
    NAME                                            READY   STATUS    VERSION              RECONCILED VERSION      AGE
    gatewaycluster.gateway.apiconnect.ibm.com/gw1   2/2     Running   10.0.1.12-eus   10.0.1.12-eus-1074  100m
    
    NAME                                                 READY   STATUS    VERSION              RECONCILED VERSION      AGE
    managementcluster.management.apiconnect.ibm.com/m1   16/16   Running   10.0.1.12-eus   110.0.1.12-eus-1074   162m
    
    
    NAME                                             READY   STATUS    VERSION              RECONCILED VERSION      AGE
    portalcluster.portal.apiconnect.ibm.com/portal   3/3     Running   10.0.1.12-eus   10.0.1.12-eus-1074   139m
  15. Validate that the upgrade was successfully deployed by running the following command:
    oc get apic -n APIC_namespace

    The response looks like the following example and should show the new product version:

    NAME                                                     READY   STATUS    VERSION              RECONCILED VERSION       AGE
    analyticscluster.analytics.apiconnect.ibm.com/prod-a7s              8/8     Running   10.0.1.12-eus		   10.0.1.12-eus   21h
    
    NAME                                        READY   STATUS   VERSION              RECONCILED VERSION       AGE
    apiconnectcluster.apiconnect.ibm.com/prod   4/4     Ready    10.0.1.12-eus	      10.0.1.12-eus             22h
    
    NAME                                         PHASE     READY   SUMMARY                           VERSION        AGE
    datapowerservice.datapower.ibm.com/prod-gw   Running   True    StatefulSet replicas ready: 3/3   10.0.1.12-eus   21h
    
    NAME                                         PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
    datapowermonitor.datapower.ibm.com/prod-gw   Running                false          false              21h
    
    NAME                                                READY   STATUS    VERSION              RECONCILED VERSION       AGE
    gatewaycluster.gateway.apiconnect.ibm.com/prod-gw   2/2     Running   10.0.1.12-eus		   10.0.1.12-eus             21h
    
    NAME                                                        READY   STATUS    VERSION              RECONCILED VERSION       AGE
    managementcluster.management.apiconnect.ibm.com/prod-mgmt   16/16   Running   10.0.1.12-eus         10.0.1.12-eus             22h
    
    NAME                                                                STATUS   ID                                  CLUSTER     TYPE   CR TYPE   AGE
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-0f583bd9   Ready    20210505-141020F_20210506-011830I   prod-mgmt   incr   record    11h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-10af02ee   Ready    20210505-141020F                    prod-mgmt   full   record    21h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-148f0cfa   Ready    20210505-141020F_20210506-012856I   prod-mgmt   incr   record    11h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-20bd6dae   Ready    20210505-141020F_20210506-090753I   prod-mgmt   incr   record    3h28m
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-40efdb38   Ready    20210505-141020F_20210505-195838I   prod-mgmt   incr   record    16h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-681aa239   Ready    20210505-141020F_20210505-220302I   prod-mgmt   incr   record    14h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-7f7150dd   Ready    20210505-141020F_20210505-160732I   prod-mgmt   incr   record    20h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-806f8de6   Ready    20210505-141020F_20210505-214657I   prod-mgmt   incr   record    14h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-868a066a   Ready    20210505-141020F_20210506-090140I   prod-mgmt   incr   record    3h34m
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-cf9a85dc   Ready    20210505-141020F_20210505-210119I   prod-mgmt   incr   record    15h
    managementbackup.management.apiconnect.ibm.com/prod-mgmt-ef63b789   Ready    20210506-103241F                    prod-mgmt   full   record    83m
    
    NAME                                                                   STATUS     MESSAGE                                                     AGE
    managementdbupgrade.management.apiconnect.ibm.com/prod-mgmt-up-649mc   Complete   Upgrade is Complete (DB Schema/data are up-to-date)         142m
    managementdbupgrade.management.apiconnect.ibm.com/prod-mgmt-up-9mjhk   Complete   Fresh install is Complete (DB Schema/data are up-to-date)   22h
    
    NAME                                               READY   STATUS    VERSION              RECONCILED VERSION       AGE
    portalcluster.portal.apiconnect.ibm.com/prod-ptl   3/3     Running   10.0.1.12-eus         10.0.1.12-eus             21h
  16. Upgrade to Red Hat OpenShift Container Platform 4.12 if you have not already done so.
    Red Hat OpenShift requires you to upgrade in stages, so that you install every version between your starting point and your ending point. For example, to upgrade from 4.10 to 4.12, you must complete 2 upgrades:
    1. Upgrade to 4.11
    2. Upgrade to 4.12
    For upgrade instructions, see the Red Hat OpenShift documentation.
  17. Optional: Install the latest version of the API Connect Toolkit and the API Connect Local Test Environment.