Deprecated: Upgrading with a bastion host using cloudctl
You can use a bastion host to perform an air-gapped upgrade of IBM® API Connect to the latest version of 10.0.1.x-eus on OpenShift Container Platform (OCP) when your cluster has no internet connectivity.
Before you begin
- Your deployment must be at 10.0.1.7-eus or later and running on OpenShift 4.10 before you can
upgrade to the latest version of API Connect 10.0.1.x-eus.Restriction: Cloud Pak for Integration 2020.4 is now End of Support and the API Management component cannot be upgraded to a version later than API Connect 10.0.1.7-eus.
- If you are upgrading to a version of API Connect that supports a newer version of Red Hat OpenShift, complete the API Connect upgrade before upgrading Red Hat OpenShift.
- The upgrade procedure requires you to use Red Hat Skopeo for moving container images. Skopeo is not available for Microsoft Windows, so you cannot perform this task using a Windows host.
- The Gateway subsystem remains available during the upgrade of the Management, Portal, and Analytics subsystems.
- Don't use the tilde ~ within double quotation marks in any command because the tilde doesn’t expand and your commands might fail.
Procedure
-
Ensure that you have completed all of the steps in Preparing to upgrade on OpenShift and Cloud Pak for Integration, including reviewing the Upgrade considerations on OpenShift and Cloud Pak for Integration.
Do not attempt an upgrade until you have reviewed the considerations and prepared your deployment.
-
Prepare a host that can be connected to the internet.
Note: If you are using the same host that you used for installing API Connect, skip this step.
The host must satisfy the following requirements:
- The host must be on a Linux x86_64 platform, or any operating system that the IBM Cloud Pak CLI, the OpenShift CLI. and RedHat Skopeo support.
- The host
locale
must be set to English. - The host must have sufficient storage to hold all of the software that is to be transferred to the local Docker registry.
Complete the following steps to set up your external host:
- Install OpenSSL version 1.1.1 or higher.
-
Install Docker or Podman:
- To install Docker (for example, on Red Hat® Enterprise Linux®), run the following commands:
yum check-update yum install docker
- To install Podman, see Podman Installation Instructions.
- To install Docker (for example, on Red Hat® Enterprise Linux®), run the following commands:
-
Install
httpd-tools
by running the following commands:yum install httpd-tools
-
Install the IBM Cloud Pak CLI by completing the following steps:
Install the latest version of the binary file for your platform. For more information, see cloud-pak-cli.
- Download the binary file by running the following
command:
wget https://github.com/IBM/cloud-pak-cli/releases/latest/download/<binary_file_name
For example:wget https://github.com/IBM/cloud-pak-cli/releases/latest/download/cloudctl-linux-amd64.tar.gz
- Extract the binary file by running the following
command:
tar -xf <binary_file_name>
- Run the following commands to modify and move the
file:
chmod 755 <file_name mv <file_name> /usr/local/bin/cloudctl
- Confirm that
cloudctl
is installed by running the following command:cloudctl --help
The
cloudctl
usage is displayed.
- Download the binary file by running the following
command:
-
Install the
oc
OpenShift Container Platform CLI tool.For more information, see Getting started with the CLI in the Red Hat OpenShift documentation.
-
Install RedHat Skopeo CLI version 1.0.0 or higher.
For more information, see Installing Skopeo from packages.
-
Run the following command to create a directory that serves as the offline store.
The following example creates a directory called "upgrade_offline", which is used in the subsequent steps.
mkdir $HOME/upgrade_offline
Notes:- The
$HOME/upgrade_offline
store must be persistent to avoid transferring data more than once. The persistence also helps to run the mirroring process multiple times or on a schedule. - The
$HOME/upgrade_offline
store must not use the same name that you used for the previous installation. If you use the same name as original directory ($HOME/upgrade_offline
), then theapi-connect-catalog
tag will not get updated correctly, the catalog source pods will not pick up the new image, and as a result, the operator will not upgrade. The environment variable should be updated in all the steps.
- The
-
On the bastion host, create environment variables for the installer and image inventory.
Note: If you are using the same bastion host that you used for installing API Connect, you can re-use the environment variables from the installation, but you must update the following variables for the new release:
CASE_VERSION
- update to the newest CASE.OFFLINEDIR
- update to reflect new folder created for upgrade; for example, $HOME/upgrade_offline.
Create the following environment variables with the installer image name and the image inventory. Set the
CASE_VERSION
to the value for the new API Connect release. The CASE version shown in the example might not be correct for your deployment – refer to Operator, operand, and CASE version for the correct CASE version.export CASE_NAME=ibm-apiconnect export CASE_VERSION=2.1.17 export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz export CASE_INVENTORY_SETUP=apiconnectOperatorSetup export OFFLINEDIR=$HOME/upgrade_offline export OFFLINEDIR_ARCHIVE=offline.tgz export CASE_REMOTE_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case/$CASE_NAME/$CASE_VERSION/$CASE_ARCHIVE export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE export BASTION_DOCKER_REGISTRY_HOST=localhost export BASTION_DOCKER_REGISTRY_PORT=443 export BASTION_DOCKER_REGISTRY=$BASTION_DOCKER_REGISTRY_HOST:$BASTION_DOCKER_REGISTRY_PORT export BASTION_DOCKER_REGISTRY_USER=username export BASTION_DOCKER_REGISTRY_PASSWORD=password export BASTION_DOCKER_REGISTRY_PATH=$OFFLINEDIR/imageregistry
-
Download the API Connect installer and image inventory by running the following command:
cloudctl case save \ --case $CASE_REMOTE_PATH \ --outputdir $OFFLINEDIR
-
Mirror the images from the ICR (source) registry to the bastion (destination) registry.
-
Store the credentials for the ICR (source) registry.
The following command stores and caches the IBM Entitled Registry credentials in a file on your file system in the
$HOME/.airgap/secrets
location.cloudctl case launch \ --case $OFFLINEDIR/$CASE_ARCHIVE \ --inventory $CASE_INVENTORY_SETUP \ --action configure-creds-airgap \ --namespace $NAMESPACE \ --args "--registry cp.icr.io --user cp --pass <entitlement-key> --inputDir $OFFLINEDIR"
-
Store the credentials for the bastion (destination) registry.
The following command stores and caches the Docker registry credentials in a file on your file system in the
$HOME/.airgap/secrets
location:cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action configure-creds-airgap \ --args "--registry $BASTION_DOCKER_REGISTRY --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD"
-
If needed, start the Docker registry service on the bastion host.
If you are using the same bastion host that you used for installing API Connect, the Docker registry service might already be running.
- Initialize the Docker registry by running the following
command:
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action init-registry \ --args "--registry $BASTION_DOCKER_REGISTRY_HOST --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD --dir $BASTION_DOCKER_REGISTRY_PATH"
- Start the Docker registry by running the following
command:
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action start-registry \ --args "--registry $BASTION_DOCKER_REGISTRY_HOST --port $BASTION_DOCKER_REGISTRY_PORT --user $BASTION_DOCKER_REGISTRY_USER --pass $BASTION_DOCKER_REGISTRY_PASSWORD --dir $BASTION_DOCKER_REGISTRY_PATH"
- Initialize the Docker registry by running the following
command:
-
Mirror the images to the registry on the bastion host.
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action mirror-images \ --args "--registry $BASTION_DOCKER_REGISTRY --inputDir $OFFLINEDIR"
-
Store the credentials for the ICR (source) registry.
- Optional:
Save the Docker registry image that you stored on the bastion host.
If your air-gapped network doesn’t have a Docker registry image, you can save the image on the bastion host and copy it later to the host in your air-gapped environment.
docker save docker.io/library/registry:2.6 -o $BASTION_DOCKER_REGISTRY_PATH/registry-image.tar
-
Configure access to the local registry for installation.
-
Create environment variables with the local Docker registry connection information.
Note: If you are using the same bastion host that you used for installing API Connect, you can re-use the environment variables from the installation, but you must update the
CASE_VERSION
variable for the new release.Create the following environment variables with the installer image name and the image inventory. Set the
CASE_VERSION
to the same value for the new API Connect release that you used in step #tapic_upgrade_OpenShift_bastion__upgrade_airgap_env_vars.For example:
export CASE_NAME=ibm-apiconnect export CASE_VERSION=2.1.17 export CASE_ARCHIVE=$CASE_NAME-$CASE_VERSION.tgz export CASE_INVENTORY_SETUP=apiconnectOperatorSetup export OFFLINEDIR=$HOME/upgrade_offline export OFFLINEDIR_ARCHIVE=offline.tgz export CASE_REMOTE_PATH=https://github.com/IBM/cloud-pak/raw/master/repo/case/$CASE_NAME/$CASE_VERSION/$CASE_ARCHIVE export CASE_LOCAL_PATH=$OFFLINEDIR/$CASE_ARCHIVE export LOCAL_DOCKER_REGISTRY_HOST=<IP_or_FQDN_of_local_docker_registry> export LOCAL_DOCKER_REGISTRY_PORT=443 export LOCAL_DOCKER_REGISTRY=$LOCAL_DOCKER_REGISTRY_HOST:$LOCAL_DOCKER_REGISTRY_PORT export LOCAL_DOCKER_REGISTRY_USER=username export LOCAL_DOCKER_REGISTRY_PASSWORD=password
-
Set up local registry credentials for mirroring.
Store the credentials of the registry that is running on the internal host.
cloudctl case launch \ --case $CASE_LOCAL_PATH \ --inventory $CASE_INVENTORY_SETUP \ --action configure-creds-airgap \ --args "--registry $LOCAL_DOCKER_REGISTRY --user $LOCAL_DOCKER_REGISTRY_USER --pass $LOCAL_DOCKER_REGISTRY_PASSWORD"
-
If you use an insecure registry, add the local registry to the cluster's insecureRegistries
list.
oc patch image.config.openshift.io/cluster --type=merge -p '{"spec":{"registrySources":{"insecureRegistries":["'$LOCAL_DOCKER_REGISTRY'"]}}}'
-
Create environment variables with the local Docker registry connection information.
- If needed, update the operator channel and the operator
for Cloud Pak common services:
- Open the OpenShift web console and click Operators > Installed Operators > IBM Foundational Services > Subscriptions.
-
Change the channel to
v3
, and update the operator to3.19
.
-
Immediately update the operator channel for API Connect:
-
Confirm that the pod
ibm-apiconnect-catalog-xxxx
in theopenshift-marketplace
namespace was updated. - Open the OpenShift web console and click Operators > Installed Operators > IBM API connect > Subscriptions.
- Change the channel to the new version (v2.1.11-eus), which triggers an upgrade of the API Connect operator.
Known issue: If you are upgrading to API Connect version 10.0.1.6-eus, 10.0.1.6-ifix1-eus, 10.0.1.7-eus, or 10.0.1.8-eus, you might encounter the following error while updating the operator:
Resolve the error by completing the following steps to update the ImageContentSourcePolicy for your deployment:Message: unpack job not completed: Unpack pod(openshift-marketplace/e9f169cee8bffacf9ab35d276a48b7207d9606e2b7a0a8087bc58b4ff7tx22l) container(pull) is pending. Reason: ImagePullBackOff, Message: Back-off pulling image "ibmcom/ibm-apiconnect-operator-bundle@sha256:ef0ce455270189c37a5dc0500219061959c041f88110f601f6e7bf8072df4943" Reason: JobIncomplete
- Log in to the OpenShift cluster UI as an administrator of your cluster.
- Click Search > Resources and search for ICSP.
- In the list of ICSPs, click ibm-apiconnect to edit it.
- In the
ibm-apiconnect
ICSP, click the YAML tab. - In the
spec.repositoryDigestMirrors
section, locate the-mirrors:
subsection containingsource: docker.io/ibmcom)
. - Add a new mirror ending with
/ibmcom
to the section as in the following example:- mirrors: - <AIRGAP_REGISTRY_ADDRESS>/ibmcom - <AIRGAP_REGISTRY_ADDRESS>/cpopen source: docker.io/ibmcom
- If the job does not automatically continue, uninstall and reinstall the API Connect operator.
When the APIC Connect operator is updated, the new pod starts automatically. -
Confirm that the pod
-
Verify that the API Connect operator was updated by completing the following steps:
-
Get the name of the pod that hosts the operator by running the following command:
oc get po -n <APIC_namespace> | grep apiconnect
The response looks like the following example:ibm-apiconnect-7bdb795465-8f7rm 1/1 Running 0 4m23s
-
Get the API Connect version deployed on that pod by running the following command:
oc describe po <ibm-apiconnect-operator-podname> -n <APIC_namespace> | grep -i productversion
The response looks like the following example:productVersion: 10.0.1.12-eus
-
Get the name of the pod that hosts the operator by running the following command:
- Use the latest version of
apicops
to validate the certificates.- Run the following command:
apicops upgrade:stale-certs -n <APIC_namespace>
- Delete any stale certificates that are managed by cert-manager. If a certificate failed the validation and it is managed by cert-manager, you can delete the stale certificate secret, and let cert-manager regenerate it. Run the following command:
oc delete secret <stale-secret> -n <APIC_namespace>
- Restart the corresponding so that it can pick up the new secret. To determine which pod to restart, see the following topics:
For information on the
apicops
tool, see The API Connect operations tool: apicops. - Run the following command:
- Ensure that the operators and operands are healthy before
proceeding.
- Operators: The OpenShift web console indicates that all operators are in
Succeeded
state without any warnings. - Operands: To verify whether operands are healthy, run the following command:
oc get apic
Check the status of the
apiconnectcluster
custom resource. All subsystems should report asREADY
- Operators: The OpenShift web console indicates that all operators are in
- Upgrade the API Connect operand:
- Update the
version
field in the top-level CR; for example:apiVersion: apiconnect.ibm.com/v1beta1 kind: APIConnectCluster metadata: labels: app.kubernetes.io/instance: apiconnect app.kubernetes.io/managed-by: ibm-apiconnect app.kubernetes.io/name: apiconnect-production name: prod namespace: APIC_namespace spec: license: accept: true use: production profile: n12xc4.m12 version: 10.0.1.12-eus storageClassName: rook-ceph-block
Specify the currently deployed
profile
anduse
values, which might not match the example. If you want to change to a different profile, you can do it after completing the upgrade (for instructions, see Changing deployment profiles on OpenShift.) - In the
spec.gateway
section of the top-level CR, remove thetemplate
override section, if it exists.You cannot perform an upgrade if the CR contains an override.
- Apply the updated top-level CR to upgrade the API Connect operand by running the
following command:
oc apply -f CR_file_name.yaml
The response looks like the following example:apiconnectcluster.apiconnect.ibm.com/prod configured
- Update the
- Verify that the upgraded subsystems report as
Running
.Run the following command:
oc get apic --all-namespaces
All subsystems should report as
Running
.Example response:
NAME READY STATUS VERSION RECONCILED VERSION AGE analyticscluster.analytics.apiconnect.ibm.com/analytics 8/8 Running 10.0.1.12-eus 10.0.1.12-eus-1074 121m NAME PHASE READY SUMMARY VERSION AGE datapowerservice.datapower.ibm.com/gw1 Running True StatefulSet replicas ready: 1/1 10.0.1.12-eus 100m NAME PHASE LAST EVENT WORK PENDING WORK IN-PROGRESS AGE datapowermonitor.datapower.ibm.com/gw1 Running false false 100m NAME READY STATUS VERSION RECONCILED VERSION AGE gatewaycluster.gateway.apiconnect.ibm.com/gw1 2/2 Running 10.0.1.12-eus 10.0.1.12-eus-1074 100m NAME READY STATUS VERSION RECONCILED VERSION AGE managementcluster.management.apiconnect.ibm.com/m1 16/16 Running 10.0.1.12-eus 110.0.1.12-eus-1074 162m NAME READY STATUS VERSION RECONCILED VERSION AGE portalcluster.portal.apiconnect.ibm.com/portal 3/3 Running 10.0.1.12-eus 10.0.1.12-eus-1074 139m
-
Validate that the upgrade was successfully deployed by running the following command:
oc get apic -n APIC_namespace
The response looks like the following example and should show the new product version:
NAME READY STATUS VERSION RECONCILED VERSION AGE analyticscluster.analytics.apiconnect.ibm.com/prod-a7s 8/8 Running 10.0.1.12-eus 10.0.1.12-eus 21h NAME READY STATUS VERSION RECONCILED VERSION AGE apiconnectcluster.apiconnect.ibm.com/prod 4/4 Ready 10.0.1.12-eus 10.0.1.12-eus 22h NAME PHASE READY SUMMARY VERSION AGE datapowerservice.datapower.ibm.com/prod-gw Running True StatefulSet replicas ready: 3/3 10.0.1.12-eus 21h NAME PHASE LAST EVENT WORK PENDING WORK IN-PROGRESS AGE datapowermonitor.datapower.ibm.com/prod-gw Running false false 21h NAME READY STATUS VERSION RECONCILED VERSION AGE gatewaycluster.gateway.apiconnect.ibm.com/prod-gw 2/2 Running 10.0.1.12-eus 10.0.1.12-eus 21h NAME READY STATUS VERSION RECONCILED VERSION AGE managementcluster.management.apiconnect.ibm.com/prod-mgmt 16/16 Running 10.0.1.12-eus 10.0.1.12-eus 22h NAME STATUS ID CLUSTER TYPE CR TYPE AGE managementbackup.management.apiconnect.ibm.com/prod-mgmt-0f583bd9 Ready 20210505-141020F_20210506-011830I prod-mgmt incr record 11h managementbackup.management.apiconnect.ibm.com/prod-mgmt-10af02ee Ready 20210505-141020F prod-mgmt full record 21h managementbackup.management.apiconnect.ibm.com/prod-mgmt-148f0cfa Ready 20210505-141020F_20210506-012856I prod-mgmt incr record 11h managementbackup.management.apiconnect.ibm.com/prod-mgmt-20bd6dae Ready 20210505-141020F_20210506-090753I prod-mgmt incr record 3h28m managementbackup.management.apiconnect.ibm.com/prod-mgmt-40efdb38 Ready 20210505-141020F_20210505-195838I prod-mgmt incr record 16h managementbackup.management.apiconnect.ibm.com/prod-mgmt-681aa239 Ready 20210505-141020F_20210505-220302I prod-mgmt incr record 14h managementbackup.management.apiconnect.ibm.com/prod-mgmt-7f7150dd Ready 20210505-141020F_20210505-160732I prod-mgmt incr record 20h managementbackup.management.apiconnect.ibm.com/prod-mgmt-806f8de6 Ready 20210505-141020F_20210505-214657I prod-mgmt incr record 14h managementbackup.management.apiconnect.ibm.com/prod-mgmt-868a066a Ready 20210505-141020F_20210506-090140I prod-mgmt incr record 3h34m managementbackup.management.apiconnect.ibm.com/prod-mgmt-cf9a85dc Ready 20210505-141020F_20210505-210119I prod-mgmt incr record 15h managementbackup.management.apiconnect.ibm.com/prod-mgmt-ef63b789 Ready 20210506-103241F prod-mgmt full record 83m NAME STATUS MESSAGE AGE managementdbupgrade.management.apiconnect.ibm.com/prod-mgmt-up-649mc Complete Upgrade is Complete (DB Schema/data are up-to-date) 142m managementdbupgrade.management.apiconnect.ibm.com/prod-mgmt-up-9mjhk Complete Fresh install is Complete (DB Schema/data are up-to-date) 22h NAME READY STATUS VERSION RECONCILED VERSION AGE portalcluster.portal.apiconnect.ibm.com/prod-ptl 3/3 Running 10.0.1.12-eus 10.0.1.12-eus 21h
- Upgrade to Red Hat OpenShift Container Platform
4.12 if you have not already done so. Red Hat OpenShift requires you to upgrade in stages, so that you install every version between your starting point and your ending point. For example, to upgrade from 4.10 to 4.12, you must complete 2 upgrades:
- Upgrade to 4.11
- Upgrade to 4.12
- Optional: Install the latest version of the API Connect Toolkit and the API Connect Local Test Environment.