Pre-upgrade preparation and checks on Kubernetes
Prepare your Kubernetes environment and API Connect deployment for upgrade.
This topic covers all the preparation steps you must complete before you can start the upgrade process.
Before completing the steps in this topic, review Planning your API Connect upgrade on Kubernetes to ensure that you are following a supported upgrade path and that you are aware of any API Connect feature changes that might affect your deployment.
Run the pre-upgrade health checks during your upgrade planning stage to identify any problems as early as possible. Run the checks again just before upgrade to confirm API Connect is still ready for upgrade.
Gathering and preparing upgrade files
- Obtain the API Connect files from IBM
Fix Central.
From the IBM Fix Central site, download the following files:
- IBM API Connect <version> for Containers
- Docker images for all API Connect subsystems. Filename is apiconnect-image-tool_<version>.
- IBM API Connect <version> Operator Release Files for Containers
- Kubernetes operators and API Connect Custom Resource (CR) templates. Filename is apiconnect-operator-release-files_<version>.
- IBM API Connect <version> Security Signature Bundle File
- Signature files for verifying the integrity of your downloads.
Filename is signatures_<version>.
- IBM API Connect <version> Toolkit for <operating_system>
- Toolkit command-line utility. Packaged as a stand-alone file, or with API
Designer:
- IBM® API Connect <version> Toolkit for <operating_system>
- IBM® API Connect <version> Toolkit Designer for <operating_system>
Not required during initial installation. After installation, you can download directly from the Cloud Manager UI and API Manager UI as explained in Installing the toolkit.
- IBM API Connect <version> Local Test Environment
- Optional test environment for testing locally as explained in Testing an API with the
Local Test Environment.
Filename is apic-lte-<operating system>_<version>.
- Complete the steps in Verifying the integrity of IBM product files to verify that the downloaded product files are not corrupted.
- Upload the image files to your docker registry.
- Load the image-tool image for the new API Connect version into
your local Docker registry:
docker load < apiconnect-image-tool-<version>.tar.gz
Ensure that the registry has sufficient disk space for the files.
- If your Docker registry requires repositories to be created before images can be
pushed, create the repositories for each of the images that the image tool lists. If your Docker
registry does not require creation of repositories, skip to step 3.c.
- Run the following command to get a list of the images from
image-tool:
docker run --rm apiconnect-image-tool-<version> version --images
- From the output of each entry of the form
<image-name>:<image-tag>
, use your Docker registry repository create command to create a repository for<image-name>
.For example, in AWS ECR the command for each<image-name>
is:aws ecr create-repository --repository-name <image-name>
- Run the following command to get a list of the images from
image-tool:
- Upload the image:
- If you do not need to authenticate with the docker registry,
use:
docker run --rm apiconnect-image-tool-<version> upload <registry-url>
- If you need to authenticate and your docker registry accepts the username and password
arguments,
use:
docker run --rm apiconnect-image-tool-<version> upload <registry-url> --username <username> --password <password>
- Otherwise, if you need the image-tool to use your local Docker credentials (for example, with
IBM Container Registry), first authenticate with your Docker registry, and then upload images with
the
command:
docker run --rm -v ~/.docker:/root/.docker --user 0 apiconnect-image-tool-<version> upload <registry-url>
Review the following installation notes for your environment:
If necessary, you can populate a remote container registry with repositories. Then you can push the images from the local registry to the remote registry.
- If you do not need to authenticate with the docker registry,
use:
- Load the image-tool image for the new API Connect version into
your local Docker registry:
- Extract the
IBM API Connect <version> Operator Release Files for Containers
that you downloaded in step 1.Create a directory called
helper_files
and extract the contents ofhelper_files.zip
from the release_files.zip into thehelper_files
directory. - If upgrading from 10.0.5.x or 10.0.7.0: If you do not
use cert-manager, create a certificate for the new consumer catalog, and add it to the management
CR.
Beginning with version 10.0.8.0, API Connect includes the Consume Catalog feature. If you are upgrading to 10.0.8.0 or later from 10.0.5.x or 10.0.7.0 and do not use cert-manager, you must provide a certificate for the Consumer Catalog during the upgrade.
- Create a certificate.
- Update the management CR and add the following endpoint for the Consumer Catalog with the secret
name for the new certificate:
consumerCatalogEndpoint: hosts: - name: consumer-catalog.<domain> secretName: <custom-management-consumer-catalog-endpoint>
Pre-upgrade health-checks
- Check that the API Connect operator and
subsystems are all running.
To verify the health of each subsystem, run the following commands:
Confirm that all pods arekubectl get ManagementCluster -n <management namespace> kubectl get GatewayCluster -n <gateway namespace> kubectl get PortalCluster -n <portal namespace> kubectl get AnalyticsCluster -n <analyticsnamespace>
READY
, for example:kubectl get PortalCluster -n apic NAME READY STATUS VERSION RECONCILED VERSION AGE portal 3/3 Running version version-95 57m
- If upgrading from 10.0.5.x: Verify that the management subsystem's
pgcluster
is working correctly:- Get the name of the
pgcluster
:kubectl get pgcluster -n <management namespace>
The response displays the name of the postgres cluster that is running in the specified namespace.
- Check the status of the
pgcluster
:kubectl get pgcluster <pgcluster_name> -n <APIC_namespace> -o yaml | grep status -A 2 | tail -n3
A successful response looks like the following example, where the state isInitialized
:status: message: Cluster has been initialized state: pgcluster Initialized
Important: If thepgcluster
returns any other state, then do not proceed with the upgrade.- If backup or restore jobs are running, wait until they complete and then check the status again.
Do not proceed with the upgrade until the status is
Initialized
. - If all background jobs are complete but the
pgcluster
does not returnstate: pgcluster Initialized
, then contact IBM Support for assistance.
- If backup or restore jobs are running, wait until they complete and then check the status again.
Do not proceed with the upgrade until the status is
- Get the name of the
- Run the pre-upgrade health check:
Note: If you have a 2DCDR deployment, run this check only on the active management subsystem.
- Verify that the
apicops
utility is installed and that you have the latest version by running the following command:apicops --version
For more information about the
apicops
utility, see The API Connect operations tool: apicops. - Run the following command to set the KUBECONFIG environment variable.
export KUBECONFIG=</path/to/kubeconfig>
- Run the apicops pre-upgrade-check check for the management, portal, and
analytics subsystems:
- Management
subsystem:
apicops system:pre-upgrade-check management -n <namespace>
Before upgrading, run the following command to check for encryption issues and invalid gateway extensions:
For a full check (requires admin password to verify topology):apicops preupgrade -n <namespace>
To skip topology verification (no admin password needed):apicops preupgrade --no-topology -n <namespace>
- Portal subsystem:
apicops system:pre-upgrade-check portal -n <namespace>
- Analytics
subsystem:
apicops system:pre-upgrade-check analytics -n <namespace>
If your API Connect deployment is working correctly, the output shows no error messages.
If the pre-upgrade check returns any errors, then collect the output and open a support case.
- Management
subsystem:
- Verify that the
Additional pre-upgrade checks and operations
Extra checks and operations to review and complete before you start the upgrade.
- Upgrades from V10.0.5.x: If the analytics persistent queue is not enabled, verify that you have an extra Physical Volume available.
For more information about this requirement, see Analytics persistent queue enablement requires extra PV.
- If analytics database backups are configured, or you want to enable them in future, then ensure that you have an extra Physical Volume available for the local backup storage. The default size is 150Gi. You can override the size during the upgrade procedure.
- If you used any microservice image overrides in any of your API Connect subsystem CRs, remove the image overrides before upgrade.
Backup your API Connect deployment
If upgrade fails, rollback is not possible. Before upgrade complete the disaster recovery preparation for all your subsystems.
- v10.0.7.0: Backing up, restoring, and disaster recovery
- v10.0.5.4 or later: Backups on Kubernetes, Disaster recovery.