Upgrading subsystems on native Kubernetes
Upgrade your API Connect subsystems to the latest version on native Kubernetes.
Before you begin
- To ensure operator compatibility, upgrade API Connect management
subsystem first and DataPower gateway subsystem second. This requirement applies to all upgrade
scenarios.
The Gateway subsystem remains available during the upgrade of the Management, Portal, and Analytics subsystems.
- Verify that your installed Kubernetes version is supported by the API Connect version that you
are upgrading to. See IBM API Connect Version 10 software product compatibility requirements. If your
Kubernetes version is older than the minimum supported version for your target API Connect version,
then upgrade Kubernetes first. Note: If you are upgrading from an old API Connect version, it is possible that it does not support the minimum supported Kubernetes version that is required by the latest API Connect release. If so, one or more intermediate upgrades of API Connect might be required to ensure that API Connect is always running on a supported Kubernetes version.
Procedure
- Complete the prerequisites:
- Ensure you are upgrading from a supported version and reviewed the upgrade requirements and limitations. See Upgrade considerations on native Kubernetes.
- Verify that the
pgcluster
is healthy:- Get the name of the
pgcluster
:kubectl get pgcluster -n <APIC_namespace>
The response displays the name of the postgres cluster running in the specified namespace.
- Check the status of the
pgcluster
:kubectl get pgcluster <pgcluster_name> -n <APIC_namespace> -o yaml | grep status -A 2 | tail -n3
The response for a healthy
pgcluster
looks like the following example, where the state isInitialized
:status: message: Cluster has been initialized state: pgcluster Initialized
Important: If thepgcluster
returns any other state, it is not healthy and an upgrade will fail.- If there are any on-going backup or restore jobs, wait until they complete and then check the
status again. Do not proceed with the upgrade until the status is
Initialized
. - If all of the background jobs complete but the
pgcluster
remains unhealthy, contact IBM Support for assistance.
- Get the name of the
- Backup the current deployment. Wait until the backup completes before starting the
upgrade.Note: Backups taken prior to v10.0.5.8 upgrades are not usable after upgrading to v10.0.5.8.
During the upgrade, the database backup path is modified internally and API Connect writes new backups under /<db_path_in_mgmt_cr_db_spec>/pg13backups.
- Do not start an upgrade if a backup is scheduled to run within a few hours.
- Do not perform maintenance tasks such as rotating key-certificates, restoring from a backup, or starting a new backup, at any time while the upgrade process is running.
- If you used any microservice image overrides in the management CR during a fresh install, you
must remove the image overrides prior to upgrade.Important: When upgrading, if you used any microservice image overrides in the management CR during a fresh install then prior to upgrade these image overrides will be automatically removed by the operator during upgrade. You can apply them again after the upgrade is complete.
- Run the pre-upgrade health check: Attention: This is a required step. Failure to run this check before upgrading could result in problems during the upgrade.
If you have a 2DCDR deployment, run this check on the active site only. You can run the check on the warm-standby after it is made stand-alone. See Steps for Management upgrades.
- Verify that the
apicops
utility is installed by running the following command to check the current version of the utility:apicops --version
If the response indicates that
apicops
is not available, install it now. See The API Connect operations tool: apicops in the API Connect documentation. - Run the following command to set the KUBECONFIG environment.
export KUBECONFIG=</path/to/kubeconfig>
- Run the following command to execute the pre-upgrade
script:
apicops version:pre-upgrade -n <namespace>
If the system is healthy, the results will not include any errors.
Note: This command asks for the Cloud Manager admin password in order to make a call to the platform API. If you do not have this password, or the platform API URL is not accessible from yourapicops
location, then this part of the pre-upgrade check can be skipped by adding the argument--no-topology
. The topology check output that is produced by this API call is typically only required for debugging purposes, when theapicops version:pre-upgrade
returns output that suggests the system is unhealthy.
- Verify that the
- Obtain the API Connect files from IBM Fix Central.
From the IBM Fix Central site, download the Docker image-tool file of the API Connect subsystems. Next, you will upload the image-tool file to your Docker local registry. If necessary, you can populate a remote container registry with repositories. Then you can push the images from the local registry to the remote registry.
You will also download the Kubernetes operators, API Connect Custom Resource (CR) templates, and Certificate Manager, for use during deployment configuration.
The following files are used for deployment on native Kubernetes:
- IBM® API Connect <version> for Containers
- Docker images for all API Connect subsystems
- IBM® API Connect <version> Operator Release Files for Containers
- Kubernetes operators and API Connect Custom Resource (CR) templates
- IBM® API Connect <version> Toolkit for <operating_system_type>
- Toolkit command line utility. Packaged standalone, or with API Designer or Loopback:
- IBM® API Connect <version> Toolkit for <operating_system_type>
- IBM® API Connect <version> Toolkit with Loopback for <operating_system_type>
- IBM® API Connect <version> Toolkit Designer with Loopback for <operating_system_type>
Not required during initial installation. After installation, you can download directly from the Cloud Manager UI and API Manager UI. See Installing the toolkit.
- IBM® API Connect <version> Local Test Environment
- Optional test environment. See Testing an API with the Local Test Environment
- IBM® API Connect <version> Security Signature Bundle File
- Checksum files that you can use to verify the integrity of your downloads.
- Next, upload the image files that you obtained from Fix Central in Step 3.
- Load the image-tool image for the new version into your Docker local
registry:
docker load < apiconnect-image-tool-<version>.tar.gz
Ensure that the registry has sufficient disk space for the files.
- If your Docker registry requires repositories to be created before images
can be pushed, create the repositories for each of the images listed by the image tool. If your
Docker registry does not require creation of repositories, skip this step and go to Step 4.c.
- Run the following command to get a list of the images from
image-tool:
docker run --rm apiconnect-image-tool-<version> version --images
- From the output of each entry of the form
<image-name>:<image-tag>
, use your Docker registry repository creation command to create a repository for<image-name>
.For example in the case of AWS ECR the command would be for each<image-name>
:aws ecr create-repository --repository-name <image-name>
- Run the following command to get a list of the images from
image-tool:
- Upload the image:
- If you do not need to authenticate with the docker registry,
use:
docker run --rm apiconnect-image-tool-<version> upload <registry-url>
- Otherwise, if your docker registry accepts authentication with username and password arguments,
use:
docker run --rm apiconnect-image-tool-<version> upload <registry-url> --username <username> --password <password>
- Otherwise, such as with IBM Container Registry, if you need the image-tool to use your local
Docker credentials, first authenticate with your Docker registry, then upload images with the
command:
docker run --rm -v ~/.docker:/root/.docker --user 0 apiconnect-image-tool-<version> upload <registry-url>
Review the following installation notes as appropriate for your environment:
- If you do not need to authenticate with the docker registry,
use:
- Load the image-tool image for the new version into your Docker local
registry:
- If needed, delete old Postgres client certificates. If you are upgrading from 10.0.1.x or 10.0.4.0-ifix1, or if you previously installed any of those versions before upgrading to 10.0.5.x, there might be old Postgres client certificates. To verify, run the following command:
kubectl -n <namespace> get certs | grep db-client
For example, if you see that both
-db-client-apicuser
andapicuser
exist,apicuser
is no longer in use. Remove the old certificates by running one of the following commands, depending on how many old certifications left in your system:kubectl -n <namespace> delete certs apicuser pgbouncer primaryuser postgres replicator
or:
kubectl -n <namespace> delete certs apicuser pgbouncer postgres replicator
- If upgrading from v10.0.4-ifix3, or
upgrading from v10.0.1.7-eus (or higher) and you want to retain
your analytics data, then you must export it before running the upgrade.
For instructions on exporting data, see Additional considerations for upgrading analytics from a version prior to 10.0.5.0.
- If upgrading from v10.0.4-ifix3, or
upgrading from v10.0.1.7-eus (or higher): Disassociate and
delete your Analytics services.
- In Cloud Manager, click Topology.
- In the section for the Availability Zone that contains the Analytics service, locate the Gateway service that the Analytics service is associated with.
- Click the actions menu, and select Unassociate analytics
service. Remember to disassociate each Analytics service from all Gateways.
- In the section for the Availability Zone that contains the Analytics services, locate each Analytics service and click Delete.
- Download and decompress IBM API Connect <version> Operator Release
Files for Containers.
Make a directory called
helper_files
and extract the contents ofhelper_files.zip
from the release_files.zip intohelper_files
. - Apply the new CRDs from the version you just extracted:
kubectl apply -f ibm-apiconnect-crds.yaml
- Apply the new DataPower Operator YAML into the namespace where the DataPower Operator is
running.
- Verify that the correct namespace is referenced in the
ibm-datapower.yaml
file:If your product was not deployed in the namespace called
default
, open theibm-datapower.yaml
file in a text editor and replace all instances ofdefault
with the appropriate name of the namespace where you deployed the product. - Specify the location of the
datapower-operator
image:Open the
ibm-datapower.yaml
file in a text editor and locate theimage:
key in thecontainers
section of the deployment file (immediately afterimagePullSecrets:
). Replace the value of theimage:
key with the location of thedatapower-operator
image, either uploaded to your own registry or pulled from a public registry. - Run the following command:
kubectl apply -f ibm-datapower.yaml -n <namespace>
The Gateway CR goes to
Pending
state when the operator is updated, and then changes toRunning
after installation of the API Connect operator in the next step.Note: There is a known issue on Kubernetes version 1.19.4 or higher that can cause the DataPower operator to fail to start. In this case, the DataPower Operator pods can fail to schedule, and will display the status message:no nodes match pod topology spread constraints (missing required label)
. For example:0/15 nodes are available: 12 node(s) didn't match pod topology spread constraints (missing required label), 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
You can workaround the issue by editing the DataPower operator deployment and re-applying it, as follows:
- Delete the DataPower operator deployment, if deployed
already:
kubectl delete -f ibm-datapower.yaml -n <namespace>
- Open
ibm-datapower.yaml
, and locate thetopologySpreadConstraints:
section. For example:topologySpreadConstraints: - maxSkew: 1 topologyKey: zone whenUnsatisfiable: DoNotSchedule
- Replace the values for
topologyKey:
andwhenUnsatisfiable:
with the corrected values shown in the example below:topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway
- Save
ibm-datapower.yaml
and deploy the file to the cluster:kubectl apply -f ibm-datapower.yaml -n <namespace>
- Delete the DataPower operator deployment, if deployed
already:
- Verify that the correct namespace is referenced in the
- Upgrade cert-manager to version 1.12.13.
- Run the following command to back up the existing certificates and issuers to a file
called backup.yaml:
kubectl get --all-namespaces -oyaml issuer,clusterissuer,cert,secret > backup.yaml
- Run the following command to determine the version of the existing cert-manager:
kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
- If the current version is older than 1.12.13, run the
following command to upgrade cert-manager to version 1.12.13:
kubectl apply -f helper-files/cert-manager-1.12.13.yaml
- Run the following command to back up the existing certificates and issuers to a file
called backup.yaml:
- If you used customized internal certificates and are upgrading from
10.0.5.1: The
helper_files/custom-certs-internal.yaml
includes a new certificate calleddbClientPrimaryuser
. This new certificate must be created before you update the Management CR.Follow the instructions in Generate custom internal certificates to update thehelper_files/custom-certs-internal.yaml
file to match your namespace and site name, and then apply the newyaml
file with the following command:kubectl apply -f custom-certs-internal.yaml -n <namespace>
Alternatively, add the new certificate to your existingcustom-certs-internal.yaml
and apply it, updating the namespace to match your deployment:--- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: db-client-primaryuser labels: { app.kubernetes.io/instance: "management", app.kubernetes.io/managed-by: "ibm-apiconnect", app.kubernetes.io/name: "db-client-primaryuser" } spec: commonName: primaryuser secretName: db-client-primaryuser dnsNames: - "*.<namespace>" - "*.<namespace>.svc" - "primaryuser.<namespace>.svc" - "primaryuser" issuerRef: name: ingress-issuer usages: - "client auth" - "signing" - "key encipherment" duration: 17520h # 2 years renewBefore: 720h # 30 days privateKey: rotationPolicy: Always ---
- Clean up the webhook configuration before deploying the newer API Connect operator:
- Run the following command to get the webhook configuration:
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration | grep ibm-apiconnect
- Use the
kubectl delete
command to delete the webhook configuration; for example:kubectl delete mutatingwebhookconfiguration ibm-apiconnect-mutating-webhook-configuration kubectl delete validatingwebhookconfiguration ibm-apiconnect-validating-webhook-configuration
- Run the following command to get the webhook configuration:
- Apply the new API Connect operator YAML into the namespace where the API
Connect operator is running.
- For single namespace deployment:
- If the operator is not running in the
default
namespace, open theibm-apiconnect.yaml
file in a text editor, and then replace all references todefault
with the name of the namespace where you deployed API Connect.Note: Skip this step if you are using Operator Lifecycle Manager (OLM). - Open
ibm-apiconnect.yaml
in a text editor. Replace the value of eachimage:
key with the location of theapiconnect
operator images (from theibm-apiconnect
container and theibm-apiconnect-init
container), either uploaded to your own registry or pulled from a public registry. - Run the following command:
kubectl apply -f ibm-apiconnect.yaml -n <namespace>
- If the operator is not running in the
- For multi-namespace deployment:
- Locate and open the newly downloaded
ibm-apiconnect-distributed.yaml
in a text editor of choice. Then, find and replace each occurrence of$OPERATOR_NAMESPACE
with the desired namespace for the deployment. - Also in
ibm-apiconnect-distributed.yaml
, locate theimage:
keys in the containers sections of the deployment yaml right belowimagePullSecrets:
. Replace the placeholder valuesREPLACE-DOCKER-REGISTRY
of theimage:
keys with the docker registry host location of the API Connect operator image (either uploaded to own registry or pulled from public registry). - Install
ibm-apiconnect-distributed.yaml
with the following commandkubectl apply -f ibm-apiconnect-distributed.yaml
- Locate and open the newly downloaded
- For single namespace deployment:
- Verify that the
ibm-datapower-operator
and theibm-apiconnect
operators are restarted. - Ensure that the
apiconnect
operator recreated the necessary microservices:Be sure to complete this step before attempting to upgrade the operands.
kubectl get apic -n <namespace>
- Upgrade the operands (subsystems):
- Verify that the upgraded subsystems report as
Running
and theRECONCILED VERSION
displays the new version of API Connect.Run the following command:
kubectl get apic --all-namespaces
Example response:
NAME READY STATUS VERSION RECONCILED VERSION AGE analyticscluster.analytics.apiconnect.ibm.com/analytics 8/8 Running 10.0.5.8 10.0.5.8-1074 121m NAME PHASE READY SUMMARY VERSION AGE datapowerservice.datapower.ibm.com/gw1 Running True StatefulSet replicas ready: 1/1 10.0.5.8 100m NAME PHASE LAST EVENT WORK PENDING WORK IN-PROGRESS AGE datapowermonitor.datapower.ibm.com/gw1 Running false false 100m NAME READY STATUS VERSION RECONCILED VERSION AGE gatewaycluster.gateway.apiconnect.ibm.com/gw1 2/2 Running 10.0.5.8 10.0.5.8-1074 100m NAME READY STATUS VERSION RECONCILED VERSION AGE managementcluster.management.apiconnect.ibm.com/m1 16/16 Running 10.0.5.8 110.0.5.8-1074 162m NAME READY STATUS VERSION RECONCILED VERSION AGE portalcluster.portal.apiconnect.ibm.com/portal 3/3 Running 10.0.5.8 10.0.5.8-1074 139m
Important: If you need to restart the deployment, wait until all Portal sites complete the upgrade. Run the following commands to check the status of the sites:- Log in as an admin
user:
apic login -s <server_name> --realm admin/default-idp-1 --username admin --password <password>
- Get the portal service ID and
endpoint:
apic portal-services:get -o admin -s <management_server_endpoint> \ --availability-zone availability-zone-default <portal-service-name> \ --output - --format json
- List the
sites:
apic --mode portaladmin sites:list -s <management_server_endpoint> \ --portal_service_name <portal-service-name> \ --format json
Any sites currently upgrading display the
UPGRADING
status; any site that completed its upgrade displays theINSTALLED
status and the new platform version. Verify that all sites display theINSTALLED
status before proceeding.For more information on the
sites
command, seeapic sites:list
and Using thesites
commands. - After all sites are in
INSTALLED
state and have the new platform listed, run:apic --mode portaladmin platforms:list -s <server_name> --portal_service_name <portal_service_name>
Verify that the new version of the platform is the only platform listed.
For more information on the
platforms
command, seeapic platforms:list
and Using theplatforms
commands.
- Log in as an admin
user:
- If you are upgrading from v10.0.4-ifix3, or upgrading from v10.0.1.7-eus (or higher): Enable analytics as explained in Enabling Analytics after upgrading.
- If you are upgrading from v10.0.5.3 or earlier, then you must update
your ingress certificates.
- If all API Connect
subsystems are in the same namespace, then update the certificates with the
ingress-issuer-v1.yaml from your target
release.
kubectl apply -f helper_files/ingress-issuer-v1.yaml -n <namespace>
- For a multi-namespace deployment, use the helper_files/multi-ns-support files, and follow the steps: 2.b to 2.f on Deploying operators in a multi-namespace API Connect cluster.
- If all API Connect
subsystems are in the same namespace, then update the certificates with the
ingress-issuer-v1.yaml from your target
release.
- Optional: For the optional components API Connect Toolkit and API Connect Local Test Environment, install the latest version of each after you complete the upgrade of the subsystems.
- Optional: Configure additional features related to inter-subsystem communication security, such as CA verification and JWT security: Optional post-upgrade steps for upgrade to 10.0.5.3 (or later) from earlier 10.0.5 release.