Upgrading subsystems on native Kubernetes

Upgrade your API Connect subsystems to the latest version on native Kubernetes.

Before you begin

To ensure operator compatibility, upgrade API Connect management subsystem first and DataPower gateway subsystem second. This requirement applies to all upgrade scenarios.
The Gateway subsystem remains available during the upgrade of the Management, Portal, and Analytics subsystems.
Check your installed Kubernetes version and ensure it is supported by the latest API Connect 10.0.1.x-eus release you are upgrading to, check IBM API Connect Version 10 software product compatibility requirements. If your current Kubernetes version is not supported then upgrade Kubernetes before API Connect.

Procedure

Complete the prerequisites:
1. Ensure you are upgrading from a supported version and reviewed the upgrade requirements and limitations. See Upgrade considerations on native Kubernetes.
2. Verify that the pgcluster is healthy:
  1. Get the name of the pgcluster:
```
kubectl get pgcluster -n <APIC_namespace>
```
    The response displays the name of the postgres cluster running in the specified namespace.
  2. Check the status of the pgcluster:
```
kubectl get pgcluster <pgcluster_name> -n <APIC_namespace> -o yaml | grep status -A 2 | tail -n3
```
    The response for a healthy pgcluster looks like the following example, where the state is Initialized:
```
status:
    message: Cluster has been initialized
    state: pgcluster Initialized
```
  Important: If the pgcluster returns any other state, it is not healthy and an upgrade will fail.
  - If there are any ongoing backup or restore jobs, wait until they complete and then check the status again. Do not proceed with the upgrade until the status is Initialized.
  - If all of the background jobs complete but the pgcluster remains unhealthy, contact IBM Support for assistance.
3. Backup the current deployment. Wait until the backup completes before starting the upgrade.
  - Do not start an upgrade if a backup is scheduled to run within a few hours.
  - Do not perform maintenance tasks such as rotating key-certificates, restoring from a backup, or starting a new backup, at any time while the upgrade process is running.
4. If you used any microservice image overrides in the management CR during a fresh install, you must remove the image overrides prior to upgrade.
  Important: When upgrading, if you used any microservice image overrides in the management CR during a fresh install then prior to upgrade these image overrides will be automatically removed by the operator during upgrade. You can apply them again after the upgrade is complete.
5. Back up API test data in the Automated API behavior testing application as explained in Backing up API test data. After the upgrade is complete, you can migrate the data as explained in Restoring API test data.
Run the pre-upgrade health check:
1. Verify that the apicops utility is installed by running the following command to check the current version of the utility:
```
apicops --version
```
  If the response indicates that apicops is not available, install it now. See The API Connect operations tool: apicops in the API Connect documentation.
2. Run the following command to set the KUBECONFIG environment.
```
export KUBECONFIG=/path/to/kubeconfig
```
3. Run the following command to execute the pre-upgrade script:
```
apicops version:pre-upgrade -n <namespace>
```
  If the system is healthy, the results will not include any errors.
Obtain the API Connect files from IBM Fix Central.

From the IBM Fix Central site, download the Docker image-tool file of the API Connect subsystems. Next, you will upload the image-tool file to your Docker local registry. If necessary, you can populate a remote container registry with repositories. Then you can push the images from the local registry to the remote registry.

You will also download the Kubernetes operators, API Connect Custom Resource (CR) templates, and Certificate Manager, for use during deployment configuration.
The following files are used for deployment on native Kubernetes:
IBM® API Connect <version> for Containers

Docker images for all API Connect subsystems

IBM® API Connect <version> Operator Release Files for Containers

Kubernetes operators and API Connect Custom Resource (CR) templates

IBM® API Connect <version> Toolkit for <operating_system_type>
Toolkit command line utility. Packaged standalone, or with API Designer or Loopback:
- IBM® API Connect <version> Toolkit for <operating_system_type>
- IBM® API Connect <version> Toolkit with Loopback for <operating_system_type>
- IBM® API Connect <version> Toolkit Designer with Loopback for <operating_system_type>
Not required during initial installation. After installation, you can download directly from the Cloud Manager UI and API Manager UI. See Installing the toolkit.
IBM® API Connect <version> Local Test Environment

Optional test environment. See Testing an API with the Local Test Environment

IBM® API Connect <version> Security Signature Bundle File

Checksum files that you can use to verify the integrity of your downloads.
Backup the API Connect subsystems.

The upgrade supports rollback if the upgrade encounters problems.
Next, upload the image files that you obtained from Fix Central in Step 3.
1. Load the image-tool image for the new version into your Docker local registry:
```
docker load < apiconnect-image-tool-<version>.tar.gz 
```
  Ensure that the registry has sufficient disk space for the files.
2. If your Docker registry requires repositories to be created before images can be pushed, create the repositories for each of the images listed by the image tool. If your Docker registry does not require creation of repositories, skip this step and go to Step 5.c.
  1. Run the following command to get a list of the images from image-tool:
```
docker run --rm apiconnect-image-tool-<version> version --images
```
  2. From the output of each entry of the form <image-name>:<image-tag>, use your Docker registry repository creation command to create a repository for <image-name>.
    For example in the case of AWS ECR the command would be for each <image-name>:
    aws ecr create-repository --repository-name <image-name>
3. Upload the image:
  - If you do not need to authenticate with the docker registry, use:
```
docker run --rm apiconnect-image-tool-<version> upload <registry-url>
```
  - Otherwise, if your docker registry accepts authentication with username and password arguments, use:
```
docker run --rm apiconnect-image-tool-<version> upload <registry-url> --username <username> --password <password>
```
  - Otherwise, such as with IBM Container Registry, if you need the image-tool to use your local Docker credentials, first authenticate with your Docker registry, then upload images with the command:
```
docker run --rm -v ~/.docker:/root/.docker --user 0 apiconnect-image-tool-<version> upload <registry-url>
```
    If necessary, review the following installation notes:
Download and decompress IBM® API Connect <version> Operator Release Files for Containers.

Make a directory helper_files and extract the contents of helper_files.zip from the release_files.zip into helper_files.

Upgrade cert-manager from v0.10.1 to v1.5.1.

Backup the existing certificates and issuers

kubectl get --all-namespaces -oyaml issuer,clusterissuer,cert > backup.yaml

Delete the old issuer and certs

kubectl delete issuers.certmanager.k8s.io selfsigning-issuer ingress-issuer -n <namespace>
kubectl delete certs.certmanager.k8s.io ingress-ca -n <namespace>
kubectl delete certs.certmanager.k8s.io portal-admin-client gateway-client-client analytics-client-client analytics-ingestion-client -n <namespace>

Delete the old cert-manager resources including the namespace:
```
kubectl delete -f helper_files/cert-manager-0.10.1-deploy.yaml
```
Note that the old cert-manager CRDs are not deleted.

Wait and make sure namespace cert-manager has been deleted:
```
$ kubectl get ns cert-manager
Error from server (NotFound): namespaces "cert-manager" not found
```

Install the new cert-manager v1.5.1

kubectl apply -f helper_files/cert-manager-1.5.1.yaml

Apply the new CRDs from the version you just extracted:
```
kubectl apply -f ibm-apiconnect-crds.yaml
```
Apply the new DataPower Operator YAML into the namespace where the DataPower Operator is running.
1. If the operator is not running in the default namespace, open the ibm-datapower.yaml file in a text editor and find and replace all references to default the name of your namespace. You do not need to take this action when using Operator Lifecycle Manager (OLM).
2. Open ibm-datapower.yaml in a text editor. Locate the image: key in the containers section of the deployment yaml immediately after imagePullSecrets:. Replace the value of the image: key with the location of the datapower-operator image, either uploaded to your own registry or pulled from a public registry.
3. kubectl apply -f ibm-datapower.yaml -n <namespace>
  The Gateway CR goes to Pending state when the operator is updated. The state of the Gateway CR will change to Running after installation of the API Connect operator in the next step.
  Note: There is a known issue on Kubernetes version 1.19.4 or higher that can cause the DataPower operator to fail to start. In this case, the DataPower Operator pods can fail to schedule, and will display the status message: no nodes match pod topology spread constraints (missing required label). For example:
```
0/15 nodes are available: 12 node(s) didn't match pod topology spread constraints (missing required label), 
3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
```
  You can workaround the issue by editing the DataPower operator deployment and re-applying it, as follows:
  1. Delete the DataPower operator deployment, if deployed already:
    kubectl delete -f ibm-datapower.yaml -n <namespace>
  2. Open ibm-datapower.yaml, and locate the topologySpreadConstraints: section. For example:
    topologySpreadConstraints: - maxSkew: 1 topologyKey: zone whenUnsatisfiable: DoNotSchedule
  3. Replace the values for topologyKey: and whenUnsatisfiable: with the corrected values shown in the example below:
    topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway
  4. Save ibm-datapower.yaml and deploy the file to the cluster:
    kubectl apply -f ibm-datapower.yaml -n <namespace>

Clean up the webhook configuration before deploying the newer apiconnect operator

kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration | grep ibm-apiconnect
kubectl delete mutatingwebhookconfiguration/ibm-apiconnect-<namespace>-mutating-webhook-configuration
kubectl delete validatingwebhookconfiguration/ibm-apiconnect-<namespace>-validating-webhook-configuration

Install the new issuers:

kubectl apply -f helper_files/ingress-issuer-v1.yaml -n <namespace>

Apply the new API Connect operator yaml into the namespace where the API Connect operator is running.
- For single namespace deployment:
  1. If the operator is not running in the default namespace, open the ibm-apiconnect.yaml file in a text editor and find and replace all references to default the name of your namespace. You do not need to take this action when using Operator Lifecycle Manager (OLM).
  2. Open ibm-apiconnect.yaml in a text editor. Replace the value of each image: key with the location of the apiconnect operator images (from the ibm-apiconnect container and the ibm-apiconnect-init container), either uploaded to your own registry or pulled from a public registry.
  3. kubectl apply -f ibm-apiconnect.yaml -n <namespace>
- For multi-namespace deployment:
  1. Locate and open the newly downloaded ibm-apiconnect-distributed.yaml in a text editor of choice. Then, find and replace each occurrence of $OPERATOR_NAMESPACE with <namespace>, replacing <namespace> with the desired namespace for the deployment. In this example, the namespace is operator.
  2. Also in ibm-apiconnect-distributed.yaml, locate the image: keys in the containers sections of the deployment yaml right below imagePullSecrets:. Replace the placeholder values REPLACE-DOCKER-REGISTRY of the image: keys with the docker registry host location of the apiconnect operator image (either uploaded to own registry or pulled from public registry).
  3. Install ibm-apiconnect-distributed.yaml with the following command
```
kubectl apply -f ibm-apiconnect-distributed.yaml
```
Use apicops to validate the certificates.
1. Run the following command:
```
apicops upgrade:stale-certs -n <namespace>
```
2. Delete any stale certificates that are managed by cert-manager.
  If a certificate failed the validation and it is managed by cert-manager, you can delete the stale certificate secret, and let cert-manager regenerate it. Run the following command:
```
kubectl delete secret <stale-secret> -n <namespace>
```
3. Restart the corresponding so that it can pick up the new secret.
  To determine which pod to restart, see the following topics:
  - List of external certificates
  - List of internal certificates
For information on the apicops tool, see The API Connect operations tool: apicops.
Verify that both operators are restarted.

Run the following command to delete the Postgres pods, which refreshes the new certificate:

kubectl get pod -n <namespace> --no-headers=true | grep postgres | grep -v backup | awk '{print $1}' | xargs kubectl delete pod -n <namespace>

Prior to upgrading the operands (subsystems), ensure that the apiconnect operator re-created the necessary microservices as part of the label updates in step 12:
```
kubectl get apic -n <my-cluster-name>
```
Upgrade the operands (subsystems):
Optional: For the optional components API Connect Toolkit and API Connect Local Test Environment, install the latest version of each after you complete upgrade of the subsystems.