Upgrading to the latest version of 10.0.1.x-eus on Kubernetes

Upgrade your API Connect deployment to the latest version of 10.0.1.x-eus on native Kubernetes.

1. Prepare for upgrading the management subsystem:
   1. Verify that the pgcluster is healthy:
      1. Get the name of the pgcluster:

         kubectl get pgcluster -n <APIC_namespace>

         The response displays the name of the postgres cluster running in the specified namespace.

      2. Check the status of the pgcluster:

         kubectl get pgcluster <pgcluster_name> -n <APIC_namespace> -o yaml | grep status -A 2 | tail -n3

         The response for a healthy pgcluster looks like the following example, where the state is Initialized:

         status:
             message: Cluster has been initialized
             state: pgcluster Initialized

      Important: If the pgcluster returns any other state, it is not healthy and an upgrade will fail.

      • If there are any ongoing backup or restore jobs, wait until they complete and then check the status again. Do not proceed with the upgrade until the status is Initialized.
      • If all of the background jobs complete but the pgcluster remains unhealthy, contact IBM Support for assistance.
2. Backup the current deployment.
   • Wait until the backup completes before starting the upgrade.
   • Do not start an upgrade if a backup is scheduled to run within a few hours.
   • Do not perform maintenance tasks )such as rotating keys and certificates, restoring from a backup, or starting a new backup) while the upgrade process is running.
3. If you used any microservice image overrides in the management CR during a fresh install, remove the image overrides prior to upgrade.
   Important: If you used any microservice image overrides in the management CR during a fresh install, these image overrides will be automatically removed by the operator during upgrade. You can apply them again after the upgrade is complete.
4. Run the pre-upgrade health check:
   1. Verify that the apicops utility is installed by running the following command to check the current version of the utility:

      apicops --version

      If the response indicates that apicops is not available, install it now. See The API Connect operations tool: apicops in the API Connect documentation.

   2. Run the following command to set the KUBECONFIG environment.

      export KUBECONFIG=/path/to/kubeconfig

   3. Run the following command to execute the pre-upgrade script:

      apicops version:pre-upgrade -n <namespace>

      If the system is healthy, the results will not include any errors.

5. Obtain the API Connect files from IBM Fix Central.

   From the IBM Fix Central site, download the Docker image-tool file of the API Connect subsystems. Next, you will upload the image-tool file to your Docker local registry. If necessary, you can populate a remote container registry with repositories. Then you can push the images from the local registry to the remote registry.

   You will also download the Kubernetes operators, API Connect Custom Resource (CR) templates, and Certificate Manager, for use during deployment configuration.

   The following files are used for deployment on native Kubernetes:

   IBM® API Connect <version> for Containers

   Docker images for all API Connect subsystems

   IBM® API Connect <version> Operator Release Files for Containers

   Kubernetes operators and API Connect Custom Resource (CR) templates

   IBM® API Connect <version> Toolkit for <operating_system_type>

   Toolkit command line utility.

   Not required during initial installation. After installation, you can download the toolkit directly from the Cloud Manager UI and API Manager UI. See Installing the toolkit for instructions.

   The Toolkit is packaged standalone, with Loopback, and with API Designer and Loopback:

   • IBM® API Connect <version> Toolkit for <operating_system_type>
   • IBM® API Connect <version> Toolkit with Loopback for <operating_system_type>
   • IBM® API Connect <version> Toolkit Designer with Loopback for <operating_system_type>

   IBM® API Connect <version> Local Test Environment

   Optional test environment. See Testing an API with the Local Test Environment

   IBM® API Connect <version> Security Signature Bundle File

   Checksum files that you can use to verify the integrity of your downloads.

6. Upload the image files that you obtained from Fix Central.
   1. Load the image-tool image into your Docker local registry:

      docker load < apiconnect-image-tool-<version>.tar.gz 

      Ensure that the registry has sufficient disk space for the files.

   2. If your Docker registry requires repositories to be created before images can be pushed, create the repositories for each of the images listed by the image tool. If your Docker registry does not require creation of repositories, skip this step.
      1. Run the following command to get a list of the images from image-tool:

         docker run --rm apiconnect-image-tool-<version> version --images

      2. From the output of each entry of the form <image-name>:<image-tag>, use your Docker registry repository creation command to create a repository for <image-name>.
         For example in the case of AWS ECR the command would be for each <image-name>:

         aws ecr create-repository --repository-name <image-name>

   3. Upload the image:
      • If you do not need to authenticate with the docker registry, use:

        docker run --rm apiconnect-image-tool-<version> upload <registry-url>

      • Otherwise, if your docker registry accepts authentication with username and password arguments, use:

        docker run --rm apiconnect-image-tool-<version> upload <registry-url> --username <username> --password <password>

      • Otherwise, such as with IBM Container Registry, if you need the image-tool to use your local Docker credentials, first authenticate with your Docker registry, then upload images with the command:

        docker run --rm -v ~/.docker:/root/.docker --user 0 apiconnect-image-tool-<version> upload <registry-url>

        If necessary, review the following installation notes:

        • Alternative Docker command for macOS if Docker is configured to use osxkeychain
        • Docker authentication notes
        • Providing a certificate for verification, or disabling TLS verification
7. Download and decompress IBM® API Connect <version> Operator Release Files for Containers.
8. If you are upgrading from 10.0.1.7-eus, upgrade cert-manager to version 1.7.1.

   API Connect 10.0.1.7-eus used cert-manager 1.5.1.

   1. Create a directory named helper_files.
   2. Extract the contents of the helper_files.zip from the release_files.zip file into the new helper_files directory.
   3. Run the following command to back up the existing certificates and issuers to a file called backup.yaml:

      kubectl get --all-namespaces -oyaml issuer,clusterissuer,cert,secret > backup.yaml

   4. Run the following command to verify that you are upgrading from cert-manager 1.5.1:

      kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
      v1.5.1

   5. Run the following command to upgrade cert-manager 1.5.1 to version 1.7.1:

      kubectl apply -f helper_files/cert-manager-1.7.1.yaml

9. If you used customized internal certificates, and are upgrading from 10.0.1.8: The helper_files/custom-certs-internal.yaml includes a new certificate that is called dbClientPrimaryuser. This new certificate must be created before you update the Management CR.
   Follow Generate custom internal certificates to update the helper_files/custom-certs-internal.yaml file to match your namespace and site name, and apply the new yaml file with:

   kubectl apply -f custom-certs-internal.yaml -n <namespace>

   Alternatively, add the new certificate to your existing custom-certs-internal.yaml and apply it, updating the namespace to match your deployment:

   ---
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: db-client-primaryuser
     labels: {
       app.kubernetes.io/instance: "management",
       app.kubernetes.io/managed-by: "ibm-apiconnect",
       app.kubernetes.io/name: "db-client-primaryuser"
     }
   spec:
     commonName: primaryuser
     secretName: db-client-primaryuser
     dnsNames:
       - "*.<namespace>"
       - "*.<namespace>.svc"
       - "primaryuser.<namespace>.svc"
       - "primaryuser"
     issuerRef:
       name: ingress-issuer
     usages:
     - "client auth"
     - "signing"
     - "key encipherment"
     duration: 17520h # 2 years
     renewBefore: 720h # 30 days
   ---
   

10. Apply the new CRDs from the Operator Release Files for Containers that you decompressed:

    kubectl apply -f ibm-apiconnect-crds.yaml

11. Apply the new DataPower Operator YAML into the namespace where the DataPower Operator is running.
    1. If the operator is not running in the default namespace, open the ibm-datapower.yaml file in a text editor and find and replace all references to default the name of your namespace. You do not need to take this action when using Operator Lifecycle Manager (OLM).
    2. Open ibm-datapower.yaml in a text editor. Locate the image: key in the containers section of the deployment YAML file immediately after imagePullSecrets:. Replace the value of the image: key with the location of the datapower-operator image, either uploaded to your own registry or pulled from a public registry.
    3. kubectl apply -f ibm-datapower.yaml -n <namespace>

       The Gateway CR goes to Pending state when the operator is updated. The state of the Gateway CR will change to Running after installation of the API Connect operator in the next step.

       For information on troubleshooting the DataPower operator, see Operator not removed in the DataPower operator documentation.

12. Apply the new API Connect operator YAML into the namespace where the API Connect operator is running.
    • Single-namespace deployment:
      1. If the operator is not running in the default namespace, open the ibm-apiconnect.yaml file in a text editor and find and replace all references to default the name of your namespace. You do not need to take this action when using Operator Lifecycle Manager (OLM).
      2. Open ibm-apiconnect.yaml in a text editor. Replace the value of each image: key with the location of the apiconnect operator images (from the ibm-apiconnect container and the ibm-apiconnect-init container), either uploaded to your own registry or pulled from a public registry.
      3. kubectl apply -f ibm-apiconnect.yaml -n <namespace>
    • Multi-namespace deployment:
      1. Locate and open the newly downloaded ibm-apiconnect-distributed.yaml in a text editor of choice. Then, find and replace each occurrence of $OPERATOR_NAMESPACE with <namespace>, replacing <namespace> with the desired namespace for the deployment. In this example, the namespace is operator.
      2. Also in ibm-apiconnect-distributed.yaml, locate the image: keys in the containers sections of the deployment yaml right below imagePullSecrets:. Replace the placeholder values REPLACE-DOCKER-REGISTRY of the image: keys with the docker registry host location of the apiconnect operator image (either uploaded to own registry or pulled from public registry).
      3. Install ibm-apiconnect-distributed.yaml with the following command

         kubectl apply -f ibm-apiconnect-distributed.yaml

    When the API Connect operator deployment is updated, it detects that the existing version 10 pods for all subsystems have labels that no longer match, and tries to fix labels. When fixing the labels, most of the microservices (for all subsystems) are recreated. All the subsystem CRs go into Pending state and then into Running state. Management subsystem microservices are recreated, with the exception of postgres/NATS components, at the end of the process.

13. Verify that all microservices and clusters report Running:
    1. Run the following command to check the deployment's status:

       kubectl get apic -n <namespace>

       All systems must report Running before you proceed with the upgrade.

    2. If any cluster reports Pending, run the following apicops (v10 version 0.10.57+ required) command to validate the certificates:

       apicops upgrade:stale-certs -n <namespace>

       For information on installing and using apicops, see The API Connect operations tool: apicops.

    3. If any certificate that is managed by cert-manager failed the validation, run the following command to delete the stale certificate secret and let cert-manager regenerate it:
       kubectl delete secret <stale-secret> -n <namespace>
    4. Run the following command to check the deployment's status again:

       kubectl get apic -n <namespace>

       All systems must report Running before you proceed with the upgrade.

14. Upgrade the subsystems (operands) by updating the CRs.
    Important: When you save each updated CR, the version change is detected by the operator, which triggers the subsystem upgrade. To ensure a successful upgrade, update the subsystem CRs in the following required sequence:

    1. Management
    2. Portal
    3. Analytics
    4. Gateway

    For each CR, make the following changes:

    1. Update the version to reflect the new release of API Connect:

       For example:

       version: 10.0.1.12-eus

    2. In the Gateway CR, remove the template override section, if it exists.

       You cannot perform an upgrade if the CR contains an override.

15. Verify that the upgraded subsystems report as Running and the RECONCILED VERSION displays the new version of API Connect.

    Run the following command:

    kubectl get apic --all-namespaces

    Example response:

    NAME                                                READY   STATUS    VERSION              RECONCILED VERSION      AGE
    analyticscluster.analytics.apiconnect.ibm.com/analytics      8/8     Running   10.0.1.12-eus   10.0.1.12-eus-1074   121m
    
    NAME                                     PHASE     READY   SUMMARY                           VERSION    AGE
    datapowerservice.datapower.ibm.com/gw1   Running   True    StatefulSet replicas ready: 1/1   10.0.1.12-eus   100m
    
    NAME                                     PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
    datapowermonitor.datapower.ibm.com/gw1   Running                false          false              100m
    
    NAME                                            READY   STATUS    VERSION              RECONCILED VERSION      AGE
    gatewaycluster.gateway.apiconnect.ibm.com/gw1   2/2     Running   10.0.1.12-eus   10.0.1.12-eus-1074  100m
    
    NAME                                                 READY   STATUS    VERSION              RECONCILED VERSION      AGE
    managementcluster.management.apiconnect.ibm.com/m1   16/16   Running   10.0.1.12-eus   110.0.1.12-eus-1074   162m
    
    
    NAME                                             READY   STATUS    VERSION              RECONCILED VERSION      AGE
    portalcluster.portal.apiconnect.ibm.com/portal   3/3     Running   10.0.1.12-eus   10.0.1.12-eus-1074   139m

16. Optional: Install the latest version of the API Connect Toolkit and the API Connect Local Test Environment.