Upgrading API Connect on Kubernetes

Upgrade your API Connect subsystems to a newer version on Kubernetes.

1. If you did not run the pre-upgrade health checks recently, run them now.
2. If you have a two data center disaster recovery deployment, then upgrade the warm-standby data center first.
   See the special steps for 2DCDR upgrades: Upgrading a 2DCDR deployment on Kubernetes and OpenShift from V10.0.9
3. Apply the API Connect CRDs that you downloaded and extracted in the pre-upgrade steps:

   kubectl apply --server-side --force-conflicts -f ibm-apiconnect-crds.yaml

4. Apply the new DataPower Operator YAML into the namespace where the DataPower operator is running.
   1. Edit the ibm-datapower.yaml file and set the following properties:the namespace property to the namespace where you deployed the operator.
      • Set all instances of the namespace property to the namespace where you deployed the operator.
      • Set the containers.image property to the location of the datapower-operator image, either uploaded to your own registry or pulled from a public registry.
   2. Run the following command:

      kubectl -n <namespace> apply -f ibm-datapower.yaml

      The gateway CR goes to Pending state when the operator is updated, and then changes to Running after installation of the API Connect operator in the next step.

      Troubleshooting: If your DataPower operator pod fails to start, see Troubleshooting upgrade.

5. Upgrade cert-manager to version 1.18.2.
   1. Run the following command to back up the existing certificates and issuers to a file called backup.yaml:

      kubectl get --all-namespaces -oyaml issuer,clusterissuer,cert,secret > backup.yaml

   2. Run the following command to verify the version of cert-manager that you are upgrading:
      • API Connect 10.0.5.4: verify that you are upgrading cert-manager 1.9.1:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.9.1

      • API Connect 10.0.5.5 or 10.0.7.0: verify that you are upgrading cert-manager 1.11.5:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.11.5

      • API Connect 10.0.5.6: verify that you are upgrading cert-manager 1.12.7:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.12.7

      • API Connect 10.0.5.7: verify that you are upgrading cert-manager 1.12.9:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.12.9

      • API Connect 10.0.5.8: no upgrade needed.
      • API Connect 10.0.8.0: verify that you are upgrading cert-manager 1.12.10:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.12.10

      • API Connect 10.0.9.0: verify that you are upgrading cert-manager 1.12.13:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.12.13

      • API Connect 10.0.10.0: verify that you are upgrading cert-manager 1.17.2:

        kubectl get crds certificates.cert-manager.io -o jsonpath='{.metadata.labels.app\.kubernetes\.io\/version}'
        v1.17.2

   3. Run the following command to upgrade cert-manager to version 1.18.2:

      kubectl apply -f helper_files/cert-manager-1.18.2.yaml

6. If you generated your internal certificates with the custom-certs-internal.yaml file when you installed API Connect (following these steps Generating custom certificates using cert-manager), then apply the new custom-certs-internal.yaml file from your target release files and renew db-server-certificate.
   Note: If you are not sure if API Connect was installed with custom-certs-internal.yaml, then check the follow these steps to confirm:

   1. Check whether the management CR has a customCertificates section:

      kubectl -n <management namespace> get mgmt -o yaml | grep customCertificates

      If the management CR does not have this section, then custom-certs-internal.yaml was not used. If it does have this section, then continue to the next check.

   2. Check the issuer of the management-server certificate:

      kubectl -n <management namespace> describe secret management-server | grep issuer-name

      If the command returns:

       cert-manager.io/issuer-name: ingress-issuer

      then custom-certs-internal.yaml was used.
   1. Edit the new custom-certs-internal.yaml file. Replace all instances of <namespace> and <site_name> with the values from the original custom-certs-internal.yaml file that you used during installation.
   2. Apply the new custom-certs-internal.yaml file.

      kubectl apply -f helper_files/custom-certs-internal.yaml

   3. Renew db-server-certificate.

      kubectl -n <management namespace> delete secret db-server-certificate

7. Apply the new API Connect operator YAML into the namespace where the API Connect operator is running.
   • For a single-namespace deployment:
     1. For a single-namespace deployment, update the configuration file by replacing each occurrence of namespace: default with namespace: <namespace>. Replace <namespace> with the actual name of the namespace where you deployed API Connect.
        Note: If you are using Operator Lifecycle Manager (OLM), then skip this step.
     2. Edit the ibm-apiconnect.yaml file and replace each occurrence of REPLACE-DOCKER-REGISTRY with the location of the apiconnect operator images, either uploaded to your own registry or pulled from a public registry. This includes replacing the REPLACE-DOCKER-REGISTRY placeholder in the following locations:
        • image properties for containers.
        • Any other instances of REPLACE-DOCKER-REGISTRY in the file, regardless of whether they are part of an image key.
     3. Run the following command:

        kubectl -n <namespace> apply -f ibm-apiconnect.yaml

   • For a multi-namespace deployment:
     1. Edit the ibm-apiconnect-distributed.yaml file and replace each occurrence of $OPERATOR_NAMESPACE with the namespace for the deployment.
     2. Also in ibm-apiconnect-distributed.yaml, set the values of the containers.image properties. Replace the placeholder values REPLACE-DOCKER-REGISTRY with the docker registry host location of the API Connect operator image (either uploaded to own registry or pulled from public registry).
     3. Install ibm-apiconnect-distributed.yaml with the following command:

        kubectl apply -f ibm-apiconnect-distributed.yaml

8. Verify that the ibm-datapower-operator and the ibm-apiconnect operators are restarted.
9. Ensure that the apiconnect operator re-created the necessary microservices:

   kubectl get apic -n <namespace>

   Do not proceed to upgrade the API Connect subsytems until the operator upgrade is complete.

   Note: If analytics backups are configured, then after the API Connect operator is upgraded, the analytics CR reports a status of Blocked until the analytics subsystem is upgraded to V10.0.8. With top-level CR deployments, the apiconnectcluster CR reports Pending until the analytics CR is no longer Blocked. For more information, see Analytics backup changes.
10. Upgrade the API Connect subsystems:
    1. Upgrading the management subsystem on Kubernetes
    2. Upgrading the gateway subsystem on Kubernetes
    3. Upgrading the portal subsystem on Kubernetes
    4. Upgrading the analytics subsystem on Kubernetes
11. Verify that the upgraded subsystems report as Running and the RECONCILED VERSION displays the new version of API Connect.

    Run the following command:

    kubectl get apic --all-namespaces

    Example response:

    NAME                                                READY   STATUS    VERSION              RECONCILED VERSION      AGE
    analyticscluster.analytics.apiconnect.ibm.com/analytics      8/8     Running   10.0.11.0   10.0.11.0-1074   121m
    
    NAME                                     PHASE     READY   SUMMARY                           VERSION    AGE
    datapowerservice.datapower.ibm.com/gw1   Running   True    StatefulSet replicas ready: 1/1   10.0.11.0   100m
    
    NAME                                     PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
    datapowermonitor.datapower.ibm.com/gw1   Running                false          false              100m
    
    NAME                                            READY   STATUS    VERSION              RECONCILED VERSION      AGE
    gatewaycluster.gateway.apiconnect.ibm.com/gw1   2/2     Running   10.0.11.0   10.0.11.0-1074  100m
    
    NAME                                                 READY   STATUS    VERSION              RECONCILED VERSION      AGE
    managementcluster.management.apiconnect.ibm.com/m1   16/16   Running   10.0.11.0  10.0.11.0-1074   162m
    
    
    NAME                                             READY   STATUS    VERSION              RECONCILED VERSION      AGE
    portalcluster.portal.apiconnect.ibm.com/portal   3/3     Running   10.0.11.0   10.0.11.0-1074   139m

    Important: If you need to restart the deployment, wait until all portal sites complete the upgrade.

    After the portal subsystem upgrade is complete, each portal site is upgraded. You can monitor the site upgrade progress from the MESSAGE column in the oc get ptl output. You can still use the portal while sites are upgrading, although a maintenance page is shown for any sites that are being upgraded. When the site upgrades are complete, the oc get ptl output shows how many sites the portal is serving:

    NAME     READY   STATUS    VERSION          RECONCILED VERSION   MESSAGE              AGE
    portal   3/3     Running   <version>        <version>            Serving 2 sites      22h

    On two data center disaster recovery deployments, the sites are not upgraded until both data centers are upgraded.

12. Optional: If upgrading from 10.0.5.x or 10.0.7.0: Provide a custom endpoint for the Consumer Catalog.

    Skip this step if you are upgrading from 10.0.8.0 or later.

    Beginning with version 10.0.8.0, API Connect includes the Consumer Catalog feature. If your deployment uses cert-manager, then during the upgrade from 10.0.5.x or 10.0.7.0, a new endpoint is added to the management CR for the Custom Catalog.

    The following definition is generated and automatically added to the management CR during the upgrade, using a certificate from cert-manager:

      consumerCatalogEndpoint:
        annotations:
          cert-manager.io/issuer: ingress-issuer
        hosts:
        - name: consumer-catalog.<domain>
          secretName: <name>-consumer-catalog-endpoint

    1. Make the following optional changes to the endpoint definition in the CR:
       • Change the name to a host name of your own choosing.
       • Specify a custom cert-manager certificate.
    2. Run the following command to save and apply the CR: wq
13. Optional: For the optional components API Connect Toolkit and API Connect Local Test Environment, install the latest version of each after you complete the upgrade of the subsystems.