Pre-upgrade preparation and checks on Kubernetes

Prepare your Kubernetes environment and API Connect deployment for upgrade.

This topic covers all the preparation steps you must complete before you can start the upgrade process.

Before completing the steps in this topic, review Planning your API Connect upgrade on Kubernetes to ensure that you are following a supported upgrade path and that you are aware of any API Connect feature changes that might affect your deployment.

Run the pre-upgrade health checks during your upgrade planning stage to identify any problems as early as possible. Run the checks again just before upgrade to confirm API Connect is still ready for upgrade.

Gathering and preparing upgrade files

  1. Obtain the API Connect files from IBM Fix Central.
    From the IBM Fix Central site, download the following files:
    IBM API Connect <version> for Containers
    Docker images for all API Connect subsystems. Filename is apiconnect-image-tool_<version>.
    IBM API Connect <version> Operator Release Files for Containers
    Kubernetes operators and API Connect Custom Resource (CR) templates. Filename is apiconnect-operator-release-files_<version>.
    IBM API Connect <version> Security Signature Bundle File
    Signature files for verifying the integrity of your downloads.

    Filename is signatures_<version>.

    IBM API Connect <version> Toolkit for <operating_system>
    Toolkit command-line utility. Packaged as a stand-alone file, or with API Designer:
    • IBM® API Connect <version> Toolkit for <operating_system>
    • IBM® API Connect <version> Toolkit Designer for <operating_system>
    Filename is toolkit-...<operating system>_<version>.

    Not required during initial installation. After installation, you can download directly from the Cloud Manager UI and API Manager UI as explained in Installing the toolkit.

    IBM API Connect <version> Local Test Environment
    Optional test environment for testing locally as explained in Testing an API with the Local Test Environment.

    Filename is apic-lte-<operating system>_<version>.

  2. Complete the steps in Verifying the integrity of IBM product files to verify that the downloaded product files are not corrupted.
  3. Upload the image files to your docker registry.
    1. Load the image-tool image for the new API Connect version into your local Docker registry:
      docker load < apiconnect-image-tool-<version>.tar.gz 

      Ensure that the registry has sufficient disk space for the files.

    2. If your Docker registry requires repositories to be created before images can be pushed, create the repositories for each of the images that the image tool lists. If your Docker registry does not require creation of repositories, skip to step 3.c.
      1. Run the following command to get a list of the images from image-tool:
        docker run --rm apiconnect-image-tool-<version> version --images
      2. From the output of each entry of the form <image-name>:<image-tag>, use your Docker registry repository create command to create a repository for <image-name>.
        For example, in AWS ECR the command for each <image-name> is:
        aws ecr create-repository --repository-name <image-name>
    3. Upload the image:
      • If you do not need to authenticate with the docker registry, use:
        docker run --rm apiconnect-image-tool-<version> upload <registry-url>
      • If you need to authenticate and your docker registry accepts the username and password arguments, use:
        docker run --rm apiconnect-image-tool-<version> upload <registry-url> --username <username> --password <password>
      • Otherwise, if you need the image-tool to use your local Docker credentials (for example, with IBM Container Registry), first authenticate with your Docker registry, and then upload images with the command:
        docker run --rm -v ~/.docker:/root/.docker --user 0 apiconnect-image-tool-<version> upload <registry-url>

        Review the following installation notes for your environment:

      If necessary, you can populate a remote container registry with repositories. Then you can push the images from the local registry to the remote registry.

  4. Extract the IBM API Connect <version> Operator Release Files for Containers that you downloaded in step 1.

    Create a directory called helper_files and extract the contents of helper_files.zip from the release_files.zip into the helper_files directory.

  5. If upgrading from 10.0.5.x or 10.0.7.0: If you do not use cert-manager, create a certificate for the new consumer catalog, and add it to the management CR.

    Beginning with version 10.0.8.0, API Connect includes the Consume Catalog feature. If you are upgrading to 10.0.8.0 or later from 10.0.5.x or 10.0.7.0 and do not use cert-manager, you must provide a certificate for the Consumer Catalog during the upgrade.

    1. Create a certificate.
    2. Update the management CR and add the following endpoint for the Consumer Catalog with the secret name for the new certificate:
        consumerCatalogEndpoint:
          hosts:
          - name: consumer-catalog.<domain>
            secretName: <custom-management-consumer-catalog-endpoint>

Pre-upgrade health-checks

  1. Check that the API Connect operator and subsystems are all running.
    To verify the health of each subsystem, run the following commands:
    kubectl get ManagementCluster -n <management namespace>
    kubectl get GatewayCluster -n <gateway namespace>
    kubectl get PortalCluster -n <portal namespace>
    kubectl get AnalyticsCluster -n <analyticsnamespace>
    Confirm that all pods are READY, for example:
    kubectl get PortalCluster -n apic
    NAME     READY   STATUS    VERSION    RECONCILED VERSION   AGE
    portal   3/3     Running   version   version-95    57m
  2. If upgrading from 10.0.5.x: Verify that the management subsystem's pgcluster is working correctly:
    1. Get the name of the pgcluster:
      kubectl get pgcluster -n <management namespace>

      The response displays the name of the postgres cluster that is running in the specified namespace.

    2. Check the status of the pgcluster:
      kubectl get pgcluster <pgcluster_name> -n <APIC_namespace> -o yaml | grep status -A 2 | tail -n3
      A successful response looks like the following example, where the state is Initialized:
      status:
          message: Cluster has been initialized
          state: pgcluster Initialized
      Important: If the pgcluster returns any other state, then do not proceed with the upgrade.
      • If backup or restore jobs are running, wait until they complete and then check the status again. Do not proceed with the upgrade until the status is Initialized.
      • If all background jobs are complete but the pgcluster does not return state: pgcluster Initialized, then contact IBM Support for assistance.
  3. Run the pre-upgrade health check:
    Note: If you have a 2DCDR deployment, run this check only on the active management subsystem.
    1. Verify that the apicops utility is installed and that you have the latest version by running the following command:
      apicops --version

      For more information about the apicops utility, see The API Connect operations tool: apicops.

    2. Run the following command to set the KUBECONFIG environment variable.
      export KUBECONFIG=</path/to/kubeconfig>
    3. Run the apicops pre-upgrade-check check for the management, portal, and analytics subsystems:
      • Management subsystem:
        apicops system:pre-upgrade-check management -n <namespace>

        Before upgrading, run the following command to check for encryption issues and invalid gateway extensions:

        For a full check (requires admin password to verify topology):
        apicops preupgrade -n <namespace>
        To skip topology verification (no admin password needed):
        apicops preupgrade --no-topology -n <namespace>
      • Portal subsystem:
        apicops system:pre-upgrade-check portal -n <namespace>
      • Analytics subsystem:
        apicops system:pre-upgrade-check analytics -n <namespace>

      If your API Connect deployment is working correctly, the output shows no error messages.

      If the pre-upgrade check returns any errors, then collect the output and open a support case.

Additional pre-upgrade checks and operations

Extra checks and operations to review and complete before you start the upgrade.

  1. Upgrades from V10.0.5.x: If the analytics persistent queue is not enabled, verify that you have an extra Physical Volume available.

    For more information about this requirement, see Analytics persistent queue enablement requires extra PV.

  2. If analytics database backups are configured, or you want to enable them in future, then ensure that you have an extra Physical Volume available for the local backup storage. The default size is 150Gi. You can override the size during the upgrade procedure.

  3. If you used any microservice image overrides in any of your API Connect subsystem CRs, remove the image overrides before upgrade.

Backup your API Connect deployment

If upgrade fails, rollback is not possible. Before upgrade complete the disaster recovery preparation for all your subsystems.

For instructions on backing up and disaster recovery preparation, see the following topics: