Pre-upgrade preparation and checks on Kubernetes

Prepare your Kubernetes environment and API Connect deployment for upgrade.

This topic covers all the preparation steps you must complete before you can start the upgrade process.

Before completing the steps in this topic, review Planning your API Connect upgrade on Kubernetes to ensure that you are following a supported upgrade path and that you are aware of any API Connect feature changes that might affect your deployment.

Run the pre-upgrade health checks during your upgrade planning stage to identify any problems as early as possible. Run the checks again just before upgrade to confirm API Connect is still ready for upgrade.

Gathering and preparing upgrade files

Obtain the API Connect files from IBM Fix Central.
From the IBM Fix Central site, download the following files:
IBM API Connect <version> for Containers

Docker images for all API Connect subsystems. Filename is apiconnect-image-tool_<version>.

IBM API Connect <version> Operator Release Files for Containers

Kubernetes operators and API Connect Custom Resource (CR) templates. Filename is apiconnect-operator-release-files_<version>.

IBM API Connect <version> Security Signature Bundle File

Signature files for verifying the integrity of your downloads.
Filename is signatures_<version>.

IBM API Connect <version> Toolkit for <operating_system>
Toolkit command-line utility. Packaged as a stand-alone file, or with API Designer:
- IBM® API Connect <version> Toolkit for <operating_system>
- IBM® API Connect <version> Toolkit Designer for <operating_system>
Filename is toolkit-...<operating system>_<version>.
Not required during initial installation. After installation, you can download directly from the Cloud Manager UI and API Manager UI as explained in Installing the toolkit.
IBM API Connect <version> Local Test Environment

Optional test environment for testing locally as explained in Testing an API with the Local Test Environment.
Filename is apic-lte-<operating system>_<version>.
Complete the steps in Verifying the integrity of IBM product files to verify that the downloaded product files are not corrupted.
Upload the image files to your docker registry.
1. Load the image-tool image for the new API Connect version into your local Docker registry:
```
docker load < apiconnect-image-tool-<version>.tar.gz 
```
  Ensure that the registry has sufficient disk space for the files.
2. If your Docker registry requires repositories to be created before images can be pushed, create the repositories for each of the images that the image tool lists. If your Docker registry does not require creation of repositories, skip to step 3.c.
  1. Run the following command to get a list of the images from image-tool:
```
docker run --rm apiconnect-image-tool-<version> version --images
```
  2. From the output of each entry of the form <image-name>:<image-tag>, use your Docker registry repository create command to create a repository for <image-name>.
    For example, in AWS ECR the command for each <image-name> is:
```
aws ecr create-repository --repository-name <image-name>
```
3. Upload the image:
  - If you do not need to authenticate with the docker registry, use:
```
docker run --rm apiconnect-image-tool-<version> upload <registry-url>
```
  - If you need to authenticate and your docker registry accepts the username and password arguments, use:
```
docker run --rm apiconnect-image-tool-<version> upload <registry-url> --username <username> --password <password>
```
  - Otherwise, if you need the image-tool to use your local Docker credentials (for example, with IBM Container Registry), first authenticate with your Docker registry, and then upload images with the command:
```
docker run --rm -v ~/.docker:/root/.docker --user 0 apiconnect-image-tool-<version> upload <registry-url>
```
    Review the following installation notes for your environment:
  If necessary, you can populate a remote container registry with repositories. Then you can push the images from the local registry to the remote registry.
Extract the IBM API Connect <version> Operator Release Files for Containers that you downloaded in step 1.
Create a directory called helper_files and extract the contents of helper_files.zip from the release_files.zip into the helper_files directory.

Pre-upgrade health-checks

Check that the API Connect operator and subsystems are all running.

To verify the health of each subsystem, run the following commands:

kubectl get ManagementCluster -n <management namespace>
kubectl get GatewayCluster -n <gateway namespace>
kubectl get PortalCluster -n <portal namespace>
kubectl get AnalyticsCluster -n <analyticsnamespace>

Confirm that all pods are READY, for example:

kubectl get PortalCluster -n apic
NAME     READY   STATUS    VERSION    RECONCILED VERSION   AGE
portal   3/3     Running   version   version-95    57m

Run the pre-upgrade health check:
Note: If you have a 2DCDR deployment, run this check only on the active management subsystem.
1. Verify that the apicops utility is installed and that you have the latest version by running the following command:
```
apicops --version
```
  For more information about the apicops utility, see The API Connect operations tool: apicops.
2. Run the following command to set the KUBECONFIG environment variable.
```
export KUBECONFIG=</path/to/kubeconfig>
```
3. Run the apicops pre-upgrade-check check for the management, portal, and analytics subsystems:
  - Management subsystem:
```
apicops system:pre-upgrade-check management -n <namespace>
```
  - Portal subsystem:
```
apicops system:pre-upgrade-check portal -n <namespace>
```
  - Analytics subsystem:
```
apicops system:pre-upgrade-check analytics -n <namespace>
```
  If your API Connect deployment is working correctly, the output shows no error messages.
  
  If the pre-upgrade check returns any errors, then collect the output and open a support case.

Additional pre-upgrade checks and operations

Extra checks and operations to review and complete before you start the upgrade.

If analytics database backups are configured, or you want to enable them in future, then ensure that you have an extra Physical Volume available for the local backup storage. The default size is 150Gi. You can override the size during the upgrade procedure.
If you used any microservice image overrides in any of your API Connect subsystem CRs, remove the image overrides before upgrade.

Backup your API Connect deployment

If upgrade fails, rollback is not possible. Before upgrade complete the disaster recovery preparation for all your subsystems.

For instructions on backing up and disaster recovery preparation for v10.0.9.0, see Backing up, restoring, and disaster recovery