Upgrading to the latest release on VMware

You can upgrade API Connect on VMware.

Before you begin

Before starting an upgrade, review the supported upgrade paths from prior versions: Upgrade considerations on VMware.
Enable keep-alive in your SSH configuration to avoid problems during the upgrade. For information, consult the documentation for your version of SSH.
The Gateway subsystem remains available during the upgrade of the Management, Portal, and Analytics subsystems.

About this task

You can upgrade the Management subsystem, Developer Portal subsystem, and Analytics subsystem.

For the optional components API Connect Toolkit and API Connect Local Test Environment, you do not need to upgrade. For these components, install the new version of each after you upgrade the subsystems.

Important notes:

When installing the new files, be sure to use the new apicup installer.
The upgrade order for subsystems is important. Upgrade the subsystems in the following order: (1) Management, (2) Portal, (3) Analytics, (4) Gateway. Management must be upgraded first. Gateway must be upgraded after Management because upgrading the Management service before the Gateway ensures that any new policies and capabilities will be available to a previously registered Gateway service.
When you run the install, the program sends the compressed tar file, which contains the upgrade images, to all cluster members. The compressed tar file is about 2 GB, and transfer can take some time. When the install command exits, the compressed tar file has arrived at each member. The upgrade process is now underway, and might continue for some time depending on factors such as the size of your deployment, your network speed, etc.
The apicup subsys install command automatically runs apicup health-check prior to attempting the upgrade. An error is displayed if a problem is found that will prevent successful upgrade.
Version 10.0.4.0 and later: When troubleshooting a problem with an upgrade, you can optionally suppress the health check. See Skipping health check when re-running upgrade.
Version 10.0.4.0 and later - Known issue:The certificate manager was upgraded in API Connect version 10.0.4.0. If you are upgrading from a version earlier than 10.0.4.0, then after you trigger the upgrade from the apim, lur, and taskmanager pods will be in a CrashLoopBackoff state due to Postgres requiring a certificate refresh. Correct the problem by completing the following steps:
1. SSH into the server:
  1. Run the following command to connect as the API Connect administrator, replacing ip_address with the appropriate IP address:
```
ssh ip_address -l apicadm
```
  2. When prompted, select Yes to continue connecting.
  3. When you are connected, run the following command to receive the necessary permissions for working directly on the appliance:
```
sudo -i
```
2. Run the following command to delete the Postgres pods, which refreshes the new certificate:
```
kubectl get pod --no-headers=true | grep postgres | grep -v backup | awk '{print $1}' | xargs kubectl delete pod
```

Important: Upon successful completion of each subsystem upgrade, restart as required:

For each subsystem, the upgrade process automatically runs apicup health-check upon completion of the upgrade. When apicup health-check gives a message that reboot is required for a server, SSH onto that server and check the subsystem custom resource status:
```
kubectl get <name_of_cr>
```
For example:
```
kubectl get managementCluster
```
If both READY is True and the RECONCILED VERSION shows the version you are upgrading to, then reboot that server.
During the upgrade, repeat this action for each subsystem when the message from apicup subsys health-check that says reboot is required for a server. However, if the health check command states that a reboot is required for the Portal subsystem, you must ensure that all of the Portal sites have been upgraded first before performing the reboot. See the instructions in Step 9 for details.

Procedure

Complete the prerequisites:
1. Ensure that your deployment meets the upgrade requirements. See Upgrade considerations on VMware.
2. Verify the API Connect deployment is healthy and fully operational. See Checking cluster health on VMware
3. Remove any stale upgrade files:
```
apic clean-upgrade-files
```
4. Verify sufficient free disk space.
  For each appliance node that you are planning to upgrade:
  1. SSH into the appliance, and switch to user root.
  2. Check disk space in /data/secure:
```
df -h /data/secure
```
    Make sure the disk usage shown is below 70%. If it is not, add disk capacity. See Adding disk space to a VMware appliance.
  3. Check free disk space in /:
```
df -h /
```
    Make sure usage is below 70%. If it is not, consider deleting or offloading older /var/log/syslog* files.
5. Complete a manual backup of the API Connect subsystems. See the instructions for your scenario: .
  - Upgrade from 10.0.1.2-eus and all of its ifixes: Backing up and restoring (10.0.1.1-eus or later)
  - Upgrade from 10.0.2.0-eus or later: Backing up and restoring on VMware
  Note:
  - Do not start an upgrade if a backup is scheduled to run within a few hours.
  - Do not perform maintenance tasks such as rotating key-certificates, restoring from a backup, or starting a new backup, at any time while the upgrade process is running.
Run the pre-upgrade health check:
1. Run the health check on the Management subsystem:
  1. SSH into the server:
    1. Run the following command to connect as the API Connect administrator, replacing ip_address with the appropriate IP address:
      ssh ip_address -l apicadm
    2. When prompted, select Yes to continue connecting.
    3. When you are connected, run the following command to receive the necessary permissions for working directly on the appliance:
      sudo -i
  2. Verify that the apicops utility is installed.
  3. Run the version check and verify that there are no errors:
```
apicops version:pre-upgrade
```
  4. Run the appliance health check and verify that there are no errors:
```
apicops appliance-checks:appliance-pre-upgrade
```
2. Run the health check on the Portal subsystem:
  1. SSH into the server.
  2. Verify that the apicops utility is installed.
  3. Run the following command to check the system status, and verify that there are no errors:
```
apicops upgrade:check-subsystem-status
```
  4. Run the appliance health check and verify that there are no errors:
```
apicops appliance-checks:appliance-pre-upgrade
```
3. Run the health check on the Analytics subsystem:
  1. SSH into the server.
  2. Verify that the apicops utility is installed.
  3. Run the following command to check the system status, and verify that there are no errors:
```
apicops upgrade:check-subsystem-status
```
  4. Run the appliance health check and verify that there are no errors:
```
apicops appliance-checks:appliance-pre-upgrade
```
Optional: Take a Virtual Machine (VM) snapshot of all your VMs; see Using VM snapshots for infrastructure backup and disaster recovery for details. This action does require a brief outage while all of the VMs in the subsystem cluster are shut down - do not take snapshots of running VMs, as they might not restore successfully. VM snapshots can offer a faster recovery when compared to redeploying OVAs and restoring from normal backups.

Important: VM snapshots are not an alternative to the standard API Connect backups that are described in the previous steps. You must complete the API Connect backups in order to use the API Connect restore feature.
Obtain the API Connect files:

You can access the latest files from IBM Fix Central by searching for the API Connect product and your installed version.

The following files are used during upgrade on VMware:

IBM® API Connect <version> Management Upgrade File for VMware

Management subsystem files for upgrade

IBM® API Connect <version> Developer Portal Upgrade File for VMware

Developer Portal subsystem files for upgrade

IBM® API Connect <version> Analytics Upgrade File for VMware

Analytics subsystem files for upgrade

IBM® API Connect <version> Install Assist for <operating_system_type>

The apicup installation utility. Required for all installations on VMware.
The following files are not used during upgrade, but can be installed as new installations used to replace prior versions
IBM® API Connect <version> Toolkit for <operating_system_type>
Toolkit command line utility. Packaged standalone, or with API Designer or Loopback:
- IBM® API Connect <version> Toolkit for <operating_system_type>
- IBM® API Connect <version> Toolkit with Loopback for <operating_system_type>
- IBM® API Connect <version> Toolkit Designer with Loopback for <operating_system_type>
Not required during initial installation. After installation, you can download directly from the Cloud Manager UI and API Manager UI. See Installing the toolkit.
IBM® API Connect <version> Local Test Environment

Optional test environment. See Testing an API with the Local Test Environment

IBM® API Connect <version> Security Signature Bundle File

Checksum files that you can use to verify the integrity of your downloads.
Verify that the subsystem upgrade files are not corrupted.
1. Run the following command separately for the Management, the Developer Portal, and the Analytics upgrade files:
  - Mac or Linux:
```
sha256sum <upgrade-file-name>
```
    Example:
```
sha256sum upgrade_management_v10.0.4.0-ifix3
```
  - Windows:
```
C:\> certUtil -hashfile C:\<upgrade-file-name> SHA256
```
    Example:
```
C:\> certUtil -hashfile C:\upgrade_management_v10.0.4.0-ifix2 SHA256
```
2. Compare the result with the checksum values to verify that the files are not corrupted.
  If the checksum does not match the value for the corresponding version and subsystem, then the upgrade file is corrupted. Delete it and download a new copy of the upgrade file. The following list shows the checksum values for the current release. If you are upgrading to an older release, see Checksum values for earlier releases.
  
  Checksum values for 10.0.4.0-ifix3:
```
v10.0.4.0-ifix3 analytics : db742d6c01a8d2f147c7c455a93395d92cc97255b847f5699fff07cd9808fd19
v10.0.4.0-ifix3 management : 8f23cd4f295362ec64ecfc29d0a7c9d67f1f09d3893b6707b0b3075c88f54a6a
v10.0.4.0-ifix3 portal : 7fbe529e5c1f3fb17e4b719d421a02b50d9052e3b9afeb66450bdb8a773c2b48
```

If necessary, download from the same Fix Pack page any Control Plane files that are needed.

Control Plane files provide support for specific Kubernetes versions. The IBM® API Connect <version> Management Upgrade File for VMware file contains the latest Control Plane file. An upgrade from the most recent API Connect version to the current version does not need a separate Control Plane file. However, when upgrading from older versions of API Connect, you must install one or more control plane files to ensure that all current Kubernetes versions are supported.

Consult the following table to see if your deployment needs one or more separate Control Plane files.

Table 1. Control Plane files needed for upgrade
Version to upgrade from	Instructions for upgrading to 10.0.4.0-ifix3
10.0.4.0-ifix2 10.0.4.0-ifix1 10.0.4.0	No control plane file is needed.
10.0.3.0-ifix1 10.0.3.0	No control plane file is needed.
10.0.2.0-ifix2 10.0.2.0-ifix1 10.0.2.0	Download `appliance-control-plane-1.21.x.tgz`
10.0.1.5-ifix4-eus 10.0.1.5-ifix3-eus 10.0.1.5-ifix2-eus 10.0.1.5-ifix1-eus 10.0.1.5-eus 10.0.1.4-eus 10.0.1.4-ifix1-eus	No control plane file is needed.
10.0.1.2-ifix2-eus 10.0.1.2-ifix1-eus 10.0.1.2-eus	Download `appliance-control-plane-1.21.x.tgz`

Install the installation utility.
1. Locate the apicup installation utility file for your operating system, and place it in your project directory
2. Rename the file for your OS type to apicup. Note that the instructions in this documentation refer to apicup.
3. OSX and Linux® only: Make the apicup file an executable file by entering chmod +x apicup.
4. Set your path to the location of your apicup file.
  OSX and Linux
  export PATH=$PATH:/Users/your_path/
  
  Windows
  set PATH=c:\your_path;%PATH%
5. From within your project directory, specify the API Connect license:
```
apicup licenses accept <License_ID>
```
  The <License_ID> is specific to the API Connect Program Name you purchased. To view the supported license IDs, see API Connect licenses.
Upgrade the management subsystem:
1. Perform the upgrade on the management subsystem:
  1. Install the management subsystem files.
```
apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive>
```
  2. Verify that the upgrade was successful.
```
apicup subsys health-check <subsys_name>
```
    If apicup returns an error, see: Postgres pods fail to start after upgrade
  3. Reboot the subsystem when the health check command displays a message instructing you to do so. See Reboot steps.
  4. After reboot for the management cluster, if a stancluster pod does not return to 1/1 Running status, see Workaround for non-ready stancluster pods after upgrade.
2. Use apicops to validate the certificates.
  For information on the apicops tool, see The API Connect operations tool: apicops.
  1. Run the following command:
```
apicops upgrade:stale-certs -n <namespace>
```
  2. Delete any stale certificates that are managed by cert-manager.
    If a certificate failed the validation and it is managed by cert-manager, you can delete the stale certificate secret, and let cert-manager regenerate it. Run the following command:
    kubectl delete secret <stale-secret> -n <namespace>
  3. Restart the corresponding so that it can pick up the new secret.
    To determine which pod to restart, see the following topics:
    
    List of external certificates
    
    List of internal certificates
Upgrade the Portal subsystem.
1. Install the Portal subsystem files:
```
apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive>
```
2. Verify that the upgrade was successful.
```
apicup subsys health-check <subsys_name>
```
  Important: If the health check command states that a reboot is required, you must ensure that all Portal sites have been upgraded first before performing the reboot. To check that the Portal sites have upgraded, inspect the platform and state values for each site. The sites should all be in INSTALLED state, and have platform equal to the new platform that you can identify with platforms:list. Note that there will be two platforms present in the list while the site upgrades are happening. There should be only one platform after they have completed. Use the commands in Step 9.c to list the sites, and then reboot.
  
  If apicup returns an error, see Upgrading a 3 node profile to IBM API Connect 10.0.3.0 or later might result in some portal-db/www pods being stuck in the Pending state.
3. Ensure that the Portal sites were upgraded:
  1. Use the toolkit apic to obtain the portal service id and endpoint:
```
apic portal-services:get -o admin -s <management_server_endpoint> \
             --availability-zone availability-zone-default <portal-service-name> \
             --output - --format json
```
  2. List the sites:
```
apic --mode portaladmin sites:list -s <management_server_endpoint> \
              --portal_service_id <portal_service_id_from_above_command> \
              --portal_service_endpoint <portal_service_endpoint_from_above_command> \ 
              --format json
```
    Any sites currently upgrading will be listed as UPGRADING. Once all sites have finished upgrading they should have the INSTALLED status and the new platform version listed.
    
    See also: apic sites:list and Using the sites commands.
  3. After all sites are in INSTALLED state and have the new platform listed, run:
```
apic --mode portaladmin platforms:list -s <management_server_endpoint> \
              --portal_service_id <portal_service_id_from_above_command> \
              --portal_service_endpoint <portal_service_endpoint_from_above_command> \ 
              --format json
```
    The new version of the platform should be the only platform listed.
    
    See also: apic platforms:list and Using the platforms commands.
4. Reboot the subsystem when the health check command displays a message instructing you to do so. See Reboot steps.
Upgrade the Analytics subsystem
1. Install the Analytics subsystem files:
```
apicup subsys install <subsystem_name> <subsystem_upgrade_tar_archive>
```
2. Verify that the upgrade was successful.
```
apicup subsys health-check <subsys_name>
```
3. Reboot the subsystem when the health check command displays a message instructing you to do so. See Reboot steps.
Complete a manual backup of the API Connect subsystems. See Backing up and restoring on VMware.
Optional: Take a Virtual Machine (VM) snapshot of all your VMs; see Using VM snapshots for infrastructure backup and disaster recovery for details. This action requires a brief outage while all of the VMs in the subsystem cluster are shut down - do not take snapshots of running VMs, as they might not restore successfully.
Complete disaster preparation steps, to ensure recovery of API Connect from a disaster event. See Preparing for a disaster.

What to do next

When you have upgraded all of the API Connect subsystems, upgrade your DataPower Gateway Service. See Upgrading DataPower Gateway Service.

OSX and Linux	`export PATH=$PATH:/Users/your_path/`
Windows	`set PATH=c:\your_path;%PATH%`