Troubleshooting upgrades on VMware

Review the following troubleshooting guidance if you encounter a problem while upgrading API Connect on VMware.

For additional issues that you might encounter when upgrading to v10.0.6.0 only, see Troubleshooting upgrades to v10.0.6.0 on VMware.

Subsystem is stuck in `Pending` state with a reason of `PreUpgradeCheckInProgress`

Before the subsystem microservices are upgraded, the operator triggers a set of pre-upgrade checks that must pass for the upgrade to proceed. If one or more of the checks fail, the subsystem status remains in Pending state with a reason of PreUpgradeCheckInProgress. Check the status condition of the subsystem CR to confirm the pre-upgrade check failed. The status.PreUpgradeCheck property contains a summary of the failed checks. Full logs for the checks that are carried out can be viewed in the ConfigMap referenced in the status.PreUpgradeCheck property. The pre-upgrade checks automatically retry until they successfully pass. If you are unable to rectify the problem that causes a check to fail, then open an IBM support case.

False positive result from health check after upgrading to 10.0.7.0

Sometimes an upgrade to API Connect 10.0.7.0 appears to complete successfully and the health-check reports that the upgraded subsystem is healthy, but the older version is still installed. A false positive response from the health check indicates a problem with the upgrade.

Note: This step only applies if you upgraded to API Connect 10.0.7.0 and is due to a known issue that will be corrected in a later release.

Complete the following steps to resolve the issue:

If you did not do so already, run the follow command on each subsystem to determine what version of API Connect is deployed:
```
kubectl get apic
```
If the new version of API Connect is deployed on a subsystem, then the upgrade was successful for that subsystem.
If the older version of API Connect is still deployed on a subsystem, run the following command to check the status of that subsystem in case there is a message about the underlying problem:
```
apic status
```
Sometimes the status indicates the problem and you can correct it before upgrading that subsystem again.
If you do not see any messages indicating the source of the problem, contact IBM Support for assistance.

V10.0.7 only: Upgrade issues a warning for management subsystem to use S3 backups

If you previously used SFTP for backups, you must configure S3 backups before upgrading to v10.0.7.0. Configuring the S3 backups requires that you use the older version of the apicup tool (from your current release of API Connect) to both set the values and run the apicup install command.

If you used the wrong version of apicup to configure S3 backup settings before upgrading, you might see a warning for the Management subsystem with the following message: Please set S3 backup configs.

Correct the problem by running the following command run the upgrade again but skip the health-check and allow the CR to be updated with the new S3 settings:

./apicup subsys install <mgt-subsys-name> --skip-health-check

Some containers in the `kube-system` namespace show a status of `ErrImageNeverPull`

When upgrading, some of the containers in the kube-system namespace might show a status ofErrImageNeverPull. This happens due to Docker not successfully loading all images from the upgrade .tgz file. To resolve this issue and enable the upgrade to proceed, complete the following steps:

Run the following command to determine which node is missing the control plane files:
```
kubectl -n kube-system get pods -owide
```
This command returns the names of the nodes containing pods with the ErrImageNeverPull status, which indicates missing control plane files.
On the node that is missing the control plane files, run the following command to determine which versions of the control plane are missing:
```
cat /var/lib/apiconnect/appliance-control-plane-current
```
V10.0.7: For each missing control plane, run the following command on the same node to add it, replacing <version> with the version of the control plane:
```
ctr -k8s.io images import /usr/local/lib/appliance-control-plane/<version>/kubernetes.tar --all-platforms
```
If you have a kubernetes.tgz file instead of a kubernetes.tar file, the run this first:
```
gunzip /usr/local/lib/appliance-control-plane/<version>/kubernetes.tgz
```
V10.0.6: For each missing control plane, run the following command on the same node to add it, replacing <version> with the version of the control plane:
```
docker load < /usr/local/lib/appliance-control-plane/<version>/kubernetes.tgz
```

Skipping health check when re-running upgrade

The apicup subsys install command automatically runs apicup health-check prior to attempting the upgrade. An error is displayed if a problem is found that will prevent successful upgrade

In some scenarios, if you encounter an upgrade failure, an attempt to rerun apicup subsys install is blocked by errors found by apicup health-check. Even when you have fixed the error (such as reconfiguration of an incorrect upgrade CR), the failed upgrade can continue to cause the health check to fail.

You can workaround the problem by adding the --skip-health-check flag to suppress the health check:

apicup subsys install <subsystem_name> --skip-health-check

In this case, use of --skip-health-check allows the upgrade to rerun successfully.

Unexpected behavior in Cloud Manager and API Manager UIs after upgrade

Stale browser cache issues can cause problems after an upgrade. To remedy this problem, clear your browser cache, and open a new browser window.

Note: In the The Help icon.

Help page of the Cloud Manager, API Manager, and API Designer user interfaces, there's a Product information tile that you can click to find out information about your product versions, and Git information about the package versions that are used. The API Designer product information is based on its associated management server, but the Git information is based on where it was downloaded from.