Troubleshooting upgrades on VMware

Review the following troubleshooting guidance if you encounter a problem while upgrading API Connect on VMware.

Ubuntu preupgrade failures during OVA upgrade from 10.0.8.0 or 10.0.5.8 to 10.0.8.1

Symptom

For the Ubuntu upgrade, API Connect uses a proxy service that responds with the required HTTP responses or packages for the upgrade. If this service is not operational, the upgrade will not continue and API Connect displays the following error:

Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
  Could not connect to localhost:8080 (127.0.0.1). - connect (111: Connection refused)
Err:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Unable to connect to localhost:8080:
Err:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Unable to connect to localhost:8080:
Err:4 http://security.ubuntu.com/ubuntu focal-security InRelease
  Could not connect to localhost:8080 (127.0.0.1). - connect (111: Connection refused)

Cause

The Ubuntu preupgrade fails to upgrade to 10.0.8.1 on Open Virtualization Archive (OVA) airgapped environments due to include failed attempts to connect to the internet during the do-release-upgrade process, indicating that the proxy service is not functioning correctly.

Resolution

To resolve the issue, complete the following steps:

Run the following commands:
1. If you need to interrupt the apt update command, press Ctrl+Cto break out of the operation
2. ```
sudo -i
```
3. ```
cd /usr/local/lib/record
```
4. ```
./proxy &
```

Create a file called fix.sh, and add the following content:

#!/bin/bash
PACKAGES=("libpq5" "shim-signed" "shim" "iotop" "sysstat" "dstat" "ioping" "atop" )
PREFERENCES_DIR="/etc/apt/preferences.d"
# Loop through the package names
for PACKAGE_NAME in "${PACKAGES[@]}"; do
 # Get the installed version of the package
 PACKAGE_VERSION=$(dpkg-query -W -f='${Version}' $PACKAGE_NAME 2>/dev/null)
 
 if [ -z "$PACKAGE_VERSION" ]; then
 echo "Package '$PACKAGE_NAME' is not installed. Skipping."
 continue
 fi
 PREFERENCES_FILE="${PREFERENCES_DIR}/${PACKAGE_NAME}-preferences"
 # Write the pinning configuration to the file
 echo "Creating pin for package '$PACKAGE_NAME' version '$PACKAGE_VERSION'"
 echo "Package: $PACKAGE_NAME" | sudo tee $PREFERENCES_FILE > /dev/null
 echo "Pin: version $PACKAGE_VERSION" | sudo tee -a $PREFERENCES_FILE > /dev/null
 echo "Pin-Priority: 1001" | sudo tee -a $PREFERENCES_FILE > /dev/null
 echo "Package '$PACKAGE_NAME' is now pinned to version $PACKAGE_VERSION."
done

Run the following script:
```
bash ./fix.sh
```
Repeat the steps i-v under Steps 2.c.
Run the following command:
```
sudo apic lock
```
Reboot the VM:
```
sudo reboot
```

Run the following commands:

```
sudo -i
```
```
systemctl stop record-proxy
```
```
cd /usr/local/lib/record
```
```
/proxy &
```
```
sudo do-release-upgrade
```

Verify that the Ubuntu upgrade is complete:
```
sudo lsb_release -a
```

The output should be:

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

Subsystem status reports `PreUpgrade check failed`

If you run:

apicup subsys status <subsystem name>

apicup subsys health-check <subsystem name>

and you see the message:

Preupgrade check failed, retrying. See status.PreUpgradeCheck for more details.

then gather logs and open a support case.

Upgrade from V10.0.7 fails with an automatic rollback

If the upgrade fails and the subsystem is in the SUBSYSTEM_ROLLBACK state, do not proceed with the upgrade. Open a support case and work with IBM Support to resolve the underlying problem before you attempt the upgrade again.

Contact IBM Support to resolve the problem that caused the upgrade to fail.
When the problem is resolved, run the upgrade again with the following command:
```
apicup subsys install <subsys> --skip-health-check --rollback-mode-update
```

Some containers in the `kube-system` namespace show a status of `ErrImageNeverPull`

When upgrading, some of the containers in the kube-system namespace might show a status ofErrImageNeverPull. This happens due to Docker not successfully loading all images from the upgrade .tgz file. To resolve this issue and enable the upgrade to proceed, complete the following steps:

Run the following command to determine which node is missing the control plane files:
```
kubectl -n kube-system get pods -owide
```
This command returns the names of the nodes containing pods with the ErrImageNeverPull status, which indicates missing control plane files.
On the node that is missing the control plane files, run the following command to determine which versions of the control plane are missing:
```
cat /var/lib/apiconnect/appliance-control-plane-current
```
For each missing control plane, run the following command on the same node to add it, replacing <version> with the version of the control plane:
```
ctr -k8s.io images import /usr/local/lib/appliance-control-plane/<version>/kubernetes.tar --all-platforms
```
If you have a kubernetes.tgz file instead of a kubernetes.tar file, the run this first:
```
gunzip /usr/local/lib/appliance-control-plane/<version>/kubernetes.tgz
```

Skipping health check when re-running upgrade

The apicup subsys install command automatically runs apicup health-check prior to attempting the upgrade. An error is displayed if a problem is found that will prevent successful upgrade

In some scenarios, if you encounter an upgrade failure, an attempt to rerun apicup subsys install is blocked by errors found by apicup health-check. Even when you have fixed the error (such as reconfiguration of an incorrect upgrade CR), the failed upgrade can continue to cause the health check to fail.

You can workaround the problem by adding the --skip-health-check flag to suppress the health check:

apicup subsys install <subsystem_name> --skip-health-check

In this case, use of --skip-health-check allows the upgrade to rerun successfully.

Unexpected behavior in Cloud Manager and API Manager UIs after upgrade

Stale browser cache issues can cause problems after an upgrade. To remedy this problem, clear your browser cache, and open a new browser window.

Note: In the The Help icon.

Help page of the Cloud Manager, API Manager, and API Designer user interfaces, there's a Product information tile that you can click to find out information about your product versions, as well as Git information about the package versions being used. Note that the API Designer product information is based on its associated management server, but the Git information is based on where it was downloaded from.