Troubleshooting upgrades on VMware

Review the following troubleshooting guidance if you encounter a problem while upgrading API Connect on VMware.

Note: In the The Help icon.

Help page of the Cloud Manager, API Manager, and API Designer user interfaces, there's a Product information tile that you can click to find out information about your product versions, as well as Git information about the package versions being used. Note that the API Designer product information is based on its associated management server, but the Git information is based on where it was downloaded from.

Topics:

Postgres pods fail to start after upgrade
Upgrading a 3 node profile to IBM API Connect 10.0.3.0 or later might result in some portal-db/www pods being stuck in the Pending state
Workaround for non-ready stancluster pods after upgrade
Issues when installing Drupal 8 based custom modules or sub-themes into the Drupal 9 based Developer Portal

Postgres pods fail to start after upgrade

Note: This topic applies only to upgrading the management subsystem from v10.0.2.0 or later to a newer version of v10.0.x. This problem does not occur when upgrading from v10.0.1.2-ifix2 to v10.0.3.0 or later.

When upgrading the management subsystem from v10.0.2.0 or later, as part of Upgrading to the latest release on VMware, you might encounter an error message when checking the subsystem health upon completion of the upgrade. For example:

apic health-check
INFO[0000] Log level: info
FATA[0006] Cluster not in good health:
ManagementCluster (current ha mode: active) is not ready | State: 15/16 Phase: Pending

To troubleshoot when a message like this occurs:

Check the state of postgres pods:

kubectl get pods | grep postgres

For example:

root@apimdev1146:~# kubectl get pods | grep postgres
fxpk-management-fd8b0b1f-postgres-577594c7f-k54pk                 0/1     Init:CrashLoopBackOff   17         22h
fxpk-management-fd8b0b1f-postgres-backrest-shared-repo-7fctp88w   1/1     Running                 2          22h 
fxpk-management-fd8b0b1f-postgres-elbx-698f445649-rlc2g           0/1     Init:CrashLoopBackOff   16         22h
fxpk-management-fd8b0b1f-postgres-pgbouncer-64f57b7cc7-52bk8      1/1     Running                 2          22h
fxpk-management-fd8b0b1f-postgres-pgbouncer-64f57b7cc7-jjjvb      1/1     Running                 1          22h
fxpk-management-fd8b0b1f-postgres-pgbouncer-64f57b7cc7-qp4zh      1/1     Running                 2          22h
fxpk-management-fd8b0b1f-postgres-stanza-create-6pt6c             0/1     Completed               0          22h
fxpk-management-fd8b0b1f-postgres-ubba-79ccdd5cc6-kj4zx           0/1     Init:CrashLoopBackOff   17         22h
postgres-operator-85fb96db4b-gk8k8                                4/4     Running                 8          22h

If any pods show Init:CrashLoopBackOff status, restart the pods. To force a restart, delete the pods:

kubectl delete pod <name_of_postgres_pod>

For example:

kubectl delete pod fxpk-management-fd8b0b1f-postgres-577594c7f-k54pk
kubectl delete pod fxpk-management-fd8b0b1f-postgres-elbx-698f445649-rlc2g
kubectl delete pod fxpk-management-fd8b0b1f-postgres-ubba-79ccdd5cc6-kj4zx

When pods are deleted, the deployment automatically restarts them.

Re-run the health check. For example:

apicup subsys health-check <subsys_name>

When health check is successful, return to the next upgrade step in Upgrading to the latest release on VMware.

Upgrading a 3 node profile to IBM API Connect 10.0.3.0 or later might result in some `portal-db/www` pods being stuck in the `Pending` state

IBM® API Connect 10.0.3.0 introduces the pod anti-affinity required rule, meaning that in a 3 node profile deployment, all 3 db and www pods can run only if there are at least 3 running worker nodes. This rule can cause some upgrades to version 10.0.3.0 or later to become stuck in the Pending state, in which case some extra steps are needed during the upgrade to workaround the issue. See the following example for detailed information about the issue, and how to continue with the upgrade.

Important: You must have a backup of your current deployment before starting the upgrade.

To run the commands that are needed to workaround this issue, you must log in to the virtual machine portal subsystem by using an SSH tool.

Run the following command to log in as apicadm, which is the API Connect ID that has administrator privileges:
```
ssh portal_ip_address -l apicadm
```
Where portal_ip_address is the IP address of the portal subsystem.
Then get a root shell by running the following command:
```
sudo -i
```

In the following example, the VMware stack has 3 VMs (worker nodes):

$ kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
apimdev0103   Ready    worker   42m   v1.20.0
apimdev0129   Ready    worker   45m   v1.20.0
apimdev1066   Ready    worker   39m   v1.20.0

The pods have been scheduled across only 2 of the 3 worker nodes due to a transient problem with apimdev1066, as shown in the following pod list. Pods without persistent storage, such as nginx-X, can be rescheduled to apimdev1066 as soon as they are restarted, but any pods with persistent local storage, such as db-X and www-X, have to be rescheduled onto the same worker node as that is where their files live.

$ kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
ejs-portal-nginx-84f57ffd8c-hbf66   1/1     Running   0          5m12s   888.16.109.208   apimdev0103   <none>           <none>
ejs-portal-nginx-84f57ffd8c-mvq96   1/1     Running   0          5m12s   888.16.142.215   apimdev0129   <none>           <none>
ejs-portal-nginx-84f57ffd8c-vpmtl   1/1     Running   0          5m12s   888.16.142.214   apimdev0129   <none>           <none>
ejs-portal-site1-db-0               2/2     Running   0          4m39s   888.16.109.209   apimdev0103   <none>           <none>
ejs-portal-site1-db-1               2/2     Running   0          6m37s   888.16.109.206   apimdev0103   <none>           <none>
ejs-portal-site1-db-2               2/2     Running   0          4m39s   888.16.142.216   apimdev0129   <none>           <none>
ejs-portal-site1-www-0              2/2     Running   0          4m9s    888.16.109.210   apimdev0103   <none>           <none>
ejs-portal-site1-www-1              2/2     Running   0          6m37s   888.16.142.213   apimdev0129   <none>           <none>
ejs-portal-site1-www-2              2/2     Running   0          4m9s    888.16.142.217   apimdev0129   <none>           <none>
ibm-apiconnect-75b47f9f87-p25dd     1/1     Running   0          5m12s   888.16.109.207   apimdev0103   <none>           <none>

The upgrade to version 10.0.3.0 is started and the following pod list is observed:

$ kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
ejs-portal-nginx-84f57ffd8c-hbf66   1/1     Running   0          10m     888.16.109.208   apimdev0103   <none>           <none>
ejs-portal-nginx-84f57ffd8c-mvq96   1/1     Running   0          10m     888.16.142.215   apimdev0129   <none>           <none>
ejs-portal-nginx-84f57ffd8c-vpmtl   1/1     Running   0          10m     888.16.142.214   apimdev0129   <none>           <none>
ejs-portal-site1-db-0               2/2     Running   0          10m     888.16.109.209   apimdev0103   <none>           <none>
ejs-portal-site1-db-1               0/2     Pending   0          91s     <none>           <none>        <none>           <none>
ejs-portal-site1-db-2               2/2     Running   0          2m41s   888.16.142.218   apimdev0129   <none>           <none>
ejs-portal-site1-www-0              2/2     Running   0          9m51s   888.16.109.210   apimdev0103   <none>           <none>
ejs-portal-site1-www-1              2/2     Running   0          12m     888.16.142.213   apimdev0129   <none>           <none>
ejs-portal-site1-www-2              0/2     Pending   0          111s    <none>           <none>        <none>           <none>
ibm-apiconnect-75b47f9f87-p25dd     1/1     Running   0          10m     888.16.109.207   apimdev0103   <none>           <none>

The pod list shows that db-2 has restarted, and has been rescheduled to apimdev0129 as there were no other db pods running on that node. However, db-1 and www-2 are both stuck in Pending state, as there is already a pod of the same type running on the worker node that is hosting the local storage that they are bound to. If you run a describe command of either pod you will see the following output:

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  10s (x4 over 2m59s)  default-scheduler  0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules, 2 node(s) had volume node affinity conflict.

To resolve this situation you need to delete the PVCs for each pod, and then delete the pod itself, so that Kubernetes will regenerate the PVCs and schedule the pod on the worker node that does not have the anti-affinity conflict.

Therefore, for the db-1 pod the following commands must be run:

$ kubectl get pvc | grep ejs-portal-site1-db-1
db-ejs-portal-site1-db-1        Bound    local-pv-fa445e30   250Gi      RWO            local-storage   15m
dblogs-ejs-portal-site1-db-1    Bound    local-pv-d57910e7   250Gi      RWO            local-storage   15m

$ kubectl delete pvc db-ejs-portal-site1-db-1 dblogs-ejs-portal-site1-db-1
persistentvolumeclaim "db-ejs-portal-site1-db-1" deleted
persistentvolumeclaim "dblogs-ejs-portal-site1-db-1" deleted

$ kubectl delete po ejs-portal-site1-db-1
pod "ejs-portal-site1-db-1" deleted

For the www-2 pod the following commands must be run:

$ kubectl get pvc | grep ejs-portal-site1-www-2
admin-ejs-portal-site1-www-2    Bound    local-pv-48799536   245Gi      RWO            local-storage   51m
backup-ejs-portal-site1-www-2   Bound    local-pv-a93f5607   245Gi      RWO            local-storage   51m
web-ejs-portal-site1-www-2      Bound    local-pv-facd4489   245Gi      RWO            local-storage   51m

$ kubectl delete pvc admin-ejs-portal-site1-www-2 backup-ejs-portal-site1-www-2 web-ejs-portal-site1-www-2
persistentvolumeclaim "admin-ejs-portal-site1-www-2" deleted
persistentvolumeclaim "backup-ejs-portal-site1-www-2" deleted
persistentvolumeclaim "web-ejs-portal-site1-www-2" deleted

$ kubectl delete po ejs-portal-site1-www-2
pod "ejs-portal-site1-www-2" deleted

If the PVC has persistentVolumeReclaimPolicy: Delete set on it, as is the case for the OVA deployments, then no cleanup is necessary as the old data will have been deleted on the worker node that is no longer running the db-1 and www-2 pods.

Kubernetes can now reschedule the pods. All pods with persistent storage are now spread across the available worker nodes, and the pods whose PVCs were deleted will get a full copy of the data from the existing running pods. The following pod list is now observed in our example:

$ kubectl get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP               NODE          NOMINATED NODE   READINESS GATES
ejs-portal-nginx-84f57ffd8c-f85wm   1/1     Running   0          30s     888.16.29.136    apimdev1066   <none>           <none>
ejs-portal-nginx-84f57ffd8c-k5klb   1/1     Running   0          103s    888.16.142.220   apimdev0129   <none>           <none>
ejs-portal-nginx-84f57ffd8c-lqhqs   1/1     Running   0          1m53s   888.16.109.212   apimdev0103   <none>           <none>
ejs-portal-site1-db-0               2/2     Running   0          6m43s   888.16.109.211   apimdev0103   <none>           <none>
ejs-portal-site1-db-1               2/2     Running   0          8m20s   888.16.29.134    apimdev1066   <none>           <none>
ejs-portal-site1-db-2               2/2     Running   0          14m     888.16.142.218   apimdev0129   <none>           <none>
ejs-portal-site1-www-0              2/2     Running   0          93s     888.16.109.213   apimdev0103   <none>           <none>
ejs-portal-site1-www-1              2/2     Running   0          3m55s   888.16.142.219   apimdev0129   <none>           <none>
ejs-portal-site1-www-2              2/2     Running   0          7m27s   888.16.29.135    apimdev1066   <none>           <none>
ibm-apiconnect-75b47f9f87-p25dd     1/1     Running   0          22m     888.16.109.207   apimdev0103   <none>           <none>

Workaround for non-ready stancluster pods after upgrade

After upgrading from 10.0.1.2 to 10.0.3.0 or later on VMware, one of the management stancluster pods may remain in non-ready state after rebooting the VM the pod is running on.

The output of kubectl get pods will look similar to the following for the affected pod:

management-stancluster-1                     0/1     Running     1          40m

Pods in a ready state show a status similar to the following:


management-stancluster-2                              1/1     Running     1          40m
management-stancluster-3                              1/1     Running     1          40m

The 0/1 status does not affect the outcome of the upgrade or the operation of the stancluster. However you should delete the affected pod after the upgrade is complete.

To delete the affected pod:

SSH into the appliance VM.
Run:
```
kubectl delete pod <stancluster pod>
```
The nats-streaming-operator will automatically re-create the deleted pod.
Verify that the pod has returned to status 1/1 Running.
```
kubectl get pod  | grep <stancluster pod>
```

Issues when installing Drupal 8 based custom modules or sub-themes into the Drupal 9 based Developer Portal

From IBM API Connect 10.0.3.0, the Developer Portal is based on the Drupal 9 content management system. If you want to install Drupal 8 custom modules or sub-themes into the Drupal 9 based Developer Portal, you must ensure that they are compatible with Drupal 9, including any custom code that they contain, and not using any deprecated APIs, for example. There are tools available for checking your custom code, such as drupal_check on GitHub, which checks Drupal code for deprecations.

For example, any Developer Portal sites that contain modules or sub-themes that don't contain a Drupal 9 version declaration will fail to upgrade, and errors like the following output will be seen in the admin logs:

[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: Checking theme: emeraldgreen
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: Incompatible core_version_requirement '' found for emeraldgreen
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: Checking theme: rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: Incompatible core_version_requirement '8.x' found for rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: Found themes incompatible with Drupal 9: emeraldgreen rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:34:49: check_d9_compat: ERROR: /tmp/restore_site.355ec8 is NOT Drupal 9 compatible
...
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: Checking module: custom_mod_1
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: Incompatible core_version_requirement '' found for custom_mod_1
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: Checking module: custom_mod_2
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: Incompatible core_version_requirement '8.x' found for custom_mod_2
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: Found modules incompatible with Drupal 9: emeraldgreen rubyred
[     queue stdout] 14834 729319:355ec8:a7d29c 2021-09-04 20:44:49: check_d9_compat: ERROR: site1.com is NOT Drupal 9 compatible

To fix version compatibility errors, all custom modules and sub-themes should declare a core_version_requirement key in their *.info.yml file that indicates Drupal 9 compatibility. For example:

name: Example module
type: module
description: Purely an example
core: 8.x
core_version_requirement: '^8 || ^9'
package: Example module

# Information added by Drupal.org packaging script on 2020-05-31
version: '8.x-1.3'
project: 'example_module'
datestamp: 1590905415

This example specifies that the module is compatible with all versions of Drupal 8 and 9. For more information, see Let Drupal know about your module with an .info.yml file on the drupal.org website.

If you have a backup of a site that you need to restore, and are getting the version compatibility error, but the module or theme *.info.yml file cannot be changed easily, then you have two options. Either:

Add an environment variable into the portal CR for the www pod of the admin container stating SKIP_D9_COMPAT_CHECK: "true". However, if you choose this method, you must be positive that all of the custom modules and themes for your sites are Drupal 9 compatible, as otherwise the sites may end up inaccessible after the upgrade or restore.
1. On VMware, create an extra values file to contain the environment variable, as follows:
```
spec:
  template:
  - containers:
    - env:
      - name: SKIP_D9_COMPAT_CHECK
        value: "true"
      name: admin
    name: www
```
2. Save the file as d9compat.yaml, and run the following command:
```
apicup subsys set <portal_subsystem_name> extra-values-file d9compat.yaml
```
3. Then, update the portal VMware with the updated setting by running the following command:
```
apicup subsys install <portal_subsystem_name>
```

Or:

Extract the site backup, edit the relevant files inside it, and then tar the backup file again. Note that this procedure will overwrite the original backup file, so ensure that you keep a separate copy of the original file before you start the extraction. For example:
1. mkdir /tmp/backup
2. cd /tmp/backup
3. tar xfz path_to_backup.tar.gz
4. Edit the custom module and theme files to make them Drupal 9 compatible, and add the correct core_version_requirement setting.
5. rm -f path_to_backup.tar.gz
6. tar cfz path_to_backup.tar.gz
7. cd /
8. rm -rf /tmp/backup

Skipping health check when re-running upgrade

The apicup subsys install command automatically runs apicup health-check prior to attempting the upgrade. An error is displayed if a problem is found that will prevent successful upgrade

In some scenarios, if you encounter an upgrade failure, an attempt to rerun apicup subsys install is blocked by errors found by apicup health-check. Even when you have fixed the error (such as reconfiguration of an incorrect upgrade CR), the failed upgrade can continue to cause the health check to fail.

You can workaround the problem by adding the --skip-health-check flag to suppress the health check:

apicup subsys install <subsystem_name> --skip-health-check

In this case, use of --skip-health-check allows the upgrade to rerun successfully.