Upgrading a 2DCDR deployment on Kubernetes and OpenShift from V10.0.7 or later

How to upgrade an API Connect V10.0.7 (or later) two data center disaster recovery (DR) deployment on Kubernetes and OpenShift.

To upgrade API Connect in a 2DCDR deployment, use the upgrade instructions for your platform, but with the extra considerations and steps documented in this topic.

Note: For OpenShift users: The steps that are detailed in this topic use the Kubernetes kubectl command. On OpenShift, use the equivalent oc command in its place.

Before you begin

Verify that API Connect is running in both data centers and that replication is working: Verifying replication between data centers.
Check that you have recent backups of management and portal subsystems: Backup and restore requirements for a 2DCDR deployment.

Key points applicable to both management and portal subsystems

Your API Connect deployments must be upgraded to the same API Connect release, down to the interim fix level.
Both data centers must be upgraded in the same maintenance window.
Upgrade steps might result in an update to the ingress-ca X.509 certificate. Extra steps must be taken at various points in the upgrade process to ensure that the ingress-ca secret in both data centers contains the same ingress-ca X.509 certificate.

Note: If you are not using cert-manager and you customized your certificates, the ingress-ca certificate might have a different name. Ensure that the CA certificate that is used by both data centers is the same during all stages of the upgrade.

Steps for management upgrades

Upgrade the warm-standby data center first.

Follow the upgrade steps for your platform:

Check the status of the warm-standby upgrade and run the following commands:

Get the management CR summary:

kubectl get mgmt -A

returns:

NAMESPACE   NAME   READY   STATUS    VERSION    RECONCILED VERSION   MESSAGE                                                                                                            AGE
apic        mgt    5/6     Pending   10.0.8.0   10.0.8.0-0           Some services are not ready - see status condition for details. HA status Ready - see HAStatus in CR for details   48m

k get mgmt m2 -n nihar -o json | jq -r '.status.conditions[] | select(.status=="True")'
{
  "lastTransitionTime": "2024-05-14T00:26:25Z",
  "message": "Not all services are ready, next pending services: apim-lur-schema-data-check. Please upgrade Active to 10.0.8.0 to match warm standby's version. HA status Ready - see HAStatus in CR for details",
  "reason": "na",
  "status": "True",
  "type": "Pending"
}

Get the management CR status:

kubectl get mgmt <mgmt CR name> -o json | jq -r '.status.conditions[] | select(.status=="True")'

returns:

{
  "lastTransitionTime": "2024-05-14T00:26:25Z",
  "message": "Not all services are ready, next pending services: apim-lur-schema-data-check. Please upgrade Active to 10.0.8.0 to match warm standby's version. HA status Ready - see HAStatus in CR for details",
  "reason": "na",
  "status": "True",
  "type": "Pending"
}

Note: The warm-standby may not remain in a pending state in all cases during an upgrade. When there are no schema changes between the source and target versions, warm-standby will move to a ready state, which is an acceptable condition. If schema changes are present, warm-standby will remain in a pending state until the active system is upgraded.

Verify that both data centers have the same ingress-ca X.509 certificate in their ingress-ca secret. Run the following command in both data centers and check that the output is the same:
```
openssl x509 -noout -fingerprint -sha256 -in <(kubectl get secret ingress-ca -n <namespace> -o yaml | grep "^  tls.crt:" | awk '{print $2}' | base64 -d)
```
if you do not have the openssl command available, you can instead run only the kubectl part, which produces a larger output:
```
kubectl get secret ingress-ca -n <namespace> -o yaml | grep "^  tls.crt:" | awk '{print $2}' | base64 -d
```
if the outputs are different, follow these steps to synchronize the certificates: Synchronizing the ingress-ca X.509 certificate across data centers.
Upgrade the active data center, following the steps for your platform:
- Upgrading API Connect on Kubernetes.
- Upgrading API Connect on OpenShift.

Steps for portal upgrades

Verify that both data centers have the same ingress-ca X.509 certificate in their ingress-ca secret. Run the following command in both data centers and check that the output is the same:
```
openssl x509 -noout -fingerprint -sha256 -in <(kubectl get secret ingress-ca -n <namespace> -o yaml | grep "^  tls.crt:" | awk '{print $2}' | base64 -d)
```
if you do not have the openssl command available, you can instead run only the kubectl part, which produces a larger output:
```
kubectl get secret ingress-ca -n <namespace> -o yaml | grep "^  tls.crt:" | awk '{print $2}' | base64 -d
```
if the outputs are different, follow these steps to synchronize the certificates: Synchronizing the ingress-ca X.509 certificate across data centers.
Start the upgrade of your warm-standby data center by following the upgrade documentation for your platform. Stop at the point where the portal CR is updated with the new API Connect version.
Verify that both data centers still have the same ingress-ca X.509 certificate, repeating step 1. If they are different, then follow these steps: Synchronizing the ingress-ca X.509 certificate across data centers.
Continue the upgrade of your warm-standby data center by updating the portal subsystem CR, following the remaining upgrade steps for your platform. Do not wait for the warm-standby to reach READY state before starting the upgrade on the active data center (in certain circumstances the warm-standby portal does not reach READY state until the active data center is upgraded).
Start the upgrade of your active data center by following the upgrade documentation for your platform. Stop at the point where the portal CR is updated with the new API Connect version.
Verify that both data centers still have the same ingress-ca X.509 certificate, repeating step 1. If they are different, then follow these steps: Synchronizing the ingress-ca X.509 certificate across data centers.
Upgrade the portal subsystem in your active data center by updating the portal subsystem CR, following the remaining upgrade steps for your platform.

How to upgrade when one data center is down

If API Connect is still running on the failed data center, follow the steps that are documented previously to upgrade both data centers before you bring the failed data center back online.

If the failed data center is expected to be down for a long time, you can convert the active data center to a stand-alone data center by following these steps: Removing a two data center deployment, but note the following points:

Ensure that the network links to the failed data center are removed.
Ensure that the failed data center is set to warm-standby in the multiSiteHA section. Do not proceed to the next step until the data center completes the transition to warm-standby. View the status of the management and portal CRs to confirm that HA Mode reports passive.
Remove the multiSiteHA section from failed data center, and verify that the failed data center resets itself to become an empty stand-alone API Connect deployment (all data is deleted).
Before you restore the network links between the data centers, do the following operations:
- Upgrade API Connect on the failed data center to the same version as the active.
- Add the multiSiteHA sections to both data centers, setting the failed data center to be warm-standby.
  Important: Do not set the failed data center to be active in the multiSiteHA section because it results in an overwrite of the data on your working data center with the empty database of your failed data center.