Installing API Connect on the warm-standby data center

Add the multiSiteHA configuration to the API Connect Management and Portal YAML files before they are applied.

Before you begin

Verify that all the secrets, certificates, and issuers are ready:

oc get secrets -n <namespace>
ingress-ca                kubernetes.io/tls                     3      3d21h
mgmt-replication-client   kubernetes.io/tls                     3      3d21h
ptl-replication-client    kubernetes.io/tls                     3      3d21h

oc get certs -n <namespace>
mgmt-replication-client   True    mgmt-replication-client   3d21h   2024-08-17T13:04:27Z
ptl-replication-client    True    ptl-replication-client    3d21h   2024-08-17T13:04:26Z

oc get issuer -n <namespace>
ingress-issuer   True    3d21h
self-signed      True    3d21h

Where <namespace> are the namespaces you created for API Connect.

Note: If you have separate namespaces for Management and Portal subsystems, then run the check in both namespaces. You should not see the mgmt-replication-client in the portal namespace, nor the ptl-replication-client in the management namespace.

Procedure

Follow these steps for installing the Management subsystem on OpenShift Installing the management subsystem in a shared namespace, but add the encryptionSecret and multiSiteHA section to the mgmt_cr.yaml file:
```
  siteName: <dc2 site name>
  encryptionSecret:
    secretName: mgmt-encryption-key
  multiSiteHA:
    mode: passive
    replicationEndpoint:
      annotations:
        cert-manager.io/issuer: ingress-issuer
      hosts:
      - name: mgmt-replication.<warm-standby data center ingress domain>
        secretName: mgmt-replication-server
    replicationPeerFQDN: mgmt-replication.<active data center ingress domain>
    tlsClient:
      secretName: mgmt-replication-client
```
where
- <dc2 site name> is the site name for this data center, as decided in Planning and initial preparation.
- <active data center ingress domain> and <warm-standby data center ingress domain> can be determined by running the command oc get ingresses.config/cluster -o jsonpath={.spec.domain} in each data center.
Note: If you are installing the Management subsystem in a separate namespace, these same steps apply.

Remember: Make sure that the management endpoints are set the same as for your active data center deployment.

Apply the update mgmt_cr.yaml file as directed in Installing the management subsystem in a shared namespace. Monitor the deployment, but note that on the warm-standby there are fewer pods than on the active:

oc get ManagementCluster -n <namespace>

NAME         READY   STATUS    VERSION    RECONCILED VERSION   MESSAGE      AGE
management   n/n     Running   10.0.8.0   10.0.8.0-1281        Management is ready. HA Status Warning - see HAStatus in CR for details      62m

Follow these steps for installing the Portal subsystem on OpenShift Installing the portal subsystem in a shared namespace, but add the encryptionSecret and multiSiteHA section to the portal_cr.yaml file.
```
  siteName: <dc2 site name>
  encryptionSecret:
    secretName: ptl-encryption-key
  multiSiteHA:
    mode: passive
    replicationEndpoint:
      annotations:
        cert-manager.io/issuer: ingress-issuer
      hosts:
      - name: ptl-replication.<warm-standby data center ingress domain>
        secretName: ptl-replication-server
    replicationPeerFQDN: ptl-replication.<warm-standby data center ingress domain>
    tlsClient:
      secretName: ptl-replication-client
```
where
- <dc2 site name> is the site name for this data center, as decided in Planning and initial preparation.
- <active data center ingress domain> and <warm-standby data center ingress domain> can be determined by running the command oc get ingresses.config/cluster -o jsonpath={.spec.domain} in each data center.
Remember: Make sure that the portal endpoints are set the same as for your active data center deployment.

Apply the updated portal_cr.yaml file as directed in Installing the portal subsystem in a shared namespace.

oc get PortalCluster -n <namespace>
NAME     READY   STATUS    VERSION    RECONCILED VERSION   AGE
portal   6/6     Running   10.0.8.0   10.0.8.0-1281        47m

Results

While the management subsystems on the warm-standby and active data centers are synchronizing their databases, the management status reports Warning, and the haStatus reports pending:

oc get mgmt -n <namespace>


NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
management   n/n     Warning   10.0.8.0-0   10.0.8.0-0           Management is ready. HA Status Warning - see HAStatus in CR for details   8m59s

oc get mgmt -n <namespace> -o yaml

...
status:
  haStatus
    {
      "lastTransitionTime": "2023-12-31T19:47:08Z",
      "message": "Replication not working, install or upgrade in progress.",
      "reason": "na",
      "status": "True",
      "type": "Pending"
   }

When the management database replication between sites is complete, the management status reports Running, and status.haStatus reports Ready:

oc get mgmt -n <namespace>

NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
management   n/n     Running   10.0.8.0-0   10.0.8.0-0           Management is ready. HA status Ready - see HAStatus in CR for details              8m59s

oc get mgmt -n <namespace> -o yaml

...
status:
  haStatus
  {
    "lastTransitionTime": "2023-03-31T19:47:08Z",
    "message": "Replication is working",
    "reason": "na",
    "status": "True",
    "type": "Ready"
  }

If the management database replication between sites fails for any reason other than because an install or upgrade is in progress, the haStatus output shows the reason, for example::

NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
management   n/n     Warning   10.0.8.0-0   10.0.8.0-0           Management is ready. HA Status Warning - see HAStatus in CR for details   8m59s

status:
  haStatus
    {
      "lastTransitionTime": "2023-12-31T19:47:08Z",
      "message": "Replication not working",
      "reason": "na",
      "status": "True",
      "type": "PeerNotReachable"
   }

If the warning persists, see Troubleshooting a two data center deployment.

You can validate that your portal deployments are synchronizing by running oc get pods on both the active and warm-standby data centers. Confirm that the number and names of the pods all match (the UUIDs in the names might be different on each site), and that all are in the Ready state.

For additional replication verification checks, see Verifying replication between data centers. It is recommended to run a test failover, and confirm that all of the expected data is present and correct on the newly active site. See Failing over to the warm-standby.

What to do next

Deploy your Gateway and Analytics subsystems for shared namespace: Installing API Connect in a single namespace on OpenShift or multiple namespaces: Installing API Connect subsystems in different namespaces or environments on OpenShift

Configure your deployment: Cloud Manager configuration checklist, and complete a test failover to verify that your data replicated to the warm-standby. See Failing over to the warm-standby for details.

Important: It is strongly recommended to take complete backups of both sites, for disaster recovery in case the API Connect installation in both data centers is lost, or if a replication failure occurs. Complete backups consist of your infrastructure configuration backup and your subsystem database backups. See Backup and restore requirements for a 2DCDR deployment.