Installing API Connect on the warm-standby data center

Follow the installation steps for API Connect described in the Cloud Pak for Integration documentation, adding the multiSiteHA configuration to the API Connect YAML.

Before you begin

Verify that all the secrets, certificates, and issuers are ready:
oc get certificate -n <namespace> | grep <apic-instance-name>
NAME                           READY   SECRET                         AGE     EXPIRATION
apic-mgmt-replication-client   True    apic-mgmt-replication-client   57s   2024-08-24T09:45:03Z
apic-ptl-replication-client    True    apic-ptl-replication-client    45s   2024-08-24T09:45:17Z

[root@api.iaincp4ilarge2.cp.fyre.ibm.com ~]# oc get secrets  -n <namespace> | grep <apic-instance-name>
NAME                                 TYPE                                  DATA   AGE
apic-ingress-ca                      kubernetes.io/tls                     3      4m6s
apic-mgmt-replication-client         kubernetes.io/tls                     3      2m40s
apic-ptl-replication-client          kubernetes.io/tls                     3      2m26s

[root@api.iaincp4ilarge2.cp.fyre.ibm.com ~]# oc get issuer -n <namespace> | grep <apic-instance-name>
NAME                  READY   AGE
apic-ingress-issuer   True    4m8s
Where <apic-instance-name> is the name you intend to use for your API Connect cluster CR, and <namespace> is the namespace you created for API Connect.

About this task

In the yaml files that are shown here, replace <apic-instance-name> with the name you intend to use for your API Connect Cluster CR. As decided in Planning and initial preparation. Set <active data center ingress domain> and <warm-standby data center ingress domain> to their appropriate values, which can be determined by running this command in each data center:
oc get ingresses.config/cluster -o jsonpath={.spec.domain}

Procedure

  1. Apply the Cloud Pak for Integration credentials that you copied from your active data center (see Copy active credentials). 
    oc -n <namespace> apply -f cp4i-creds.json
  2. Follow the API Connect installation steps using the Platform UI, as described in the Cloud Pak for Integration documentation https://www.ibm.com/docs/en/cloud-paks/cp-integration. At the point where you specify the configuration properties of your API Connect instance, switch to the YAML tab so you can edit the YAML directly.
    Remember: Use the same <apic-instance-name> value when you specify the API Connect instance name in the Cloud Pak for Integration Platform UI.
  3. Add the management.encryptionSecret.secretName property, and the management.multiSiteHA section to the YAML file inside the spec section. 
      management:
    ...
    encryptionSecret:
      secretName: mgmt-encryption-key
    multiSiteHA:
      mode: passive
      replicationEndpoint:
        annotations:
          cert-manager.io/issuer: <apic-instance-name>-ingress-issuer
        hosts:
        - name: mgmt-replication.<warm-standby data center ingress domain>
          secretName: <apic-instance-name>-mgmt-replication-server
      replicationPeerFQDN: mgmt-replication.<active data center ingress domain>
      tlsClient:
        secretName: <apic-instance-name>-mgmt-replication-client
    Note: Warm-standby is referred to as 'passive' in the CR YAML.

    Also set the management.<endpoint>.hosts.name properties to the endpoint URLs you decided on in Planning and initial preparation.

  4. Add the portal.encryptionSecret.secretName property, and the portal.multiSiteHA section to the YAML file inside the spec section. 
      portal:
    ...
    encryptionSecret:
      secretName: ptl-encryption-key
    multiSiteHA:
      mode: passive
      replicationEndpoint:
        annotations:
          cert-manager.io/issuer: <apic-instance-name>-ingress-issuer
        hosts:
        - name: ptl-replication.<warm-standby data center ingress domain>
          secretName: <apic-instance-name>-ptl-replication-server
      replicationPeerFQDN: ptl-replication.<active data center ingress domain>
      tlsClient:
        secretName: <apic-instance-name>-ptl-replication-client

    Also set the portal.<endpoint>.hosts.name properties to the endpoint URLs you decided on in Planning and initial preparation.

  5. Continue the installation steps as described in Deploying on OpenShift and Cloud Pak for Integration.

Results

Installation can take around 1 hour to complete.

While the management subsystems on the warm-standby and active data centers are synchronizing, the management status reports Warning, and the haStatus reports pending:
oc get mgmt -n <namespace>

NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
management   n/n     Warning   10.0.8.0-0   10.0.8.0-0           Management is ready. HA Status Warning - see HAStatus in CR for details   8m59s

status:
  haStatus
    {
      "lastTransitionTime": "2023-03-31T19:47:08Z",
      "message": "Replication not working, install or upgrade in progress.",
      "reason": "na",
      "status": "True",
      "type": "Pending"
   }
When the management database replication between sites is complete, the management status reports Running, and status.haStatus reports Ready:
NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
management   n/n     Running   10.0.8.0-0   10.0.8.0-0           Management is ready. HA status Ready - see HAStatus in CR for details              8m59s

status:
  haStatus
  {
    "lastTransitionTime": "2023-03-31T19:47:08Z",
    "message": "Replication is working",
    "reason": "na",
    "status": "True",
    "type": "Ready"
  }
After the management subsystems in the active and warm-standby sites are synchronized, the other subsystems are deployed. Verify on both active and warm-standby sites that all subsystems reach Running state:
oc get all -n <namespace>
...
NAME                                                             READY   STATUS    VERSION    RECONCILED VERSION   AGE
analyticscluster.analytics.apiconnect.ibm.com/apis-minimum-a7s   n/n     Running   10.0.8.0   10.0.8.0-1281        27m

NAME                                                READY   STATUS   VERSION    RECONCILED VERSION   AGE
apiconnectcluster.apiconnect.ibm.com/apis-minimum   n/n     Ready    10.0.8.0   10.0.8.0-1281        2d20h

NAME                                                 PHASE     READY   SUMMARY                           VERSION    AGE
datapowerservice.datapower.ibm.com/apis-minimum-gw   Running   True    StatefulSet replicas ready: 1/1   10.5.0.0   25m

NAME                                                 PHASE     LAST EVENT   WORK PENDING   WORK IN-PROGRESS   AGE
datapowermonitor.datapower.ibm.com/apis-minimum-gw   Running                false          false              25m

NAME                                                        READY   STATUS    VERSION    RECONCILED VERSION   AGE
gatewaycluster.gateway.apiconnect.ibm.com/apis-minimum-gw   n/n     Running   10.0.8.0   10.0.8.0-1281        27m

NAME                                                                READY   STATUS    VERSION    RECONCILED VERSION   AGE
managementcluster.management.apiconnect.ibm.com/apis-minimum-mgmt   8/8     Running   10.0.8.0   10.0.8.0-1281        2d20h

NAME                                                       READY   STATUS    VERSION    RECONCILED VERSION   AGE
portalcluster.portal.apiconnect.ibm.com/apis-minimum-ptl   n/n     Running   10.0.8.0   10.0.8.0-1281        27m
Note: The warm-standby site has fewer managementcluster pods than the active site.
If the management database replication between sites fails for any reason other than because an install or upgrade is in progress, the haStatus output shows the reason, for example::
NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
management   n/n     Warning   10.0.8.0-0   10.0.8.0-0           Management is ready. HA Status Warning - see HAStatus in CR for details   8m59s

status:
  haStatus
    {
      "lastTransitionTime": "2023-12-31T19:47:08Z",
      "message": "Replication not working",
      "reason": "na",
      "status": "True",
      "type": "PeerNotReachable"
   }
If the warning persists, see Troubleshooting a two data center deployment.

You can validate that your portal deployments are synchronizing by running oc get pods on both the active and warm-standby data centers. Confirm that the number and names of the pods all match (the UUIDs in the names might be different on each site), and that all are in the Ready state.

For additional replication verification checks, see Verifying replication between data centers. It is recommended to run a test failover, and confirm that all of the expected data is present and correct on the newly active site. See Failing over to the warm-standby.

What to do next

Configure your deployment: Cloud Manager configuration checklist.
Important: It is strongly recommended to take complete backups of both sites, for disaster recovery in case the API Connect installation in both data centers is lost, or if a replication failure occurs. Complete backups consist of your infrastructure configuration backup and your subsystem database backups. See Backup and restore requirements for a 2DCDR deployment.