Verifying replication between data centers

How to verify that your API Connect data is replicating to your warm-standby data center.

The management and portal UIs are not accessible on the warm-standby data center, so it is not possible to check that data replication is working by logging in to the UIs. Instead, you can verify that the warm-standby management and portal databases are in sync with command-line operations.

Note: For OpenShift users: The steps that are detailed in this topic use the Kubernetes kubectl command. On OpenShift, use the equivalent oc command in its place.

Verifying Management replication

The management CR reports a warning if it detects a problem with management data replication:

kubectl get mgmt

NAME         READY   STATUS    VERSION      RECONCILED VERSION   MESSAGE                                                                          AGE
management   n/n     Warning   10.0.8.0-0   10.0.8.0-0           Management is ready. HA Status Warning - see HAStatus in CR for details   8m59s

kubectl get mgmt -o yaml

...
status:
  haStatus
    {
      "lastTransitionTime": "2023-03-31T19:47:08Z",
      "message": "Replication not working",
      "reason": "na",
      "status": "True",
      "type": "Warning"
   }

If the warning persists, then see Troubleshooting a two data center deployment.

To verify replication by creating new data on the active and checking that it is copied to the warm-standby, follow these steps:

Login to the Cloud Manager UI.
Create a local user registry called 2dcdrtest: Configuring a Local User Registry.
Run the following command in your warm-standby data center to query the management database for the new user registry:
```
kubectl exec -n <namespace> `kubectl -n <namespace> get po -l role=primary -o name` -- bash -c "psql -U postgres apim -c \"select name,created_at from user_registry where name = '2dcdrtest'\""
```
Expected output is:
```
Defaulted container "database" out of: database, set-libpq-certs (init)
   name    |       created_at       
-----------+------------------------
 2dcdrtest | <date and time when step 2 was done>
(1 row)
```
If 2dcdrtest is returned and has a creation time consistent with when you did step 2, then replication is working. If it is not returned, then either replication is not working, or the data is not yet replicated. If an older creation date is returned, then it's possible the registry was not deleted after a previous check, in which case delete the registry and repeat this test. If replication is not working, follow the 2DCDR troubleshooting steps: Troubleshooting a two data center deployment.
Return to the Cloud Manager UI, and delete the 2dcdrtest user registry.

Verifying Portal replication

Follow these steps to verify that the portal database is replicating from the active data center to the warm-standby data center.

In the active data center, identify the active db pod:
```
kubectl get pods -n <namespace> | grep "<active site name>-db"
```
Example output:
```
<portal instance name>-<active site name>-db-0                                                2/2     Running     0          22h
```
where <active site name> is the siteName property from the portal CR on your active data center. The active data center db pods have 2 containers, so show a status of 2/2. A three replica deployment has three db pods and you can pick any one of them for the next step.
In the active data center, run the following command:
```
kubectl exec -ti -n <namespace> <active portal db pod> -c db -- bash -ic "dbstatus 2>&1 | grep Last | sed 's/.*committed: \([0-9]*\).*/\1/'"
```
Example output:
```
355175
```
where <active portal db pod> is the pod name that is returned from step 1. The command returns a number.
In the warm-standby data center, identify the warm-standby db pod:
```
kubectl get pods -n <namespace> | grep "<warm-standby site name>-db"
```
Example output:
```
<portal instance name>-<warm-standby site name>-db-0                                                2/2     Running     0          22h
```
where <warm-standby site name> is the siteName property from the portal CR on your warm-standby data center. The warm-standby data center db pods have 1 container, so show a status of 1/1. A three replica deployment has three db pods and you can pick any one of them for the next step.
In the warm-standby data center, run the following command:
```
kubectl exec -ti -n <namespace> <passive portal db pod> -c db -- bash -ic "dbstatus 2>&1 | grep Last | sed 's/.*committed: \([0-9]*\).*/\1/'"
```
Example output:
```
355495
```
where <passive portal db pod> is the pod name that is returned from step 3. The command returns a number.
The numbers that are returned from steps 2 and 4 represent the portal database's last committed transaction number. When the active data center commits a new transaction the number is incremented, this update is replicated to the warm-standby data center. If replication is working the number that is returned from the warm-standby data center should always be close to the number that is returned from the active data center. It is unlikely to be the same number that is returned, unless you are able run step 4 and step 2 at precisely the same time.
Repeat steps 2 and 4 several times to confirm that the warm-standby number is keeping up with the active number. If the number returned from the warm-standby data center is far behind, or ahead of the number that is returned from the active, then there is likely to be a replication problem, see Troubleshooting a two data center deployment.