Troubleshooting backup and restore on VMware

Troubleshooting steps for problems with backup and restore of the analytics database on VMware.

Common problems with backup and restore

These are some frequent issues that can cause backup or restore to fail:

Invalid login credentials for your remote backup server. Check that the username and password that you configured in your subsystem database backup settings is correct.
The user does not have write permissions on the backup server. Check that the username you configured in your subsystem database backup settings has write permission to the specified backup path.
Remote backup server storage full. Check your remote backup server and if the storage is full either extend the storage space or delete older backups.
No network access to the remote backup server. Check that you can communicate with your remote backup server from your API Connect environment.
TLS handshaking failure with the remote backup server. If your remote backup server has a self-signed CA certificate, check that this certificate is trusted by your API Connect deployment, see the database backup configuration steps for your subsystem for more information.

Backup failure in three replica deployment due to node failure

The analytics database backup service is responsible for staging and exporting analytics database backups to your remote SFTP or S3 object-store. The backup service runs on a single node on all deployment profiles. In a three replica deployment, if the node where the backup service is running fails, then take the manual steps that are documented here to recover the service.

To recover the backup service on a different node, complete the following steps:

Log in to one of the analytics VMs, and switch to the root user:
1. Log in to the analytics VM:
```
ssh apicadm@<analytics subsystem hostname>
```
2. ```
sudo -i
```
Disable analytics database backups.
Edit your analytics CR:
```
kubectl edit a7s
```
set spec.databaseBackup.enabled = false.

Identify the name of the warehouse PVC:

kubectl get pvc | grep warehouse

Look for the warehouse PVC in the output:


NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
warehouse-data-analytics-warehouse-0   Bound    pvc-4cbbb8a0-10ad-45de-b7da-ae6c26bde1de   150Gi      RWO            standard       24h

Delete the warehouse PVC:

kubectl delete pvc <name of warehouse pvc>

Re-enable analytics database backups.
Edit your analytics CR:
```
kubectl edit a7s
```
Set spec.databaseBackup.enabled = true.
The PVC is then re-created on one of the two available nodes, and the warehouse pod starts on this same node.
Exit from the analytics VM.