Removing tombstone records from CouchDB databases

Remove tombstone records from CouchDB databases to speed up the backup and restore process.

Before you begin

Install the command-line interface (CLI) utility cpctl from the cp-serviceability pod. For more information, see Installing the cpctl utility.

Procedure

To make sure that the list of available cpctl actions is up to date, enter the following command.
```
cpctl load
```
The cpctl load command retrieves all of the available actions that can be run on QRadar® Suite Software. The actions are cached to your local environment.
Log in to your Red Hat® OpenShift® Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
- Using a username and password.
```
oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
```
- Using a token.
```
oc login --token=<token> --server=<openshift_url>
```
Set the $CP4S_NAMESPACE environment variable by typing the following command, where <cp4s_namespace> is the namespace where you are installing QRadar Suite Software.

Important: If you install QRadar Suite Software in the all namespace mode, set the <cp4s_namespace> value as openshift-operators.
```
export CP4S_NAMESPACE=<cp4s_namespace>
```

Forward CouchDB to port 5984.

Get your CouchDB pod name by typing the following command.

oc get pods -n $CP4S_NAMESPACE | grep couch

In the following example output, the CouchDB pod name is c-default-couchdbcluster-m-0.

c-default-couchdbcluster-m-0                                   1/1     Running     0               14h
c-default-couchdbcluster-m-1                                   1/1     Running     0               127m
c-default-couchdbcluster-m-2                                   1/1     Running     0               127m

Forward CouchDB to port 5984 by typing the following command, where <pod_name> is your CouchDB podname.
```
oc port-forward <pod_name> 5984
```

Get your CouchDB login credentials.

Get your CouchDB username by typing the following command.

oc get secret couch-secret-default -ojsonpath='{.data.username}' -n $CP4S_NAMESPACE | base64 -d && echo

Get your CouchDB password by typing the following command.

oc get secret couch-secret-default -ojsonpath='{.data.password}' -n $CP4S_NAMESPACE | base64 -d && echo

Identify the databases that contain tombstone records.
1. In your browser, go to http://localhost:5984/_utils and log in with your CouchDB credentials.
2. Look for any databases that have an exclamation point (!) in the # of Docs column. An exclamation point (!) in the # of Docs column indicates tombstone records.
For each database that contains tombstone records, remove the tombstones by typing the following command, where <database> is the name of the database.
```
./cpctl remediation remove_tombstone --dbToTarget <database> --couchInstanceName default --token $(oc whoami -t)
```
For example, if the tis-connector-queue database contains tombstone records, type the following command.
```
./cpctl remediation remove_tombstone --dbToTarget tis-connector-queue --couchInstanceName default --token $(oc whoami -t)
```
Warning: Do not run the step again until it is complete.

To monitor the replication status, go to http://localhost:5984/_utils/#/replication in your browser and log in with your CouchDB credentials.

If you lose your connection to the cp-serviceability pod, type the following commands to continue monitoring.
```
oc rsh -n $CP4S_NAMESPACE deploy/cp-serviceability
```
```
pid=$(ps -ef | grep "/opt/app-root/bin/python3.9 /opt/app-root/bin/ansible-playbook /etc/ansible/playbooks/tis/remove_tombstone.yaml" | head -n 1  |awk '{print $2}') &&
cat /proc/$pid/fd/1
```