Removing tombstone records from CouchDB databases

Remove tombstone records from CouchDB databases to speed up the backup and restore process.

Before you begin

Install the command-line interface (CLI) utility cpctl from the cp-serviceability pod. For more information, see Installing the cpctl utility.

Procedure

  1. To ensure that the list of available cpctl actions is up to date, enter the following command.
    cpctl load
    The cpctl load command retrieves all of the available actions that can be run on QRadar® Suite Software. The actions are cached to your local environment.
  2. Log in to your Red Hat® OpenShift® Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  3. Set the $CP4S_NAMESPACE environment variable by typing the following command, where <cp4s_namespace> is the namespace where you are installing QRadar Suite Software.
    Important: If you installed QRadar Suite Software in all namespace mode, then set the <cp4s_namespace> value as openshift-operators.
    export CP4S_NAMESPACE=<cp4s_namespace>
  4. Forward CouchDB to port 5984.
    1. Get your CouchDB pod name by typing the following command.
      oc get pods -n $CP4S_NAMESPACE | grep couch

      In the following example output, the CouchDB pod name is c-default-couchdbcluster-m-0.

      c-default-couchdbcluster-m-0                                   1/1     Running     0               14h
      c-default-couchdbcluster-m-1                                   1/1     Running     0               127m
      c-default-couchdbcluster-m-2                                   1/1     Running     0               127m
    2. Forward CouchDB to port 5984 by typing the following command, where <pod_name> is your CouchDB podname.
      oc port-forward <pod_name> 5984
  5. Get your CouchDB login credentials.
    1. Get your CouchDB username by typing the following command.
      oc get secret couch-secret-default -ojsonpath='{.data.username}' -n $CP4S_NAMESPACE | base64 -d && echo
    2. Get your CouchDB password by typing the following command.
      oc get secret couch-secret-default -ojsonpath='{.data.password}' -n $CP4S_NAMESPACE | base64 -d && echo
  6. Identify the databases that contain tombstone records.
    1. In your browser, go to http://localhost:5984/_utils and log in with your CouchDB credentials.
    2. Look for any databases that have an exclamation point (!) in the # of Docs column. An exclamation point (!) in the # of Docs column indicates tombstone records.
  7. For each database that contains tombstone records, remove the tombstones by typing the following command, where <database> is the name of the database.
    ./cpctl remediation remove_tombstone --dbToTarget <database> --couchInstanceName default --token $(oc whoami -t)
    For example, if the tis-connector-queue database contains tombstone records, type the following command.
    ./cpctl remediation remove_tombstone --dbToTarget tis-connector-queue --couchInstanceName default --token $(oc whoami -t)
    Warning: Do not run the step again until it is complete.

    To monitor the replication status, go to http://localhost:5984/_utils/#/replication in your browser and log in with your CouchDB credentials.

    If you lose your connection to the cp-serviceability pod, type the following commands to continue monitoring.

    oc rsh -n $CP4S_NAMESPACE deploy/cp-serviceability
    pid=$(ps -ef | grep "/opt/app-root/bin/python3.9 /opt/app-root/bin/ansible-playbook /etc/ansible/playbooks/tis/remove_tombstone.yaml" | head -n 1  |awk '{print $2}') &&
    cat /proc/$pid/fd/1