Administering CouchDB data storage

Manage disk space used by runbook executions to avoid outages when the available disk space has been exhausted.

You can offload and delete historic runbook executions to reduce database footprint and increase database performance. Offloaded runbook executions are deleted from the production database and stored on a file system instead to preserve them for future audit purposes. They can be stored either in JSON or HTML format. A script for offloading runbook executions is included in the product and can be run by using the fully automated sample runbook Example: Automatically reduce amount of data in runbook data stores. Running this fully automated runbook has the negative side effect that the API key (which is contained in the parameters of the fully automated runbook) is stored and can be viewed within the runbook execution details. This description shows how you can avoid this limitation and still use the script that is available at the core of the fully automated runbook.

Runbook executions can be offloaded to store them somewhere else, in a different format (JSON or HTML), or to reduce the amount of data stored in the database. This is especially important if either the database is getting short of disk space, or a performance degradation that is caused by too much data is becoming noticeable. It is also a good practice to regularly offload runbook executions that exceed a certain age and are therefore no longer required on the production system.

When you offload and delete runbook execution records, the referenced action execution records are included in the offloading output and deleted as well. However, additional action execution records might remain for runbook executions that were already deleted separately, or were executed outside of a runbook context. You can use the following instructions on how to offload and delete those action execution records. Make sure to delete the action execution records only after the runbook execution records for the same time frame have been offloaded and deleted. Otherwise, the runbook execution records contain broken links to action execution records that have been deleted already.

Note:

Once runbook executions are offloaded and deleted, they are no longer accessible from the runbook history page and are not reflected in the aggregated statistics of their runbook anymore.
The size of the database does not shrink immediately after the runbook executions have been deleted, but only after the next CouchDB database compaction has been performed by CouchDB itself.

It is recommended to set up a scheduled task that offloads runbook execution data is regularly. For your planning you should consider the following aspects:

What is the maximum size of the persistent volume that hosts the database where the runbook execution data is stored? For example, 20 GB.
What is the average size of a single runbook execution in your environment? For simple runbooks, you can assume a value of about 15 KB per runbook execution.
How many runbooks are executed per day?

Create an initial schedule for offloading the runbooks based on these values, allowing some buffer for days with higher load. Observe the behavior for some time, and tune the frequency of the scheduled task or the parameters of the runbook to match your needs.

The offloading script must run on a Linux system that meets the following requirements:

Network access to the OpenShift or Linux® cluster where IBM Cloud Pak® for AIOps is installed.
The curl tool must be installed.
The jq tool must be installed. Starting with RHEL 8.0, jq can be installed by using the AppStream repository. On earlier RHEL versions, jq can be installed by following the instructions on https://stedolan.github.io/jq/.
Some free disk space must exist to store copies of the offloaded runbook executions. See the FOLDER parameter for more information.
The base64 encoding tool must be installed.

The offloading script requires several environment variables to be set:

API: The Runbook API URL as described in Automation (runbook) API. For example, set the value of the API parameter to https://cpd-cp4waiops.apps.mycomp.com/aiops/api/story-manager/rba/v1.
APIKEY_NAME: The user ID that you used to log in when creating an API key. As described in Accessing APIs, section Get access token by using your API key, step Generate your platform API key.
APIKEY_PW: The API key password that you created when following the instructions in Accessing APIs, section Get access token by using your API key, step Generate your platform API key.
FOLDER: The fully qualified path name of an existing directory on the Linux system that will be used for storing the offloaded runbooks.
MODE: Used to specify a format for the offloaded runbook executions. You can choose between html and json.
DELETE: Whether to delete the offloaded runbook executions from the runbook database after successful offload. Recommended to be set to false while performing early tests, but set to true once you are ready to delete documents and reduce the size of the database.
OLDER_THAN_MONTHS: A number of months to specify the timeframe for offloading runbook executions. Only runbook executions older than the specified number of months are offloaded. If you set this value to 0 then all runbook executions that were created before today will be offloaded. You can set this parameter to -1 and instead use the parameters AFTER and BEFORE if you need to have more granular control on the time period.
AFTER: Only offload execution records that are started after this timestamp. You can specify a date like 2022-03-31 or a timestamp in the moment.js format, for example, 2022-03-31T12:59:59. If you want to use this parameter, the parameter OLDER_THAN_MONTHS must be set to the empty string.
BEFORE: Only offload execution records that are started before this timestamp. You can specify a date like 2022-04-04 or a timestamp in the moment.js format, for example, 2022-04-04T12:59:59. If you want to use this parameter, the parameter OLDER_THAN_MONTHS must be set to the empty string.

1. One-time set up to create the script

Create an API key as described in Accessing APIs, section Get access token by using your API key, step Generate your platform API key.
If it does not exist already, create the SSH integration as described in Creating SSH integrations. You do not need to define a jumpserver and you do not need to copy the SSH key to any target system for the purpose of this procedure.
Load the sample runbooks as described in Load and reset example runbooks.
Switch to the Actions tab.
Select the action Example: Reduce amount of data in runbook data stores and click Edit.
Copy the content of the Script field to the clipboard.
Log in to the Linux system and change to the directory where you want the script to be stored.
Create a file called "offloadRunbookExecutions.sh" and paste the content of the clipboard into this file.
Make the file "offloadRunbookExecutions.sh" executable, for example, run the following command:
```
chmod a+x offloadRunbookExecutions.sh
```

Create a file called "offload.config" and paste the following content into this file:

export API="https://cpd-cp4waiops.apps.mycomp.com/aiops/api/story-manager/rba/v1"
export APIKEY_NAME="<api_key_user_id>"
export APIKEY_PW="<api_key_password>"
export FOLDER=$(pwd)
export MODE=json
export DELETE=false
export OLDER_THAN_MONTHS=-1
export AFTER="2022-03-01"
export BEFORE="2022-04-13T12:00:00"
#export CURL_OPTIONS="-k"

Adjust the values of the environment variables as needed (see the variable descriptions listed earlier) and save the file.

2. Test the script

Run the following command to load the environment variables into the current shell environment:
```
. ./offload.config
```
Run the script:
```
./offloadRunbookExecutions.sh
```
If the action fails and you see the error curl: \(60\) SSL certificate problem: self-signed certificate in certificate chain, you can either replace the IBM Netcool Operations Insight server certificate with a trusted certificate, or enable the environment variable CURL\_OPTIONS="-k" to ignore the certificate errors.
If the action fails and you see the following error:
```
ARBD0420E: Could not fetch list of runbook instances:
curl: (22) The requested URL returned error: 500 Internal Server Error
```
Then reduce the length of the time interval for offloading execution records (for example, by increasing the value of OLDER_THAN_MONTHS) and try again.
Review the results that were created by the script. If needed adjust the environment variables and repeat the procedure until you are fine with the results.

3. Schedule the script

Convert this script command to the format that is expected by your scheduling tool, and set up a schedule to regularly offload the runbook data.
Make sure that the DELETE environment variable is set to true once you are finished with the testing of the script and the schedule, and when you are ready to delete the offloaded runbook execution instances from the database.

Offloading and deleting remaining action execution records

Once you have deleted the runbook execution records for a particular time frame you should also check if there are remaining action execution records (also called action instances) for that time frame, and delete them as well.

Step 1: Prepare and verify the API URL and the credentials Adjust the values of the environment variables as needed (see the variable descriptions listed earlier) and run the following commands:

export API="https://cpd-cp4waiops.apps.mycomp.com/aiops/api/story-manager/rba/v1"
export APIKEY_NAME="<api_key_user_id>"
export APIKEY_PW="<api_key_password>"
export CURL_AUTHENTICATION=$(echo -n "${APIKEY_NAME}:${APIKEY_PW}" | base64 -w 0 | echo "Authorization: ZenApiKey $(cat -)")
export CURL_OPTIONS="-ks"
curl "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/status"

The command should run without any authentication errors, and the output should look similar to this (for the purpose of this task, you can ignore the value of the "connection" property, in particular any potential "errors" and "warnings" within the respective properties):

{"rbs":0,"as":0,"db":0,"connection":{"AWX":0,"IMPACT":2,"SCRIPT":0,"GITHUB":2},"errors":[],"warnings":[]}

Step 2: Get the remaining action execution records Adjust the "to" time stamp in the following command and run it to retrieve a list of the records that were created up to that date, and store their IDs in an environment variable:

curl "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/actioninstances?from=2010-01-01&to=2022-12-01" > actionInstances.json
export ACTION_INST_IDS=`cat actionInstances.json | jq '.[]' | jq -r '.[]._actionInstanceId'`

The records in actionInstances.json do not contain the actual output of each action. If you want to preserve the output for each of the actions, run the following command:

for actionInstId in ${ACTION_INST_IDS}; do echo "Getting details for ${actionInstId}"; curlOutput=$(curl -X GET "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/actioninstances/${actionInstId}" > ${actionInstId}.json); if (test ${#curlOutput} -gt 0); then echo ${curlOutput}; fi; done

Store the each of these action execution record files as appropriate.

Step 3: Delete the remaining action execution records Run the following command to delete the records:

for actionInstId in ${ACTION_INST_IDS}; do echo "Deleting ${actionInstId}"; curlOutput=$(curl -X DELETE "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/actioninstances/${actionInstId}"); if (test ${#curlOutput} -gt 0); then echo ${curlOutput}; fi; done