Administering data storage

Manage disk space used by runbook executions to avoid outages when the available disk space has been exhausted.

Before you begin

You can offload and delete historic runbook executions to reduce database footprint and increase database performance. Offloaded runbook executions are deleted from the production database and stored on a file system instead to preserve them for future audit purposes. They can be stored either in JSON or HTML format. A script for offloading runbook executions is included in the product and can be run by using the fully automated sample runbook "Example: Automatically reduce amount of data in runbook data stores". Running this fully automated runbook has the negative side effect that the API key (which is contained in the parameters of the fully automated runbook) is stored and can be viewed within the runbook execution details. This description shows how you can avoid this limitation and still use the script that is available at the core of the fully automated runbook.

About this task

Runbook executions can be offloaded to store them somewhere else, in a different format (JSON or HTML), or to reduce the amount of data stored in the database. This is especially important if either the database is getting short of disk space, or a performance degradation that is caused by too much data is becoming noticeable. It is also a good practice to regularly offload runbook executions that exceed a certain age and are therefore no longer required on the production system.

When you offload and delete runbook execution records, the referenced automation execution records are included in the offloading output and deleted as well. However, additional automation execution records might remain for runbook executions that were already deleted separately, or were executed outside of a runbook context. You can find instructions below on how to offload and delete those automation execution records as well. Make sure to delete the automation execution records only after the runbook execution records for the same time frame have been offloaded and deleted. Otherwise, the runbook execution records contain broken links to automation execution records that have been deleted already.

Note:

Once runbook executions are offloaded and deleted, they are no longer accessible from the runbook history page and are not reflected in the aggregated statistics of their runbook anymore.
The size of the database does not shrink immediately after the runbook executions have been deleted, but only after the next CouchDB database compaction has been performed by CouchDB itself. For more information about CouchDB, see CouchDB.

It is recommended to set up a scheduled task that offloads runbook execution data is regularly. For your planning you should consider the following aspects:

What is the maximum size of the persistent volume that hosts the database where the runbook execution data is stored? For example, 20 GB.
What is the average size of a single runbook execution in your environment? For simple runbooks, you can assume a value of about 15 KB per runbook execution.
How many runbooks are executed per day?

Create an initial schedule for offloading the runbooks based on these values, allowing some buffer for days with higher load. Observe the behavior for some time, and tune the frequency of the scheduled task or the parameters of the runbook to match your needs.

The offloading script must run on a Linux® system that meets the following requirements:

Network access to the Red Hat® OpenShift® cluster where Netcool® Operations Insight® is installed.
The curl tool must be installed.
The jq tool must be installed. Starting with RHEL 8.0, jq can be installed by using the AppStream repository. On earlier RHEL versions, jq can be installed by following the instructions on https://stedolan.github.io/jq/.
Some free disk space must exist to store copies of the offloaded runbook executions. See the FOLDER parameter for more information.

The offloading script requires several environment variables to be set:

API: The Runbook API URL as described in Runbooks API. For example, set the value of the API parameter to https://netcool-noi.apps.cp.mycomp.com/api/v1/rba.
APIKEY_NAME: The name of the API key that was created by using the instructions in API Keys.
APIKEY_PW: The password of the API key that was created by using the instructions in API Keys.
FOLDER: The fully qualified path name of an existing directory on the Linux system that will be used for storing the offloaded runbooks.
MODE: Used to specify a format for the offloaded runbook executions. You can choose between html and json.
DELETE: Whether to delete the offloaded runbook executions from the runbook database after successful offload. Recommended to be set to false while performing early tests, but set to true once you are ready to delete documents and reduce the size of the database.
OLDER_THAN_MONTHS: A number of months to specify the timeframe for offloading runbook executions. Only runbook executions older than the specified number of months are offloaded. If you set this value to 0 then all runbook executions that were created before today will be offloaded. You can set this parameter to -1 and instead use the parameters AFTER and BEFORE if you need to have more granular control on the time period.
AFTER: Only offload execution records that are started after this timestamp. You can specify a date like 2022-03-31 or a timestamp in the moment.js format, for example, 2022-03-31T12:59:59. If you want to use this parameter, the parameter OLDER_THAN_MONTHS must be set to the empty string.
BEFORE: Only offload execution records that are started before this timestamp. You can specify a date like 2022-04-04 or a timestamp in the moment.js format, for example, 2022-04-04T12:59:59. If you want to use this parameter, the parameter OLDER_THAN_MONTHS must be set to the empty string.

Procedure

Step 1: One-time set up to create the script
1. Create an API key for managing runbooks as described in API Keys.
2. Configure the script connection as described in Configuring an SSH script automation provider.
3. Load the sample runbooks as described in Load and reset example runbooks.
4. Switch to the Automations tab.
5. Select the automation Example: Reduce amount of data in runbook data stores and click Edit.
6. Copy the content of the Script field to the clipboard.
7. Log in to the Linux system and change to the directory where you want the script to be stored.
8. Create a file called "offloadRunbookExecutions.sh" and paste the content of the clipboard into this file.
9. Make the file "offloadRunbookExecutions.sh" executable, for example, run the following command:
```
chmod a+x offloadRunbookExecutions.sh
```
10. Create a file called "offload.config" and paste the following content into this file:
```
export API="https://netcool-noi.apps.cp.mycomp.com/api/v1/rba"
export APIKEY_NAME="<api_key_name>"
export APIKEY_PW="<api_key_password>"
export FOLDER=/root/noi164offload/scripted
export MODE=json
export DELETE=false
export OLDER_THAN_MONTHS=-1
export AFTER="2022-03-01"
export BEFORE="2022-04-04T12:00:00"
#export CURL_OPTIONS="-k"
```
11. Adjust the values of the environment variables as needed (see the previous descriptions) and save the file.
Step 2: Test the script
1. Run the following command to load the environment variables into the current shell environment:
```
. ./offload.config
```
2. Run the script:
```
./offloadRunbookExecutions.sh
```
3. If the automation fails and you see the error "curl: (60) SSL certificate problem: self-signed certificate in certificate chain" you can either replace the NOI server certificate with a trusted certificate, or enable the environment variable CURL_OPTIONS="-k" to ignore the certificate errors.
4. If the automation fails and you see the following error:
```
ARBD0420E: Could not fetch list of runbook instances:
curl: (22) The requested URL returned error: 500 Internal Server Error
```
  Then reduce the length of the time interval for offloading execution records (for example, by increasing the value of OLDER_THAN_MONTHS) and try again.
5. Review the results that were created by the script. If needed adjust the environment variables and repeat the procedure until you are fine with the results.
Step 3: Schedule the script
1. Convert this script command to the format that is expected by your scheduling tool, and set up a schedule to regularly offload the runbook data.
2. Make sure that the DELETE environment variable is set to true once you are finished with the testing of the script and the schedule, and when you are ready to delete the offloaded runbook execution instances from the database.

Offloading and deleting remaining automation execution records

Edit online

About this task

Once you have deleted the runbook execution records for a particular time frame, you should also check if there are remaining automation execution records (also called automation instances) for that time frame, and delete them as well.

Procedure

Step 1: Prepare and verify the API URL and the credentials
Adjust the values of the environment variables as needed (see the variable descriptions listed earlier) and run the following commands:
```
export API="https://netcool-noi.apps.cp.mycomp.com/api/v1/rba"
export APIKEY_NAME="<api_key_name>"
export APIKEY_PW="<api_key_password>"
export CURL_AUTHENTICATION=$(echo -n "${APIKEY_NAME}:${APIKEY_PW}" | base64 -w 0 | echo "Authorization: Basic $(cat -)")
export CURL_OPTIONS="-ks"
curl "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/status"
```
The command should run without any authentication errors, and the output should look similar to the following example (for the purpose of this task, you can ignore the value of the "connection" property, in particular any potential "errors" and "warnings" within the respective properties):
```
{"rbs":0,"as":0,"db":0,"connection":{"AWX":0,"IMPACT":2,"SCRIPT":0,"GITHUB":2},"errors":[],"warnings":[]}
```

Step 2: Get the remaining automation execution records

Adjust the "to" time stamp in the following command and run it to retrieve a list of the records that were created up to that date, and store their IDs in an environment variable:

curl "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/actioninstances?from=2010-01-01&to=2022-12-01" > actionInstances.json
export ACTION_INST_IDS=`cat actionInstances.json | jq '.[]' | jq -r '.[]._actionInstanceId'`

The records in actionInstances.json do not contain the actual output of each action. If you want to preserve the output for each of the actions, run the following command:

for actionInstId in ${ACTION_INST_IDS}; do echo "Getting details for ${actionInstId}"; curlOutput=$(curl -X GET "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/actioninstances/${actionInstId}" > ${actionInstId}.json); if (test ${#curlOutput} -gt 0); then echo ${curlOutput}; fi; done

Store the each of these automation execution record files as appropriate.

Step 3: Delete the remaining automation execution records

Run the following command to delete the records:

for actionInstId in ${ACTION_INST_IDS}; do echo "Deleting ${actionInstId}"; curlOutput=$(curl -X DELETE "${CURL_OPTIONS}" -H "${CURL_AUTHENTICATION}" "${API}/actioninstances/${actionInstId}"); if (test ${#curlOutput} -gt 0); then echo ${curlOutput}; fi; done