Administering CouchDB data storage

Manage disk space used by runbook and action activities to avoid outages when the available disk space has been exhausted.

You can offload and delete historic runbook and action activities to reduce database footprint and increase database performance. Offloaded activities are deleted from the production database and stored on a file system instead to preserve them for future audit purposes. A script for offloading activities is included in the product and can be run by using the fully automated sample runbook Example: Offload runbook and automation action activities. Running this fully automated runbook has the negative side effect that the API key (which is contained in the parameters of the fully automated runbook) is stored and can be viewed within the runbook activity details. This description shows how you can avoid this limitation and still use the script that is available at the core of the fully automated runbook.

Runbook and action activities can be offloaded to store them on filesystem in JSON format or to reduce the amount of data stored in the database. This is especially important if either the database is getting short of disk space, or a performance degradation that is caused by too much data is becoming noticeable. It is also a good practice to regularly offload activities that exceed a certain age and are therefore no longer required on the production system.

When you offload runbook activities, the referenced action activities are offloaded as well in separate files. Files with offloaded runbook activities use the prefix ri- while offloaded action activities use the prefix ai-. Action activities that are not referenced in runbook activities but match the selected time frame are offloaded as the last step of the offloading script.

Note:
  • Once runbook activities are offloaded and deleted, they are no longer accessible from the runbook history page and are not reflected in the aggregated statistics of their runbook anymore.
  • The size of the database does not shrink immediately after the activities have been deleted, but only after the next CouchDB database compaction has been performed by CouchDB itself.

It is recommended to set up a scheduled task that offloads activities regularly. For your planning you should consider the following aspects:

  • What is the maximum size of the persistent volume that hosts the database where the runbook activities are stored? For example, 20 GB.
  • What is the average size of a single runbook activity in your environment? For simple runbooks, you can assume a value of about 15 KB per activity.
  • How many runbooks are executed per day?

Create an initial schedule for offloading the activities based on these values, allowing some buffer for days with higher load. Observe the behavior for some time, and tune the frequency of the scheduled task or the parameters of the runbook to match your needs.

The offloading script must run on a Linux system that meets the following requirements:

  • Network access to the OpenShift or Linux cluster where IBM Cloud Pak for AIOps is installed.
  • Node.js must be installed.
  • Some free disk space must exist to store the offloaded activities. See the FOLDER parameter for more information.

The offloading script requires several environment variables to be set:

  • OFFLOAD_API: The Runbook API URL as described in Automation (runbook) API. For example, set the value of the API parameter to https://cpd-cp4waiops.apps.mycluster.example.com/aiops/api/story-manager/rba/v1.
  • OFFLOAD_APIKEY_NAME: The user ID that you used to log in when creating an API key. As described in Accessing APIs, section Get access token by using your API key, step Generate your platform API key.
  • OFFLOAD_APIKEY_PW: The API key password that you created when following the instructions in Accessing APIs, section Get access token by using your API key, step Generate your platform API key.

Depending on your certificate setup you may need to use either of these environment variables:

  • NODE_EXTRA_CA_CERTS: Path to a file containing the CA certificate(s) in PEM format.
  • OFFLOAD_REJECT_UNAUTHORIZED: Can be set to false to bypass certificate validation.

The following optional environment variables can be set in order to adjust the offload according to your needs:

  • OFFLOAD_OLDER_THAN_DAYS: Only offload activities older than the given number of days. Will be ignored if OFFLOAD_BEFORE is specified. (defaults to 60)
  • OFFLOAD_AFTER: Only offload activities started after timestamp. Use ISO format, e.g. 2024-05-13 or 2024-05-13T09:00:00Z.
  • OFFLOAD_BEFORE: Only offload activities started before timestamp. Use ISO format, e.g. 2025-02-21 or 2025-02-21T18:00:00Z. Will overrule OFFLOAD_OLDER_THAN_DAYS.
  • OFFLOAD_RUNBOOK: Only offload activities associated to the specified runbook ID. Use a comma-separated list for multiple runbooks.
  • OFFLOAD_DOWNLOAD: If set to false, activities will not be downloaded. (defaults to true)
  • OFFLOAD_DELETE: If set to false, activities will not be deleted. (defaults to true)
  • OFFLOAD_STANDALONE_ACTIONS: If set to false, records of standalone automation actions that were run outside of runbooks will not be offloaded. If OFFLOAD_RUNBOOK is specified or OFFLOAD_DELETE is set to false, standalone automation actions will never be offloaded. (otherwise defaults to true)
  • OFFLOAD_FOLDER: The fully qualified path name of an existing directory on the Linux system that will be used for storing the offloaded runbook activities. (defaults to the current working directory)

Additionally these environment variables are supported to adjust the offloading script itself but should be left untouched in most situations:

  • OFFLOAD_SOCKETS: Number of sockets to be used for connecting to runbook API. (defaults to 50)
  • OFFLOAD_PROGRESS_BAR: Set to true to show progress bar. (defaults to false)
  • OFFLOAD_LOG_LEVEL: Configure the amount of logging. Log level can be one of trace, debug, info, warn, error, fatal. (defaults to info)

1. One-time set up to create the script

  1. Create an API key as described in Accessing APIs, section Get access token by using your API key, step Generate your platform API key.

  2. If it does not exist already, create the SSH integration as described in Creating SSH integrations. You do not need to define a jumpserver and you do not need to copy the SSH key to any target system for the purpose of this procedure.

  3. Load the sample runbooks as described in Load and reset example runbooks.

  4. Switch to the Actions tab.

  5. Select the action Example: Offload runbook and automation action activities and click Edit.

  6. Copy the content of the Script field to the clipboard.

  7. Log in to the Linux system and change to the directory where you want the script to be stored.

  8. Create a file called "offloadRunbookExecutions.js" and paste the content of the clipboard into this file.

  9. Make the file "offloadRunbookExecutions.js" executable, for example, run the following command:

    chmod a+x offloadRunbookExecutions.js

  10. Create a file called "offload.config" and paste the following content into this file:

    export OFFLOAD_API="https://cpd-cp4waiops.apps.mycluster.example.com/aiops/api/story-manager/rba/v1"
export OFFLOAD_APIKEY_NAME="<api_key_user_id>"
export OFFLOAD_APIKEY_PW="<api_key_password>"
export OFFLOAD_FOLDER=$(pwd)
export OFFLOAD_DOWNLOAD=true
export OFFLOAD_DELETE=false
#export OFFLOAD_OLDER_THAN_DAYS=60
export OFFLOAD_AFTER="2022-03-01"
export OFFLOAD_BEFORE="2022-04-13"
#export OFFLOAD_REJECT_UNAUTHORIZED="false"

  11. Adjust the values of the environment variables as needed (see the variable descriptions listed earlier) and save the file.

2. Test the script

  1. Run the following command to load the environment variables into the current shell environment:

    . ./offload.config

  2. Run the script:

    ./offloadRunbookExecutions.js

  3. If the action fails and you see the error curl: \(60\) SSL certificate problem: self-signed certificate in certificate chain, you can either replace the IBM CloudPak for AIOps server certificate with a trusted certificate, set the environment variable OFFLOAD_REJECT_UNAUTHORIZED="false" to ignore the certificate errors, or use the environment variable NODE_EXTRA_CA_CERTS to configure the correct CA.

  4. Review the results that were created by the script. If needed adjust the environment variables and repeat the procedure until you are fine with the results.

3. Schedule the script

  1. Convert this script command to the format that is expected by your scheduling tool, and set up a schedule to regularly offload the runbook data.

  2. Make sure that the OFFLOAD_DELETE environment variable is set to true once you are finished with the testing of the script and the schedule, and when you are ready to delete the offloaded runbook execution instances from the database.