Troubleshooting Runbook Automation

Learn how to isolate and resolve problems with Runbook Automation.

SSH action output shows "Permission denied" error

Symptom: When attempting to run an SSH action script on some Linux target system, an error such as bash: /tmp/tmp.gXO9WCs7Zy: Permission denied is displayed in the script output area.

Cause: In order to run an SSH script on a Linux target system, Runbook Automation writes a temporary script file to the /tmp directory of the target system, and executes it from there. So the user on the target system that is configured to run the SSH script remotely does not only need to have write access to the /tmp directory , but also the /tmp directory itself must not be mounted with the noexec property.

Solution: Mount the /tmp directory without the noexec property, and verify that scripts can be run from within /tmp. Then retry running the SSH action.

If using the /tmp directory with write and execute access is prohibited in your organization then you can configure mktemp to use a different directory, for example, by generally setting the target system user's TMPDIR environment variable to an alternative directory path. See Creating an SSH integration for more details on the requirements for target systems of SSH connections.

SSH automation action output for PowerShell shows garbled multi-byte characters

Symptom: When running a PowerShell script on a Windows target system that is expected to print multi-byte or other non-ASCII characters, these characters are not properly displayed. Instead, these characters are replaced with question marks.

Cause: Runbook Automation expects the script output to be returned in UTF-8 format, but PowerShell is not configured to use UTF-8 as the default console output encoding.

Solution: Add an instruction to your PowerShell script code to set the console output encoding to UTF-8. For example, start your script with the following command:

$OutputEncoding = [Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8

See the Microsoft documentation for more details on character encoding in PowerShell.

Exported runbook includes password from referenced action in clear text (if any)

Symptom: If an HTTP action has parameters for basic authentication configured, and the action is referenced by a runbook, then exporting the runbook using the RBA API includes the password in clear text.

Cause: The exported runbook is intended to be self-contained, so when imported to the same or a different deployment, the runbook is usable again (which includes having the password available). However, there is no password encryption/decryption mechanism in place for this scenario.

Solution:

  • Avoid specifying passwords within actions.
  • Store the exported runbook file in a place that only persons with a need-to-know for the password(s) have access to.
  • Encrypt the exported file manually.

Runbook approval request not visible

Symptom: When the approval flow is enabled, and a runbook has been previously published, then a new request for approval for an updated draft of the runbook is not visible in the runbook library table. The expected approval status is "Approval pending". Instead, the runbook continues to be marked as "Approved".

Cause: The runbook library table displays the data from the last published version of the runbook, or from the draft runbook if it has never been published before. Therefore, approval requests for new draft runbooks are visible, but not for runbooks that have been previously published.

Solution: As author of the change, notify the approver which runbook needs to be reviewed and approved. As approver, open the runbook in "edit" or "preview" mode to see the latest changes. Directly communicate with the author if changes are needed. Once the runbook is ready to be published, disable the approval flow temporarily, and publish the runbook.

CouchDB reduce_overflow_error in runbook automation

Symptom: CouchDB logs might show the errors similar that follows:

2025-01-16T11:25:24.374927Z couchdb@c-example-couchdbcluster-m-2.c-example-couchdbcluster-m <0.16675786.0> -------- OS Process Error <0.10589392.0> :: {reduce_overflow_error,<<"Reduce output must shrink more rapidly: input size: 5820 output size: 5800">>}

Cause: The errors are occurred due to the outdated CouchDB design document that triggers an index update. When CouchDB asynchronously scans all design documents and their indexes, this function triggers reduce_overflow_error messages.

Solution:

Delete the outdated design document by using the following steps

  1. Run the following command to get the CouchDB secret name:
SECRET_NAME=$(oc get secret | grep couchdb-secret | awk '{ print $1 }')
  1. Retrieve CouchDB credentials:
COUCH_USER=$(oc get secret ${SECRET_NAME} -o json | grep "username" | cut -d : -f2 | cut -d '"' -f2 | base64 -d)
COUCH_PASSWORD=$(oc get secret ${SECRET_NAME} -o json | grep "password" | cut -d : -f2 | cut -d '"' -f2 | base64 -d)
  1. Get the design document revision:
DESIGN_REVISION=$(oc rsh c-example-couchdbcluster-m-0 curl -u "$COUCH_USER:$COUCH_PASSWORD" -k
https://localhost:6984/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/_design/admin_61dd14dc75089d48f79368e12343a7e59c4caa878e316c02bde30ab6477b15d6 | jq -r ._rev)
  • For NOI on premises, the database name is:
icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs
  • Example document ID:
_design/admin_61dd14dc75089d48f79368e12343a7e59c4caa878e316c02bde30ab6477b15d6

The document ID’s hex part (for example, 61dd14dc75089d48f79368e12343a7e59c4caa878e316c02bde30ab6477b15d6) is not fixed and might differ across environments. Always verify the actual document ID in your environment before you run the commands.

  1. Delete the outdated design document:
oc rsh c-example-couchdbcluster-m-0 curl -X DELETE -u "$COUCH_USER:$COUCH_PASSWORD" -k \
https://localhost:6984/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/_design/admin_61dd14dc75089d48f79368e12343a7e59c4caa878e316c02bde30ab6477b15d6?rev=$DESIGN_REVISION
  1. Run the following command to confirm the deletion:
oc rsh c-example-couchdbcluster-m-0 curl -X GET -u "$COUCH_USER:$COUCH_PASSWORD" -k \
https://localhost:6984/icp-63666439356237652d336263372d343030362d613461382d613733613739633731323535-rba-rbs/_design/admin_61dd14dc75089d48f79368e12343a7e59c4caa878e316c02bde30ab6477b15d6
  • Expected output:
{"error":"not_found","reason":"deleted"}