Resolving incidents with runbooks

Runbooks provide structured steps to help solve incidents.

Before you begin

To have runbooks available to use for your alerts, you must first define runbooks as described in the Managing runbooks and automations link at the end of this topic. The you must set up triggers where runbooks are associated with alerts as described in the Triggers link at the end of this topic.

The following is an example of how to use runbooks to address the alerts that form an incident, and as a result resolve the incident itself.

Procedure

  1. Go to the Incidents tab of the event management user interface.
  2. Go to My incidents and click Investigate to retrieve more information about the incident. The Resolution view displays suggested runbooks for the type of incident.
  3. In the Resolution view, click Menu overflow Menu overflow > Run next to the runbook you want to apply.

    If the runbook uses parameters, the parameter values are based on the alert policy, and depend on the alerts associated with the selected runbook:

    • If there is only one alert, or if there are multiple alerts all with the same parameter values, then the parameter values for the runbook are taken from a single alert, and the runbook is launched using those values.
    • If multiple alerts with different parameter values are correlated into an incident, each alert's parameter values are displayed. Select the value you want to run the runbook against and click Run.
      Note: It is best practice to only select non-group alerts to send the context to Runbook Automation. Unless group alerts are specifically what is wanted. Group alerts can be enriched to have the parameter values needed by the runbooks, but in most cases will lack this context. You can optionally set up your trigger to only associate runbooks with raw alerts, and avoid associating runbooks with group alerts. For example, by avoiding use of the summary field in the trigger conditions.

      The Runbook Automation UI is displayed where you can work with the runbook. For more information, see the Run a runbook link at the end of this topic.

    Tip: You can also apply runbooks associated with the alerts from the Alerts tab. Click the Alerts tab, and identify those alerts that have a big dot icon Big dot icon in their Runbook column. Click the big dot icon Big dot icon for the runbook that you want to execute. Parameters values for the runbook are derived from the alert, or you might be prompted to enter a value manually either if it requires information such as a user name, or if the runbook is set up to request the value at runtime.

    For more information about viewing the available runbooks, reviewing the runbooks that you have used to date, and running the runbooks, see the Library link at the end of this topic.

    Important: If you take an action against an incident that is not assigned to an owner, such as running a runbook manually, the incident status is automatically set to In progress, and the incident is assigned to you. The incident is also automatically assigned to you if you manually set the incident state to In progress. If you are a member of more than one group, then you must choose a group. You will be taking ownership of the incident and working to resolve it as a member of the selected group.

    Re-selecting No owner will clear any other status.

  4. The runbook completes and solves the underlying problem causing the incident. The alerts that formed the incident are then cleared, and in turn the incident is automatically set to resolved and closed.

What to do next

For information about creating and managing runbooks, see Runbooks.