Smart Alerts for infrastructure

To define the scope of the alerting, complete the following steps in the Scope section:

1. Select a metric from the Metric list by using one of the following options:

   • On the List tab, search the metrics list by using keywords.
   • On the Regex tab, define the metric scope through regular expressions.

   

2. Define the aggregation as follows:

   • Time aggregation: Select the preferred cross-time aggregation. This method combines data-points into a single bucket.
   • Cross-series aggregations: To sum the buckets across the data series, set Use SUM for cross-series aggregation to on. Usually, the cross-series aggregation is the same as the cross-time aggregation.

3. Select one of the following alerting methods:

   • Custom aggregation: To group the metric data based on specific custom tags that you define, click Custom aggregation. This method is enabled by default.

     

     Use custom aggregation in the following scenarios:

     • Monitoring aggregated metrics across multiple entities (for example, average CPU usage across all hosts in a specific zone or region).
     • Reducing alert noise by grouping similar entities together.
     • Analyzing overall trends and patterns rather than individual entity behavior.
     • Tracking overall capacity or resource utilization across a cluster or environment.

     Example: Monitor average CPU load across all production hosts grouped by availability zone. You receive a single alert when the average CPU load in a zone exceeds the threshold, rather than individual alerts for each host.

   • Per entity alerting: To monitor each metric individually and trigger alerts for each individual entity, click Per entity alerting.

     

     Use per entity alerting in the following scenarios:

     • Identifying and responding to issues on specific individual entities (for example, a particular host, container, or database instance).
     • Providing separate attention and remediation for each entity.
     • Monitoring critical resources where individual failures matter.
     • Tracking entity-specific SLAs or performance requirements.

     Example: Monitor CPU usage on each production database server individually. You receive a separate alert for each database server that exceeds the CPU threshold, allowing you to pinpoint and address the specific problematic instance.

     Per entity alerting supports grouping by metric-related tags such as metricId or custom tags such as device, mountpoint, and state. This grouping enables individual alerts for each unique metric variant when using metric patterns or metrics with custom tags.

     

4. Add filters to further narrow the scope.

5. Group the metric results with the available grouping tags. You can use up to 5 tags for grouping metrics.

When you set up a Smart Alert for infrastructure, you can choose to use static or adaptive thresholds.

For the alert that is triggered, you can add more conditions in the Time threshold section on when the defined threshold for the selected metric is violated.

The following typical conditions, often used in practice, are as follows:

• Persistence over time: Select a time window and the number of consecutive times of violation. You receive an alert when the metric violates a defined threshold over the defined time window.

You can set up forecast alerting to receive proactive alerts that help you address potential issues before they affect your system. For example, you might want to receive an alert when a disk is nearing full capacity or when the memory usage of a process is close to the container limits, which might indicate a memory leak. With Instana, you can configure alerts based on metric forecasts.

When you opt for the forecast alerting feature, configure the following two time windows:

1. Historical data time window: This window specifies the time frame of the metric that is used to fit the model for forecasting. It allows you to specify whether you care about the short-term or long-term trends of the selected metric.
2. Forecasted time window: This window specifies the time frame of the linear forecast that is used for alerting. Larger forecasted time values increase the chance of false alerts.

The following image illustrates these time windows for two example metrics and their linear forecast with different outcome in alerting:

An alert is triggered when either the metric value or the metric forecast is exceeding the threshold based on the configured rule.

You can configure different alert channels for different severities in Smart Alerts for Infrastructure. To add alert channels, complete the following steps:

1. Click Select Alert Channel.
2. From the list of preconfigured channels, select the channels from which you want to receive the alerts.

If a threshold value is set for warning and critical severities, you can set the alert channels for each severity. If a threshold value is set for both severities, all the alert channels are selected for the warning severity by default.

Alert channels with both severities configured:

If a threshold value is set only for one severity, the severity is displayed for every alert channel as the alert level.

Alert channels with one severity configured:

For more information about creating channels, see Alert Channels.

In this section, you can optionally configure various properties related to the alerts created by using the Smart Alert configuration.

To include an additional payload that is relevant to you in alert notifications for specific alert configuration that is sent by Instana, click Add Row in the Custom Payloads section.

Both global custom payloads and alert-specific custom payloads are included in alert notifications if applicable, but the alert-specific configuration takes precedence over the global configuration. As a result, if you use the same key, the value of the global custom payload field is overridden by the alert-specific one.

The following image shows globally defined custom payloads that are used in the alert configuration:

For information about global custom payloads, see Configure Custom Payload Globally.

As of now, public preview does not support dynamic global custom payloads.

Custom aggregation groups metric data based on tags you define and evaluates the aggregated metric against the threshold. This approach is useful for monitoring overall trends across multiple entities and reducing alert noise. You receive a single alert when the aggregated metric violates the threshold.

Per entity alerting monitors each entity individually and triggers separate alerts for each entity that violates the threshold. This approach is useful when you need to identify and respond to issues on specific entities. You receive individual alerts for each affected entity.

Choose the alerting method based on whether you want to monitor aggregated behavior (custom aggregation) or individual entity behavior (per entity alerting).

An alert is triggered when either the current metric value or the forecasted value exceeds the threshold. This proactive approach helps you to address potential issues before they impact your system, such as disk space running out or memory leaks approaching container limits.

Larger forecasted time windows increase the chance of false alerts, so balance proactive alerting with alert accuracy based on your specific use case.