Smart Alerts for infrastructure

To define the scope of the alerting, complete the following steps in the Scope section:

1. Select the Entity type.

   

2. Select the Alerting Method. You can choose either of the following evaluation methods for the metric:
   • Per group: Group defined metrics based on specific custom tags that you define, and create one event to track all metrics. This method is enabled by default. Use the method in the following scenarios:
     • Monitoring aggregated metrics across multiple entities (for example, average CPU usage across all hosts in a specific zone or region).
     • Reducing alert noise by grouping similar entities together.
     • Analyzing overall trends and patterns rather than individual entity behavior.
     • Tracking overall capacity or resource utilization across a cluster or environment.

     Example: Monitor average CPU load across all production hosts grouped by availability zone. You receive a single alert when the average CPU load in a zone exceeds the threshold, rather than individual alerts for each host.

     

   • Per entity: Create one event for each metric that violated the threshold to monitor each metric individually. Use per entity alerting in the following scenarios:
     • Identifying and responding to issues on specific individual entities (for example, a particular host, container, or database instance).
     • Providing separate attention and remediation for each entity.
     • Monitoring critical resources where individual failures matter.
     • Tracking entity-specific SLAs or performance requirements.

     Example: Monitor CPU usage on each production database server individually. You receive a separate alert for each database server that exceeds the CPU threshold, allowing you to pinpoint and address the specific problematic instance.

     

     Per entity alerting supports grouping by metric-related tags such as metricId or custom tags such as device, mountpoint, and state. This grouping enables individual alerts for each unique metric variant when using metric patterns or metrics with custom tags.

3. (Optional) Add any required Filters based on the chosen entity type to further narrow the alert scope.

   

4. (Optional) Define any Grouping for the metrics: Define how the metric results are to be organized by using grouping tags. You can use up to five tags for grouping metrics. Groups can be used as variables when generating the Smart Alert's name and description.

   

1. Click Add metrics. The Add metrics dialog opens.

   

2. Click Select metric and select the metric type by using one of the following options:
   • On the List tab, search the metrics list by using keywords.
   • On the Regex tab, define the metric scope through regular expressions

   

3. Click Select metric filter and select a metric tag filter to narrow down the metric value to monitor.
4. Click Select metric group and select the metric grouping tags to define how the metrics are organized.
5. Define the aggregation by using the following options:
   • Cross-time aggregation: Select the preferred cross-time aggregation. This method combines data-points into a single bucket.
   • Cross-series aggregations: To sum the buckets across the data series, enable Use SUM for cross-series aggregation. Usually, the cross-series aggregation is the same as the cross-time aggregation.

   

6. Define the Threshold by selecting whether to use Static threshold or Adaptive threshold.
   • Static threshold: Static thresholds do not change over time. You can set them when you create or modify the Smart Alert. You can provide different thresholds for the warning and critical severities. A static threshold might stop being relevant after the underlying metric changes significantly. In response, you can manually adjust or recalculate the threshold at any point in time. You can select a threshold operator to define the threshold condition.
     Static thresholds work best in the following situations:

     • Regardless of seasonality of the underlying metric, the metric must not exceed or fall below a constant value.
     • The underlying metric is seasonal, and therefore different thresholds exist depending on the time of day or week. However, these thresholds themselves don't change over time. Gradual changes to these thresholds over long periods of time are undesirable.
   • Adaptive threshold: Adaptive thresholds continuously evolve and adjust themselves with new data that Instana observes. This means that the threshold continuously accounts for seasonal changes to the underlying metric without any human intervention. You can provide different thresholds for the warning and critical severities. For more information, see the Adaptive thresholds.
     Adaptive thresholds work best in the following situations:

     • The underlying metric is not seasonal. The threshold is expected to gradually change over time, but any sudden deviation from this trend is undesirable.
     • The underlying metric is seasonal and different thresholds exist for different times of the day or week. The thresholds themselves are expected to gradually change over time, but any sudden deviation from this trend is undesirable.
     Adaptive threshold requirements:

     • The adaptive threshold requires at least 6 hours of continuous metric data. If this requirement is not fulfilled, you can still create the Smart Alert. Issue detection and alerting start working as soon as the data requirement is met to initialize the used model.

     

7. Review the alert preview.

   After you define the scope and threshold, the chart is plotted based on the historical data against the metrics. A maximum of 7 days of historical data is available for visualization in the chart. You can switch between the last 24 hours to 7 days of historical data to visualize the historical variations of metric data.

   Based on the historical data and threshold conditions, the chart displays the alerts that the current threshold value might trigger.

   

   If you select any grouping options, the grouping results might appear as a table following the chart. To analyze the metric data trends in the chart against each grouping, select the respective rows in the table.

   

8. k Add metric to return to the Configure alert dialog.
9. Set the Metric granularity to define how each metric value is calculated during the specified time bucket.
10. Set the Failure threshold to define the number of consecutive failures in the time window required to trigger an alert.
11. (Optional) configure the Grace period to set how long an issue stays open after the violation threshold is no longer met. Once this time elapses, the issue is automatically closed.
12. (Optional) Configure Forecast alerting to get alerted in advance: You can set up forecast alerting to receive proactive alerts based on the trend of historical data to help you anticipate and address potential issues before they affect yours system.

For example, you might want to receive an alert when a disk is nearing full capacity or when the memory usage of a process is close to the container limits, which might indicate a memory leak. With Instana, you can configure alerts based on metric forecasts.

If a threshold value is set for warning and critical severities, you can set the alert channels for each severity. If a threshold value is set for both severities, all the alert channels are selected for the warning severity by default.

Alert channels with both severities configured:

If a threshold value is set only for one severity, the severity is displayed for every alert channel as the alert level.

Alert channels with one severity configured:

For more information about creating channels, see Alert channels.

In this section, you can optionally configure additional alert properties that are related to the alerts that are created by using the Smart Alert configuration. You can also add custom payloads.

• Title: Instana suggests a default title based on the used blueprint type and related configuration options. However, you can override the default title with custom static text or use a dynamic title by inserting variables.
  You can insert dynamic variables into the alert title in the Insert variable dropdown. These variables help identify the context of the alert more clearly when it is triggered.

  • You can include ${severity} as a variable in the alert title to indicate the severity level. This variable is useful when you configure multiple severity levels within a single alert. For example, a title such as High CPU Usage - ${severity} indicates the severity level directly in the alert title.
  • Additional variables are available based on the selected alerting method:
    • With custom aggregation, grouping by a tag (zone), and then, use the grouped tags as variables ( ${zone}). This variable is now available in the Insert variable dropdown.
    • When configuring per-entity alerts, the ${entity.label} variable is available. This variable identifies the specific entity that triggered the alert.
    • When you use metric patterns with regex capturing groups (for example, fs\.(.+)\.free), you can include the captured values as variables in the title. The Insert variable menu contains options similar to the following items:
      • Regex 1st capturing group for the first capturing group in your pattern
      • Regex 2nd capturing group, Regex 3rd capturing group, and so on, for additional capturing groups
      • With these variables, you can include parts of the matched metric name in your alert titles dynamically.
• Description: Optionally, add a description to the alert. A good description includes a short summary of the alert and the steps for investigation or resolution when the alert is received. You can also create a dynamic description by inserting variables, which can be selected from the Insert variable dropdown. These variables can provide contextual information when the alert is triggered.

  

• Triggers an incident: Enabling this toggle creates an Incident when the alert is generated. The alert is recorded as the triggering event for the incident. Other events related to this incident are referenced as related events.
• Custom payloads: To include an additional payload that is relevant to you in alert notifications for specific alert configuration that is sent by Instana, click Add Row in the Custom Payloads section.

  Both global custom payloads and alert-specific custom payloads are included in alert notifications if applicable, but the alert-specific configuration takes precedence over the global configuration. As a result, if you use the same key, the value of the global custom payload field is overridden by the alert-specific one.

  The following image shows globally defined custom payloads that are used in the alert configuration:

  

  For information about global custom payloads, see Configure Custom Payload Globally

  As of now, public preview does not support dynamic global custom payloads.

Click Create to create the infrastructure Smart Alert. The created alert displays on the Smart Alerts tab in the list of configured alerts.

Custom aggregation groups metric data based on tags you define and evaluates the aggregated metric against the threshold. This approach is useful for monitoring overall trends across multiple entities and reducing alert noise. You receive a single alert when the aggregated metric violates the threshold.

Per entity alerting monitors each entity individually and triggers separate alerts for each entity that violates the threshold. This approach is useful when you need to identify and respond to issues on specific entities. You receive individual alerts for each affected entity.

Choose the alerting method based on whether you want to monitor aggregated behavior (custom aggregation) or individual entity behavior (per entity alerting).

An alert is triggered when either the current metric value or the forecasted value exceeds the threshold. This proactive approach helps you to address potential issues before they impact your system, such as disk space running out or memory leaks approaching container limits.

Larger forecasted time windows increase the chance of false alerts, so balance proactive alerting with alert accuracy based on your specific use case.