Anomaly Detection

Edit online

Anomaly detection in Cloudability helps organizations identify unusual or unexpected patterns in cost spending. This cloud-agnostic feature provides a comprehensive analysis across all services to detect cost anomalies, enabling users to mitigate sudden billing spikes through timely alerts. Users can monitor anomalies directly within Cloudability or receive notifications via email and PagerDuty.

Cost Segment Formations

To effectively identify anomalies, cost data is categorized into two segments: Standard Cost and Service-Level Cost. This segmentation helps isolate independent spending patterns, allowing for a more precise analysis of unusual fluctuations in costs.

Standard Cost Segment formation is calculated using the total costs by date, service name, usage family, account, best five tags, and best five business mapping dimensions.

Service-level Cost Segment formation is calculated using the total costs by date, service name and usage family.

Below is an example where we have defined the tags, and the Business Mapping algorithm is chosen for identifying anomalies.

.

Note:

Users do not specify which tag and biz mapping dimension to use, we algorithmically identify the tag dimension and biz mapping dimension that have the best quality values. For example, a Tag dimension with mostly missing values is not as useful as a Tag dimension with well-populated values.

Setting Spending Thresholds and Alert Notifications

You can set up an alert for anomalies via:

  • Email-based Alerts

  • Pager Duty Alerts

You can configure alerts by setting a threshold value, which can be defined as either an absolute amount or a percentage. When the cost surpasses the specified threshold which can be specified either as a fixed value or a percentage increase, an alert is automatically triggered.

Based on the selected notification preference, you will receive alerts via email or through PagerDuty. Additionally, you have the flexibility to set both absolute and percentage-based thresholds to ensure timely notifications.

To set up an alert, define and identify the situation. For example, you can start by creating a view in the Views section of the Cloudability application. While creating the view, select appropriate filters which will trigger alerts. For more information on setting views in Cloudability, please see Setting Views in Cloudability.

After the view is set up, create an alert in the Anomaly Detection section of Cloudabilty. Consider the following example.

A user wants to monitor cloud spending for their AWS account using Cloudability. They set up an alert with the following thresholds for a selected View:

  • Absolute Value Threshold: $50,000
  • Percentage-Based Threshold: 20% increase from the previous cost.

Now consider the following scenarios:

  • Scenario 1: Absolute Value Threshold Triggered: If the total AWS cost reaches $50,001 (exceeding the $50,000 threshold), the alert is triggered, notifying the user.
  • Scenario 2: Percentage-Based Threshold Triggered: If last month's AWS cost was $40,000, and this month’s cost increases to $48,000 (a 20% increase), the alert is triggered.

This makes it easy to track spending and receive alerts when costs exceed set limits. You also have the flexibility to choose either one of the threshold values or both values.

Accessing Anomaly details

To explore an anomaly in more detail, click the View Report button on the detail page. This will open the reporting interface, where you can add additional dimensions or adjust the date range for deeper analysis.

To access anomaly details, follow the steps below.

  1. Navigate to Insights > Anomaly Detection.
  2. On the Spending Anomalies page, click Details.
  3. Select View Report to add more dimensions for analysis.
  4. Use the Edit option to modify the time period or date range for a customized view.

This provides a comprehensive investigation of cost anomalies helping you gain better insights into unusual spending patterns.

Frequently Asked Questions

  1. At which frequency are Anomalies detected?

    Anomalies are detected every 24 hours, Once the cost data is updated. Our goal is to deliver timely alerts while minimizing unnecessary notifications. Each time the anomaly detection algorithm runs, it re-evaluates and regenerates anomalies for the past seven days.

  2. Why do Anomalies change/ disappear?

    Each time the anomaly detection algorithm runs, it regenerates anomalies for the past seven days. Within this seven-day window, the model becomes more informed, making it possible for anomalies to change or disappear.
    • If the amount for the given anomaly changes, it is mainly due to updates in the cost data.
    • If an anomaly disappears, it is primarily because the five daily selected tags/ dimensions differ, causing the previous anomaly to be overwritten.

  3. Does Anomaly Detection show downward spend?

    No, anomaly detection only shows unusual upward spend.

  4. Can we change the way Anomalies are detected?

    The formulas that detect an anomaly cannot be changed, however, customers can change the threshold for the notifications they receive.

  5. Why didn’t I receive an alert for the first-time spend?

    An alert was not triggered for the first-time spend because the system does not support first-time spend until a baseline is established. Alerts are generated based on deviations from historical data, and without prior spending history, the system cannot detect anomalies or send alerts.

  6. Can a reprocess trigger new Anomalies or alerts?

    Yes, Anomalies can appear or disappear as new billing data comes in, or as tag mappings and business mappings are updated.

  7. Why was I alerted about an Anomaly, but it is no longer visible in the UI?

    Cloudability's anomaly detection algorithm identifies anomalies based on the latest available billing data. When an anomaly is detected, an alert is sent to notify the user. However, cloud service providers (CSPs) continuously update billing files, which can lead to changes in reported costs.

    If subsequent billing updates modify the total spend for a given day, the anomaly detection algorithm may no longer classify that day's spending as unusual. As a result, the previously detected anomaly is removed from the UI.

    Example: If a vendor applies a credit adjustment, reducing the total spend for a specific day, the previously flagged anomaly may no longer be considered unusual and will no longer appear in the UI.

  8. What are the scenarios user will not receive the alert?

    • The anomaly reports an unusual spend of $300, but the customer has set a threshold of $350. In this case, an alert will not be sent.
    • A user sets a subscription with a specific View, but their permissions change, and they no longer have access to that View. As a result, an alert will not be sent.
    • A View is configured to trigger alerts when the tag "team=cldy" appears. If the tag is missing or is set as "team=apptio", the Alerter will not send an alert.
    • An anomaly with the same date, total spend, unusual spend, account, tags, and business mappings has already been reported for a particular user. If an identical anomaly is received again, another alert will not be sent.
    Anomalies are detected every 24 hours, Once the cost data is updated. Our goal is to deliver timely alerts while minimizing unnecessary notifications. Each time the anomaly detection algorithm runs, it re-evaluates and regenerates anomalies for the past seven days.

  9. Can I customize Anomaly Detection?

    Currently, Cloudability does not support customization of the anomaly detection feature. However, our Product Team would love to hear about your use case and requirements. We recommend that you open a product feature request in our community forums with your specific use case. This will enable the Product Team to review and consider your needs for future updates.