Anomaly detection rules

Anomaly detection rules test the results of saved flow or events searches to detect when unusual traffic patterns occur in your network.

Anomaly detection rules require a saved search that is grouped around a common parameter, and a time series graph that is enabled. Typically the search needs to accumulate data before the anomaly rule returns any result that identifies patterns for anomalies, thresholds, or behavior changes.

Anomaly rules

Test event and flow traffic for changes in short-term events when you are comparing against a longer timeframe. For example, new services or applications that appear in a network, a web server crashes, firewalls that all start to deny traffic.

Example: You want to be notified when one of your firewall devices is reporting more often than it usually does because your network might be under attack. You want to be notified when you receive twice as many events in 1 hour. You follow these steps:

Create and save a search that groups by log source, and displays only the count column.
Apply the saved search to an anomaly rule, and add the rule test, and when the average value (per interval) of count over the last 1 hour is at least 100% different from the average value (per interval) of the same property over the last 24 hours.

Threshold rules

Test events or flows for activity that is greater than or less than a specified range. Use these rules to detect bandwidth usage changes in applications, failed services, the number of users connected to a VPN, and detecting large outbound transfers.

Example: A user who was involved in a previous incident has large outbound transfer.

When a user is involved in a previous offense, automatically set the Rule response to add to the Reference set. If you have a watch list of users, add them to the Reference set. Tune acceptable limits within the Threshold rule.

A reference set, WatchUsers, and Key:username are required for your search.

Complete the following search, and then apply it to a Threshold rule.

select assetuser(sourceip, now()) as 'srcAssetUser',
Applicationname(applicationid)as 'AppName', long(sum(sourcebytes
+destinationbytes)) as 'flowsum' from flows where flowdirection = 'L2R' and 
REFERENCESETCONTAINS('Watchusers', username)group by 'srcAssetUser',
applicationid order by 'flowsum' desc last 24 hours

Behavioral rules

Test events or flows for volume changes that occur in regular patterns to detect outliers. For example, a mail server that has an open relay and suddenly communicates with many hosts, or an IPS (intrusion protection system) that starts to generate numerous alert activities.

A behavioral rule learns the rate or volume of a property over a pre-defined season. The season defines the baseline comparison timeline for what you're evaluating. When you set a season of 1 week, the behavior of the property over that 1 week is learned and then you use rule tests to alert you to any significant changes.

After a behavioral rule is set, the season adjusts automatically. When the data in the season is learned, it is continually evaluated so that business growth is profiled within the season; you do not have to change your rules. The longer a behavioral rule runs, the more accurate it becomes. You can then adjust the rule responses to capture more subtle changes.

The following table describes the behavioral rule test parameter options.

Table 1. Behavioral rule test definitions
Rule test parameter	Description
Season	The most important value. The season defines the baseline behavior of the property that you are testing and which the other rule tests use. To define a season, consider the type of traffic that you are monitoring. For example, for network traffic or processes that include human interaction, 1 week is a good season timeframe. For tracking automated services where patterns are consistent, you might want to create a season as short as 1 day to define that pattern of behavior.
Current traffic level	Weight of the original data with seasonal changes and random error accounted for. This rule test asks the question, "Is the data the same as yesterday at the same time?" The weight must be in the range of 1 to 100. A higher value places more weight on the previously recorded value.
Current traffic trend	Weight of changes in the data for each time interval. This rule test asks the question, "How much does the data change when it compares this minute to the minute before?" The weight must be in the range of 1 to 100. A higher value places more weight on traffic trends than the calculated behavior.
Current traffic behavior	Weight of the seasonal effect for each period. This rule test asks the question, "Did the data increase the same amount from week 2 to week 3, as it did from week 1 to week 2?" The weight must be in the range of 1 to 100. A higher value places more weight on the learned behavior.
Predicted value	Use predicted values to scale baselines to make alerting more or less sensitive. The sensitivity must be in the range of 1 to 100. A value of 1 indicates that the measured value cannot be outside the predicted value. A value of 100 indicates that the traffic can be more than four times larger than the predicted value.

The forecast for value from (n+1)^th interval is calculated by using the following formula:

F_n+1 = B_n + T_n+ T_n+1-s

Where F is the predicted value, B is the base value for interval n, T is the trend value for interval n, and T is the trend value for season intervals ago and s is the number of intervals within the season.

The base value is calculated by using the following formula:

B_n+1 = (0.2 + 0.3*(<Current traffic level> / 100.0))*(value_n+1 – T_n+1-s) + (1 – (0.2 + 0.3*(<Current traffic level> / 100.0)))*T_n

The trend value is calculated by using the following formula:

T_n+1 = (0.2 + 0.3*(<Current traffic trend> / 100.0))*(B_n+1 - B_n) + (1 - (0.2 + 0.3*(<Current traffic trend> / 100.0)))*T_n

Smoothed deviation D is calculated by using the following formula:

D_n+1 = (0.2 + 0.3*(<Current traffic behavior> / 100.0))*|value_n+1 – F_n+1| + (1 – (0.2 + 0.3*(<Current traffic behavior> / 100.0)))*D_n+1-s

The behavioral rule produces an alert for the interval if the following expression is false:

F – (1 + (sensitivity / 100.0)*3)*D <= value <= F + (1 + (sensitivity / 100.0)*3)*D

During the first season, the behavioral rule learns for future calculations and doesn't produce any alerts.