IBM Support

QRadar: Accumulator Roll-up overview

Question & Answer


Question

What is an accumulation and what does QRadar do with accumulated data?

Answer

The Accumulator is a QRadar process that counts and prepares Events and Flows in data accumulations to assist with searches, displaying charts, and report performance.

Accumulated Data is an aggregate data view used to draw a Time Series graph or run Scheduled Reports. We reference the data that is created by the term Global View. The Accumulator service runs on all appliances with local storage (Console, 16xx, 17xx, 18xx) to create the minute by minute accumulations. The accumulator_rollup service runs once per hour and creates the hourly roll-up's. At 12:15 AM it creates the daily roll-up.

There are two ways in which accumulation can be enabled:

Capture Time Series data

The following steps outline how to capture Time Series data for accumulation:

  1. Access the Log Activity or Network Activity tab.

  2. Conduct a search, ensuring that the search criteria is grouped and specifies a time range. Charts are not displayed when you view events or flows in Real Time (streaming) mode.



  3. In the Charts pane, click the Configure icon.

  4. In the Chart Type drop down, select Time Series.
  5. Enable the Capture Time Series Data option to enable time series data capture. When you enable this option, the chart feature begins accumulating data for time series charts. By default, this option is disabled. This option is only available on Time Series charts.



  6. Click the Save icon.

  7. If the search used was not a previously Saved Search Criteria, a window will be presented in which a search criteria name will need to be provided.

  8. Click the OK icon.

Note: After enabling time series data capture for a selected parameter, an asterisk (*) is displayed next to the parameter in the Value to Graph list box. You will need to select the Capture Time Series Data check box and click the Save icon for each parameter for which you would like to accumulate data.

Scheduled Reports

If a previously Saved Search Criteria is used in a Hourly, Daily, Weekly, or Monthly Report, then the data matching the Saved Search Criteria will be accumulated. For more information on creating Reports, review the Creating custom reports documentation.


Overview

This data is rolled up in to three different resolutions by time in order to reduce the amount of data that needs to be queried when a search is performed across a data set.

Minute Roll-up

The first roll-up is the normal roll-up. The normal roll-up is 1 minute long by default. This can also be referred to as the minute roll-up. Every minute, the normalized data collected in the previous minute is aggregated and rolled up based on the accumulation type in the criteria.

Hourly Roll-up

Every hour, the normal roll-up's are rolled up into hourly data. This means the 60 data points produced by the normal roll-up will be rolled into 1 data point for the hour.

Daily Roll-up

Every day, the hourly roll-up's are rolled up into daily data. That means the 24 data points produced by the hourly roll-up will be rolled into 1 data point for the day.

The different roll-up's are basically different resolutions of the data. They are placed in a flat file within the Ariel database. Each file is subdivided as Years, Months, Days Hours. The minute roll-up's are files in the hours file.


Example of how the Accumulator rolls up data

The accumulator data for this example is based on the following search criteria:
Search criteria: Grouped by IP and Policy, Column = Risk Score (sum)

Example data:
  • 01/15/2013 01:01:02 IP=1.1.1.1, Policy A, Risk Score = 5
  • 01/15/2013 01:01:23 IP=1.1.1.1, Policy A, Risk Score = 7
  • 01/15/2013 01:01:12 IP=1.1.1.1, Policy B, Risk Score = 10
  • 01/15/2013 01:02:43 IP=1.1.1.1, Policy A, Risk Score = 5
  • 01/15/2013 02:05:14 IP=1.1.1.1, Policy A, Risk Score = 5

Minute Roll-up

Data is grouped by IP and Policy and Risk Scores are summed. In the example data above, there are two events that occurred within the same minute. During the Minute Roll-up, these two events will be rolled into one, as the results of their grouped search parameters (IP and Policy) are the same. The Risk Scores for these two events will be summed together as seen below:

   01/15/2013 01:01:02 IP=1.1.1.1, Policy A, Risk Score = 5
+ 01/15/2013 01:01:23 IP=1.1.1.1, Policy A, Risk Score = 7
= 01/15/2013 01:01 IP=1.1.1.1, Policy A, Risk Score = 12

The accumulated Minute Roll-up from the example data will result in the following:
  • 01/15/2013 01:01 IP=1.1.1.1, Policy A, Risk Score = 12
  • 01/15/2013 01:01 IP=1.1.1.1, Policy B, Risk Score = 10
  • 01/15/2013 01:02 IP=1.1.1.1, Policy A, Risk Score = 5
  • 01/15/2013 02:05 IP=1.1.1.1, Policy A, Risk Score = 5

Hourly Roll-up

The Hourly Roll-up will further roll up the resulting data from the Minute Roll-up each hour. There were three results for the first hour, all were from the same IP, but two were from Policy A, and one was from Policy B. The two results that have the same IP and Policy will be rolled together as the results of their grouped search parameters (IP and Policy) are the same. The Risk Scores for these two results will be summed together as seen below:

   01/15/2013 01:01 IP=1.1.1.1, Policy A, Risk Score = 12
+ 01/15/2013 01:02 IP=1.1.1.1, Policy A, Risk Score = 5
= 01/15/2013 01 IP=1.1.1.1, Policy A, Risk Score = 17

The accumulated Hourly Roll-up from the results of the Minute Roll-up data will result in the following:
  • 01/15/2013 01 IP=1.1.1.1, Policy A, Risk Score = 17
  • 01/15/2013 01 IP=1.1.1.1, Policy B, Risk Score = 10
  • 01/15/2013 02 IP=1.1.1.1, Policy A, Risk Score = 5

Daily Roll Up

Once a day, the Daily Roll-up will again roll up the resulting data from the Hourly Roll-up even further.
There were three for the day, but only two have both the same IP and Policy, so those two will be rolled together as the results of their grouped search parameters (IP and Policy) are the same. The Risk Scores for these two results will be summed together as seen below:

   01/15/2013 01 IP=1.1.1.1, Policy A, Risk Score = 17
+ 01/15/2013 02 IP=1.1.1.1, Policy A, Risk Score = 5
= 01/15/2013 IP=1.1.1.1, Policy A, Risk Score = 22

The accumulated Daily Roll-up from the results of the Hourly Roll-up data will result in the following:
  • 01/15/2013 IP=1.1.1.1, Policy A, Risk Score = 22
  • 01/15/2013 IP=1.1.1.1, Policy B, Risk Score = 10


Where do you find more information?

[{"Product":{"code":"SSBQAC","label":"IBM QRadar SIEM"},"Business Unit":{"code":"BU008","label":"Security"},"Component":"Reports","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3;7.2","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
21 June 2018

UID

swg21677942