Question & Answer
Question
Why is it when running raw data against the data found in a report, the values are not equal?
Answer
Aside from deploys being run and restarts of the accumulator, the reason why results are going to be smaller than the raw results is because of unique counts versus non-unique counts.
Example:
The Search Compliance: Username Involved in Compliance Rules and it is Grouped By Username.
There is a Report toggled to keep track of the counts of the Usernames. We want to see all of the events for each Username. Every minute the accumulator wakes up and counts each of the events for the bucket. If in minute 1 of the interval we saw 5 Usernames. We add 5 to the count bucket. The second minute comes in then the third, and so forth. The 60 intervals in the hour that the count of Usernames varied from 3-15.
Every hour the Accumulator_rollup process wakes up and rolls up the 60 intervals worth of data into one Hourly Rollup. The rollup process, in this case, takes the MAXIMUM value of all intervals, and that becomes the value for that interval. In this case, it would be 15.
This is not always necessarily correct. Let's say in interval one we saw 5 unique Usernames, 10 separate Usernames (different from the first interval), and then in the third interval 10 unique Usernames were seen. The count should now be at least 25, ignoring what comes in from the remaining 57 intervals.
This is why the counts are off. To avoid this scenario for when we need the counts to be accurate. You can Enable Unique Counts from the search the reports are using and toggle the enable unique counts option. Search / Edit Search (Select the search):
Caution: This can have performance implications because now this needs to keep a running hash map of all the unique entries seen in the interval when processes begin to roll up.
Example:
The Search Compliance: Username Involved in Compliance Rules and it is Grouped By Username.
There is a Report toggled to keep track of the counts of the Usernames. We want to see all of the events for each Username. Every minute the accumulator wakes up and counts each of the events for the bucket. If in minute 1 of the interval we saw 5 Usernames. We add 5 to the count bucket. The second minute comes in then the third, and so forth. The 60 intervals in the hour that the count of Usernames varied from 3-15.
Every hour the Accumulator_rollup process wakes up and rolls up the 60 intervals worth of data into one Hourly Rollup. The rollup process, in this case, takes the MAXIMUM value of all intervals, and that becomes the value for that interval. In this case, it would be 15.
This is not always necessarily correct. Let's say in interval one we saw 5 unique Usernames, 10 separate Usernames (different from the first interval), and then in the third interval 10 unique Usernames were seen. The count should now be at least 25, ignoring what comes in from the remaining 57 intervals.
This is why the counts are off. To avoid this scenario for when we need the counts to be accurate. You can Enable Unique Counts from the search the reports are using and toggle the enable unique counts option. Search / Edit Search (Select the search):
- Data is currently being accumulated for the search.
- Unique counts are disabled. Enable Unique Counts
Caution: This can have performance implications because now this needs to keep a running hash map of all the unique entries seen in the interval when processes begin to roll up.
[{"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Reports","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3;7.2;7.1","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]
Was this topic helpful?
Document Information
Modified date:
27 January 2021
UID
swg21692739