IBM Support

QRadar: How to troubleshoot accumulator issues using collectGvStats.sh

Troubleshooting


Problem

You may see the following system notifications:
 
  • "The accumulator was unable to aggregate all events or flows for this interval."
  • "The accumulator has fallen behind. See Aggregated Data Management for details."

Symptom

Look in /var/log/qradar.error for similar messages.

Payload: Aug  9 05:02:15 127.0.0.1  [accumulator.accumulator] [AccumulationService] com.q1labs.cve.accumulation.AccumulationService:
[WARN] [NOT:0310004100][10.xxx.xxx.40/- -] [-/- -]Interval processing time (65 seconds) exceeded threshold (60 seconds)


Note: This may cause certain dashboard items to display false or incomplete data.

Cause

Accumulator issues are caused by searches that are not properly tuned or too many global views in the system. By default, we only allow a maximum of 130 Global Views for 7.2.x or 300 Global Views in 7.3.x.

Resolving The Problem


What is Global Views (GV) or Accumulated data

Think of a Global View as a description of the data you want to be collected. In it, there is a description of the criteria that match the search and keys (columns) of the data you are interested in.

What is Virtual Views

Virtual Views are like a database view. They are based on an existing description of data (Global View) and simply contain references to Columns in the Global View. Virtual Views are contained and described within a Global View and can only be associated with one Global View. They can only reference columns that are available in the Global View. To collect data for a column a reference must be added to the column in a Virtual View.

Note: Virtual Views are the only way to collect data. The Virtual View can contain multiple references to a single column, however when all references are gone the data collection will also stop.


What is Accumulated data

Accumulated Data is an aggregate data view used to draw a Time Series graph or run Scheduled Reports when you create a search that groups by 1 or more properties. We reference the data that is created by the term Global View. The Accumulator service runs on all appliances with local storage (Console, 16xx, 17xx, 18xx) to create the minute by minute accumulations. The accumulator_rollup service runs once per hour and creates the hourly roll-ups. At 12:15 AM it creates the daily roll-up.

Accumulated data, Global Views, Virtual Views, or Time Series are commonly used terms in QRadar that basically mean the same thing which is aggregate data values that are created into 'global views', when a saved search, dashboard (time series graph) or scheduled report is requested (daily or longer). These aggregations are only created when your search, dashboard or report is looking at data that is grouped by one or more data properties. When data is viewed in a group by (aggregate), we can create a 'rollup" or "total' values on a schedule. That same saved search used in a report, will not automatically show the time series in Log or Network Activity. You must still select the time series yourself to see the time series graph.

Global views can get expensive when there are a large number of 'unique values' in the global view or the search is not optimized.

For example, a global view exists when you create a search that groups by some property, such as 'Destination IP Address' or 'Destination Port'. If you think about common ports in use on a network as services, such as SMTP (25), WEB (80,443), SSH (22), this would be a pretty small number of unique values.  However, when you start talking about Destination IP addresses, each server would be a unique value. If you then consider that a group by or aggregate can be on multiple properties, such as Destination IP Address and Destination Port, then you get a multiplying effect. A hundred hosts with ten ports each becomes a thousand unique objects. Then, if you think about someone doing a group by on say, 'Source IP Address' and 'Destination IP Address', that could be up into the thousands. If you were to add 'Destination Port' and maybe 'Web URL', that could get into the hundreds of thousands. This is how global view costs to process, create and can grow.

To find out which Views may be causing the accumulator to fall behind, you can run the following command on the Event Processor:

/opt/qradar/support/collectGvStats.sh -s | less

This should return results such as the following:

Result:
view[10094] time: 83748ms. Refs: {unknown}                              
view[10098] time: 82968ms. Refs: {unknown}

We want to look for the amount of time it takes for the GVs to load. In this case, it took over 80 seconds for EACH of the Global View to load listed in our example to load. Since we only have 60 seconds to load ALL Global Views, 80 seconds to load each of these are excessive.

In this case, we need to find out which searches are associated with
those Views. To do that, you can run the following command on the console this time.

/opt/qradar/support/collectGvStats.sh -m 10094
/opt/qradar/support/collectGvStats.sh -m 10098

This will give you the following results:

view[10094]
Disk space usage of accumulated data for all hosts over the last 24    
hours: 29.33 KB                                                        
searchtitle                  | reporttitle            | useradded                          
-----------------------------+------------------------+-----------                                                              
This is username1's search:  |                        | username1
This is username2's search:  |                        | username2
(2 rows)                                                                
                                                                       
view[10098]                                                            
Disk space usage of accumulated data for all hosts over the last 24    
hours: 29.33 KB                                                        
searchtitle                  | reporttitle            | useradded             -----------------------------+------------------------+-----------                                                              
This is username1's search:  |                        | username1
This is username2's search:  |                        | username2
(2 rows)                                        

At this point, we know that the search "This is username's1 search" and "This is username's2 search" are the culprits and needs to be tuned or optimized.

From here, look at the saved search in question to see how it can be tuned or optimized. We want to look at the 'Group By' properties as well as the Filters used in the search. Reducing the amount of 'Group By' or adding more filters is a great way to reduce the footprint of the Global Views. The order of the filters is important as well. We also recommend using Quick Filter instead of Payload Contains since it is more efficient. However, the Quick Filter can only be a "positive" filter and not "negative" and you can only have 1 Quick Filter per search.

A good example of a search is as follows:
  1. Log in to the QRadar User Interface.
  2. Click the Log Activity tab.
  3. Click New Search.
  4. Scroll to Column Definition.
  5. Remove Username from Columns
  6. Add Username it to Group By.
  7. Scroll to Search Parameters
  8. Choose Parameter Event Name > Equals
  9. Click Browse

    image-20190206123906-4
  10. Enter QID/Name Malware-object > search
  11. Enter Log Source Type FireEye
  12. Enter High-Level Category: Malware > Low-Level Category Misc Malware
  13. Click OK

    image-20190206123800-3
  14. Click Filter

Another thing that may cause accumulator to fall behind is the number of views. We only allow 130 Global Views for 7.2.x or 300 Global View in 7.3.x


There may be views that have no search references that could be deleted. When a user creates a saved search or dashboard item and later deletes that saved search or dashboard item, the result is an empty view. This is eating up resources and should be best practice for an administrator to delete those views when they are no longer in use.

You can find out which view has no search reference by running the following command on the console:

/opt/qradar/support/collectGvStats.sh -M | grep -B5 "0 rows" | less

The result would be similar to:

view[10212]
Disk space usage of accumulated data for all hosts over the last 24 hours: 0 B
 searchtitle | reporttitle | useradded
-------------+-------------+-----------
(0 rows)

You can then go in Admin Tab > Aggregated Data Management. Then change the
Display to Aggregated Data View and delete whichever view that has no
search references from the previous command result.

Lastly, disk IO may cause accumulator falling behind as well.
In this case, you can look at dstat command to see real-time disk IO
image-20190206132321-6
or look at the /var/log/systemStabMon/year/month/day/iostat_sda log file for historical result.

Related Information

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"Aggregate Veiw managment;Reports;Searches","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.2;7.3","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
21 April 2021

UID

ibm10870446