Question & Answer
Question
IBM QRadar users might see several notifications about accumulator falling behind. Most commonly notifications such as these are seen:
The accumulator has fallen behind. See Aggregated Data Management for detailsThe accumulator was unable to aggregate all events/flows for this interval.How can you resolve this issue when it is related to default EPS and FPS views?
Cause
When you see these notifications, the first thing you need to think about is system performance. Those messages are not really to say errors but performance issues causing the accumulator process to fall behind for the 60 seconds cycle. By design, the accumulator process has 60 seconds to accumulate on all global views configured on the system. If in one cycle, it is not able to complete all of the data accumulation within 60 seconds, then the error messages noted in the Question are generated. Receiving those messages once or twice a day is not normally significant, as documented here. However, when the accumulator constantly generates these errors then that can indicate a problem and indicates that performance tuning needs to be done on the system.
Answer
In this article, we focus on one specific scenario that can cause decreased performance with Payload Contains criteria in Saved Searches associated with Global Views. In particular, if in the top consumers there are GVs corresponding to the Event Rate (EPS) or Flow Rate (FPS) default searches, or any other Saved Search that uses a Payload Contains filter, then this article applies.
To start, follow the step-by-step instructions here to run the collectGvStats tool to determine which global views are taking the most time to complete.
For example,
view[10013] time: 8559ms. Refs: {saved_search: Flow Rate (FPS), unknown} view[10012] time: 8316ms. Refs: {saved_search: Event Rate (EPS), unknown, report: Event Rate (EPS), report: My_DashBoard_Event Rate (EPS)}
or
view[10259] time: 1078ms. Refs: {unknown}
view[10256] time: 1034ms. Refs: {saved_search: Flow Rate (FPS)}These default searches are not meant to be edited by IBM QRadar users, but when they are edited, the editor adds a 'locale' criteria, which is considerably slower than the default Global View configuration for these searches.
In order to work around the performance impact of the 'locale' criteria, you need to rework the corresponding searches to use Payload Matches instead of Payload Contains.
1. Make sure each user that uses the Event Rate (EPS) and Flow Rate (FPS) views has their locale set to their locale language. For example, in US, the Locale would be set to "English" as shown here.
2. Log in as a user with full administrative privileges and re-create the two searches by using the following AQL:
Event Rate (EPS) Custom
SELECT "Parent" AS 'Parent (custom)', AVG("Events per Second Coalesced - Peak 1 Sec") AS 'Events per Second Coalesced - Peak 1 Sec (custom) (Average)', AVG("Events per Second Raw - Peak 1 Sec") AS 'Events per Second Raw - Peak 1 Sec (custom) (Average)', AVG("Events per Second Coalesced - Average 1 Min") AS 'Events per Second Coalesced - Average 1 Min (custom) (Average)', AVG("Events per Second Raw - Average 1 Min") AS 'Events per Second Raw - Average 1 Min (custom) (Average)', COUNT(*) AS 'Count' from events where "deviceType"='147' AND ( (UTF8(payload) MATCHES '.*Events per second.*') AND (UTF8(payload) MATCHES '.*StatFilter.*') ) GROUP BY "Parent" order by "Count" desc LIMIT 10000 last 15 minutes
Flow Rate (FPS) Custom
SELECT "Flow Source" AS 'Flow Source (custom)', AVG("Flows per Second - Peak 1 Min") AS 'Flows per Second - Peak 1 Min (custom) (Average)', AVG("Flows per Second - Average 15 Min") AS 'Flows per Second - Average 15 Min (custom) (Average)', COUNT(*) AS 'Count' from events where "deviceType"='147' AND (UTF8(payload) MATCHES '.*Incoming flow rate.*') GROUP BY "Flow Source" order by "Count" desc LIMIT 1000 last 12 hours
3. Enable Capture Time Series on these two searches in order to build new dashboard widgets to be shared with everyone. Follow the instructions here to build the widgets.
Once the searches are rebuilt, you do need to remove the original versions of those searches. First, make sure no user is using the old versions; then, delete the original version of the searches from the system. In order to pass the dependency checker in the UI, you would have to have each user delete their original Event Rate (EPS) and Flow Rate (FPS) widgets from their dashboard. If that is not possible, then create a new System Monitoring dashboard to share for use by all users.
[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtIAAQ","label":"Dashboard"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Version(s)"}]
Was this topic helpful?
Document Information
Modified date:
01 June 2022
UID
ibm16438689