Question & Answer
Question
How does the QRadar Event Rate (EPS) graph on the System Monitoring Dashboard derive its values?
Cause
QRadar reports event rates at various locations along its event pipeline. This can lead to confusion about the actual event load on the system and questions about licensing.
Quick Links:
Answer
The 'Event Rate (EPS)' Dashboard Graph in QRadar
The QRadar Event Rate (EPS) graph on the System Monitoring Dashboard derives its values from the StatFilter data, which is the last component of the ECS-EC portion of the event pipeline before data enters the processing phase (ECS-EP). Both routing rules and the licensing components occur before StatsFilter values are calculated, so values plotted in the Event Rate (EPS) graph do not include data like events dropped by a routing rule from the event pipeline. This is a common question from administrators who are using routing rules to drop events, however, unwanted events that are purposefully dropped is not reflected in the default dashboard graph due to the data used to generate the graph.
Figure 1: To view the search results that a graph is based on, click View in Log Activity.
The Event Rate (EPS) graphs are based on a search, which you can access by clicking on the “View in Log Activity” link present inside the Widget. When reviewing the results of the Event Rate (EPS) search in the Log Activity tab, you will notice that the search itself is filtering QRadar System Notifications that contain the phrase "StatFilter" and "Events per seconds" in their payloads.
Figure 2: The search filters that make up graphed data in QRadar
Where do you find the StatFilter information in QRadar?
The QRadar System notifications in turn are based on the qradar.log file in the /var/log/ directory in your QRadar Appliances. To see the raw notifications themselves you can open an SSH session from the Console to the managed host and run the following command:
grep StatFilter /var/log/qradar.log
The keyword administrators can use to view the event rate notification is StatFilter. The information used from the StatFilter notification data is used to generate the graph in Figure 1 and Figure 2.
The result will include many lines in the following form:
Jun 12 16:03:09 ::ffff:127.0.0.1 [ecs-ec]
[type=com.q1labs.semsources.filters.stat.StatFilter][parent=Lab-primary.q1labs.lab:ecs-ec/EC/Processor2]]
com.q1labs.semsources.filters.stat.StatFilter: [INFO] [NOT:0000006000][172.16.xx.xx/- -] [-/- -]
Events per second: 1s:5094,5094 (peak 7423,7423) (compression: 0%) 5s:5040,5040 (peak 5507,5507)
(compression: 0%) 10s:5045,5045 (peak 5269,5269) (compression: 0%) 30s:5043,5043 (peak 5120,5120)
(compression: 0%) 60s:5034,5034 (peak 5079,5079) (compression: 0%)
Events per second:
1s: 5094,5094 (peak 7423,7423) (compression: 0%)
5s: 5040,5040 (peak 5507,5507) (compression: 0%)
10s:5045,5045 (peak 5269,5269) (compression: 0%)
30s:5043,5043 (peak 5120,5120) (compression: 0%)
60s:5034,5034 (peak 5079,5079) (compression: 0%)
| Term | Definition |
|---|---|
| Time Period | The time period that the information is calculated over. |
| Average Coalesced EPS | The average of the events per second calculated after coalescing. The average is calculated over readings taken during the time period. |
| Average Raw EPS | The average of the events per second calculated before coalescing. The average is calculated over readings taken during the time period. |
| Peak Coalesced EPS | The highest events per second seen during the time period, after coalescing. |
| Peak Raw EPS | The highest events per second seen during the time period, before coalescing. |
| Compression | QRadar events are compressed when stored on disk. This value is the percentage size of the compressed events compared to the original size. |
Important: An enhancement was made to the regular expression used to parse StatFilter notification as part of the Baseline Maintenance Extension for QRadar. Administrators on QRadar 7.2.6 and later should download and install the Baseline Maintenance Extension on their Console.
How to view raw event statistics
The event rate before licensing and forwarding can be also obtained by similarly reviewing the SourceMonitor values found in the same log file, which reflect the Event Rate earlier in the event processing pipeline.
To see SourceMonitor values, use the command:
grep SourceMonitor /var/log/qradar.log
Note: The keyword administrators can use to view the log event rate notifications is SourceMonitor.
The output shows the full information for a SourceMonitor log event:
Jun 12 16:04:42 ::ffff:127.0.0.1 [ecs-ec-ingress.ecs-ec-ingress]
[SourceMonitor-4/ecs-ec-ingress.ecs-ec-ingress] com.q1labs.sem.monitors.SourceMonitor: [INFO]
[NOT:0000006000][127.0.xxx.xxx/- -] [-/- -]Incoming raw event rate (5s: 8422.00 eps), (10s: 8482.10
eps), (15s: 8458.13 eps), (30s: 8459.87 eps), (60s: 8514.48 eps), (300s: 8496.89 eps), (900s:
8496.89 eps). Peak in the last 60s: 8778.20 eps. Max Seen 8903.40 eps. EC Throttles/5s (60s: 10.00).
Total EC Throttles in the last 60s: 120. Total EC Throttles: 5442. Appliance Threshold: 5024.00
event processing pipeline; before or after event coalescing. Both stages produce messages in the
same format. You can distinguish which stage they are from by looking at this segment of the message:
If it says ecs-ec.ecs-ec, it is based on normalized and coalesced events.
| Incoming raw event rate | Incoming events per second, averaged over different time periods. |
| Peak in the last 60s | Highest events per second measured in the last sixty seconds. |
| Max Seen | Highest events per second measured since the process last started. |
| EC Throttles/5s | Amount of times the event collection is throttling incoming events due to licensing limits or delays over the last 5 seconds. |
| Total EC Throttles in the last 60s | Amount of times the event collection process has had to throttle incoming events in the last minute. |
| Total EC Throttles | Amount of times the event collection process has had to throttle incoming events since it was last restarted. |
| Appliance Threshold | EPS value at which event collection is throttled due to the license. |
These examples are intended to contrast the two EPS statistics and show that they provide quite different values even though they were sampled less than 60 seconds apart. The SourceMonitor counter measures 8514.48 (60 second average EPS), while StatFilter reports almost exactly the same period with an Event Rate of 5034 EPS.
The events being received in excess of the license rate are being buffered and processed at license rate as explained in Technote QRadar: Event and Flow Burst Handling (Buffer). When the actual event load exceeds your license capacity there will be other notifications indicating this, but the Event Rate (EPS) graph which is based on the StatFilter data will not reflect this. Furthermore, if you have Routing Rules that drop certain types of events, the value reported by StatsFilter may be even lower than your license limit. This can lead to situations where you are receiving notifications indicating you are above your license rate even though the EPS graph shows values at or below the license rate.
License giveback explained in Technote QRadar: License EPS rates and giveback
Was this topic helpful?
Document Information
Modified date:
12 August 2024
UID
swg21984283