Troubleshooting
Problem
How do I determine how many events are dropped when the EPS license limit is reached?
Cause
Events that exceed the licensed rate are sent to a spillover queue where the data can be buffered by ecs-ec-ingress. If the 5GB disk buffer fills and the license is exceeded, then events can be dropped from the event pipeline as there is no room in the queue for the event data.
Resolving The Problem
The number of incoming or peak, spillover queue, or dropped events can be viewed by using the QRadar Deployment Intelligence application or users with root access can review the logs in /var/log/qradar.log
file.
- To view EPS rates from the command-line interface of the QRadar appliance, type:
less -iS /var/log/qradar.log | grep peak
Incoming raw event rate (5s: 221.20 eps), (10s: 167.90 eps), (15s: 150.67 eps), (30s: 114.40 eps), (60s: 130.25 eps), (300s: 129.94 eps), (900s: 129.94 eps). Peak in the last 60s: 229.20 eps. Max Seen 301.40 eps. EC Throttles/5s (60s: 0.00). Total EC Throttles in the last 60s: 0. Total EC Throttles: 2. License Threshold: 5020.00
- To view the number of files in the spillover queue in /store/transient, type:
less -iS /var/log/qradar.log | grep spillover
com.ibm.si.ecingress.filters.QueuedEventThrottleFilter: [INFO] [NOT:0000006000] [IP ADDRESS/- -] [-/- -](Current events spillover: 1; Events added last 60 seconds: 7679; Events removed last60 seconds: 7678; Files in use/max: 1/2500; Remaining capacity: 10240000)
- To view dropped events, type:
less -iS /var/log/qradar.log | grep -i "license restrictions"
tail -n 15 /var/log/qradar.log | grep "peak of"
Line 81403: [ecs-ec] [e0dxxxxx-a9xx-4exx-b2a9-cf4dc3xxxxxx/SequentialEventDispatcher]com.q1labs. sem.monitors.SourceMonitor: [WARN] [NOT:0060005100][xxx.xxx.xxx.xxx/- -] [-/- -] A total of 171368542 dropped raw event(s)have been detected. 39914 raw event(s) have been dropped in the last 60 seconds.License restrictions have been applied 120 times in the last 60 seconds. The average event rate in the last 60 seconds was 1695.18 eps (with a peak of 1805.80 eps), and within that time has exceeded the 1024.00 eps license set on the system 12 times.
- You can get similar data for Events and Flows from the QRadar Deployment Intelligence app (QDI). This app displays licensing, Events per second, Flows per second and more.
Figure 1 Events Total and Events Dropped.
Results
The Use of these tools from the command line or from the QDI app can help administrators determine whether they are experiencing dropped or spillover events.
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"Licensing","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]
Was this topic helpful?
Document Information
Modified date:
01 March 2023
UID
swg21995164