IBM Support

QRadar: How to view the number of events exceeding the Event Processor System (EPS) licensed limit

Troubleshooting


Problem

How do I determine how many events have been dropped when the EPS license limit is reached?

Cause

Events that exceed the licensed rate are sent to a spillover queue where the data can be buffered by ecs-ec-ingress. If the 5GB disk buffer fills and the license is exceeded, then events can be dropped from the event pipeline as there is no room in the queue for the event data.

Resolving The Problem

The number of incoming or peak, spillover queue, or dropped events can be viewed using the QRadar Deployment Intelligence application or users with root access can review the logs in /var/log/qradar.log file.
 

  • To view EPS rates from the command-line interface of the QRadar appliance, type: less -iS  /var/log/qradar.log | grep peak

    Example
    Incoming raw event rate (5s: 221.20 eps), (10s: 167.90 eps), (15s: 150.67 eps), (30s: 114.40 eps), (60s: 130.25 eps), (300s: 129.94 eps), (900s: 129.94 eps). Peak in the last 60s: 229.20 eps. Max Seen 301.40 eps. EC Throttles/5s (60s: 0.00). Total EC Throttles in the last 60s: 0. Total EC Throttles: 2. License Threshold: 5020.00
     
  • To view the number of files in the spillover queue in /store/transient, type: less -iS  /var/log/qradar.log | grep spillover

    Example
    com.ibm.si.ecingress.filters.QueuedEventThrottleFilter: [INFO] [NOT:0000006000][IP ADDRESS/- -] [-/- -] (Current events spillover: 1; Events added last 60 seconds: 7679; Events removed last60 seconds: 7678; Files in use/max: 1/2500; Remaining capacity: 10240000)
     
  • To view dropped events, type: less -iS  /var/log/qradar.log | grep -i "license restrictions"
    Optionally, you can also tail for this information with the command: tail -n 15  /var/log/qradar.log | grep "peak of"

    Example
    Line 81403: Sep 13 13:33:13 ::ffff:xxx.xxx.xxx.xxx [ecs-ec] [e0dxxxxx-a9xx-4exx-b2a9-cf4dc3xxxxxx/SequentialEventDispatcher]com.q1labs.sem.monitors.SourceMonitor: [WARN] [NOT:0060005100][xxx.xxx.xxx.xxx/- -] [-/- -]A total of 171368542 dropped raw event(s) have been detected. 39914 raw event(s) have been dropped in the last 60 seconds. License restrictions have been applied 120 times in the last 60 seconds. The average event rate in the last 60 seconds was 1695.18 eps (with a peak of 1805.80 eps), and within that time has exceeded the 1024.00 eps license set on the system 12 times.

  • You can get similar data for Events and Flows from the QRadar Deployment Intelligence app (QDI). This app allows you to see licensing, Events per second, Flows per second and more.

    image-20191116111240-1
    Figure 1 Events Total and Events Dropped.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"Licensing","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
07 January 2021

UID

swg21995164