Troubleshooting
Problem
Reference data inserts and updates configured in rule responses are not processed because the RefDataDestination queue gets full, resulting in the error: "EP Queue [RefDataDestination] has detected xxx dropped event(s)"
Symptom
Administrators might see the following alerts under the notification bell or the qradar.error log files. It can trigger on hosts with an instance of the ecs-ep service, which are the console, event processors and flow processors.
[ecs-ep.ecs-ep] com.q1labs.sem.monitors.ECSQueueMonitor: [WARN] [NOT:0060005100][IPADDRESS/- -] [-/- -]ECS Queue Monitor has detected a total of 33397 dropped event(s). 923 event(s) were dropped in the last 60 seconds. EP Queues: 923 dropped event(s). MPC Queues: 0 dropped event(s).
[ecs-ep.ecs-ep] com.q1labs.sem.monitors.ECSQueueMonitor: [WARN] [NOT:0000004000][IPADDRESS/- -] [-/- -]EP Queue [RefDataDestination] has detected 930 dropped event(s) in the last 60 seconds and is at 0 percent capacity
Cause
Due to misconfigured rules that are updating reference data collections excessively, there could be a congestion while the elements are updated, which results in new rule response requests being dropped. This error does not indicate that events are dropped, but that the responses that come from the rules to the reference data collections are dropped.
Resolving The Problem
To resolve the problem, we need to pull out the list of rules that write the most to the reference data collections per day. You can correlate the timestamp of the alert with the number of inserts that happened that day.
- SSH in to the QRadar console as root.
- Run the following command
psql -Uqradar -c "SELECT rde.last_seen::date as day, COUNT(rde.id), rde.source as rule_name, rd.name as reference_data_collection FROM reference_data_element rde JOIN reference_data_key rdk on rde.rdk_id = rdk.id JOIN reference_data rd on rdk.rd_id = rd.id where rde.source != 'reference data api' GROUP BY last_seen::date, source, rd.name HAVING COUNT(rde.id) > 500 ORDER BY last_seen::date DESC, COUNT(rde.id) DESC LIMIT 100" | less -iSR - Examine the output. You must identify the name of the rules that are updating reference data collections the most per day, the dates the problem occurred. See the following example output:
day | count | rule_name | reference_data_collection ------------+-------+----------------+----------------------------- 2025-07-02 | 69908 | Custom Rule #1 | Rerence data collection #1 2025-07-02 | 64131 | Custom Rule #2 | Rerence data collection #2 2025-07-02 | 30795 | Custom Rule #3 | Rerence data collection #3 2025-07-02 | 25470 | Custom Rule #4 | Rerence data collection #4 2025-07-02 | 11035 | Custom Rule #5 | Rerence data collection #5 2025-07-01 | 48507 | Custom Rule #1 | Rerence data collection #1 2025-07-01 | 30969 | Custom Rule #2 | Rerence data collection #2 2025-07-01 | 15895 | Custom Rule #3 | Rerence data collection #3 2025-07-01 | 15258 | Custom Rule #4 | Rerence data collection #4 2025-07-01 | 9128 | Custom Rule #5 | Rerence data collection #5 - Open the GUI, go to Offenses, Rules and search for each of the top offender rules based on the previous output. Edit them and make sure they have a proper response limiter. Without a response limiter, every single event that hits the rule will trigger an update to the reference data collection.
Result
A properly configured response limiter prevents the rule from making too many updates to the reference data collections and getting the RefDataDestination queue full. Response requests are no longer dropped and the error does not trigger.
A properly configured response limiter prevents the rule from making too many updates to the reference data collections and getting the RefDataDestination queue full. Response requests are no longer dropped and the error does not trigger.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtrAAA","label":"Rules"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
01 May 2024
UID
ibm17005987