Troubleshooting
Problem
The following error can appear in the Event processor error logs: "ECS Queue Monitor has detected a total of xxx dropped event(s)". The error indicates that reference responses are not processed due to a queue that is full or the threshold is crossed.
Symptom
Errors similar to the following appear in the qradar.error log on the affected event processor:
[ecs-ep.ecs-ep] com.q1labs.sem.monitors.ECSQueueMonitor: [WARN] [NOT:0060005100][IPADDRESS/- -] [-/- -]ECS Queue Monitor has detected a total of 33397 dropped event(s). 923 event(s) were dropped in the last 60 seconds. EP Queues: 923 dropped event(s). MPC Queues: 0 dropped event(s).
[ecs-ep.ecs-ep] com.q1labs.sem.monitors.ECSQueueMonitor: [WARN] [NOT:0000004000][IPADDRESS/- -] [-/- -]EP Queue [RefDataDestination] has detected 930 dropped event(s) in the last 60 seconds and is at 0 percent capacity
Cause
Due to many rules that are updating the reference data, there could be congestion while the elements are updated in the reference sets, which results in new response requests being dropped. However, the error does not indicate that there are events dropped but that the responses sent to the reference sets are being dropped.
Resolving The Problem
To resolve the problem, we need to pull out a list of rules that are writing to reference sets.
- SSH in to the QRadar console.
- Run the following command
psql -Uqradar -c "select (xpath('//referenseSetResponse/@name',rule_data::text::xml))[1]::text as \"Add To Set\", (xpath('//removeFromReferenseSetResponse/@name',rule_data::text::xml))[1]::text as \"Remove from Set\", string_agg((xpath('/rule/name/text()',rule_data::text::xml))[1]::text,',') as \"Rules\" from custom_rule where rule_data::text ~* 'referenseSetresponse' group by (xpath('//referenseSetResponse/@name',rule_data::text::xml))[1]::text, (xpath('//removeFromReferenseSetResponse/@name',rule_data::text::xml))[1]::text ;" | less - Examine the output. You must determine which rules need a Response Limiter. See the following example output:
Add To Set | Remove from Set | Rules ----------------------------------------+----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------- Compromised Hosts | | Encoded Command Malicious Usage in a Programming Environment Critical Temptation Target | High Temptation Target | New Critical Temptation Target Detected High Priority Target | | New High Priority Target Detected High Temptation Target | Critical Temptation Target | New High Temptation Target Detected URL Access Watchlist | | Restricted URL Access Watchlist George | Critical Temptation Target | Critical Temptation Target Changed to Lower Temptation | High Priority Target | High Priority Target Changed to Low Priority | High Temptation Target | High Temptation Target Changed to Lower Temptation - Add a response limiter to the rule:
Result
The limiter prevents the rule from making so many changes that it causes response request congestion. Requests are no longer dropped and the error does not appear.
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtrAAA","label":"Rules"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
01 May 2024
UID
ibm17005987