Troubleshooting
Problem
This article explains how to understand PIPELINE STATUS messages in QRadar application logs.
The PIPELINE STATUS messages in the /var/log/qradar.log file indicate the state of the queues of the pipeline, and provides insight into portions of the pipeline that require attention.
[ecs-ep.ecs-ep] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=<hostname>:ecs-ep/EP/Processor2]]
com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] ---- PIPELINE STATUS -- Initiated From: EPCRE
Symptom
The symptoms can vary, typically dropped events or events sent to storage.
Example logs for dropped events:
[ecs-ep.ecs-ep] [5bbe913a-55e2-494e-b712-4a0ecf0ec78e/SequentialEventDispatcher] com.q1labs.sem.monitors.ECSQueueMonitor: [WARN] [NOT:0060005100][x.x.x.x/- -] [-/- -]ECS Queue Monitor has detected a total of 10228 dropped event(s). 1252 event(s) were dropped in the last 60 seconds. EP Queues: 1252 dropped event(s). MPC Queues: 0 dropped event(s).
Example logs for stored events:
[ecs-ep.ecs-ep] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=<hostname>:ecs-ep/EP/Processor2]] com.q1labs.semsources.cre.CRE: [WARN] [NOT:0080004101][x.x.x.x/- -] [-/- -]Custom Rule Engine has sent a total of 787911838 event(s) directly to storage. 99125 event(s) were sent in the last 60 seconds. Queue is at 100 percent capacity.
Cause
The cause can be determined based on the part of the pipeline shown after the Initiated From, field. Common values include:
- DSMFilter
- Data is normalized incorrectly (i.e. Parse a Linux PAM log as a Unity One event).
- Expensive custom properties with Parse in Advanced (Optimized) enabled.
- Incorrect configuration of a log source causing parsing or normalization to take a long time.
- Expensive Log Source Extension being used.
- Ingesting partial events. This typically happens when the payload size is not large enough and the log source is sending half of an event from the same log source identifier. Ensuring that all payloads are complete is important.
- TrafficAnalysis
- Data is coming from a device that is not easy to identify, likely a mis-configured log source.
- An unknown log source that QRadar does not support (or a subset of logs we don't) causing all of our auto-detection to take a long time.
- EPCRE
- Some rule or rules might not be well constructed and are therefore expensive on the engine.
- Payload-related rule tests might also be expensive and cause performance degradation.
Environment
Any QRadar host where Performance Degradation is occurring (stored or dropped events for example), such as an EC, EP, or Console.
The "PIPELINE STATUS" messages are logged in /var/log/qradar.log
Resolving The Problem
Use the following command to simplify the output:
grep -i 'Pipeline' /var/log/qradar.log | sed -s 's/::fff.*-]//' | less
A sample output would look like this:
---- PIPELINE STATUS -- Initiated From: EPCRE
MPC (Filters: 0.00 pc) (Queues: 0.00 pc) (Sources: 0.00 pc)
EC_Ingress (Filters: 0.00 pc) (Queues: 0.00 pc) (Sources: 0.00 pc)
EC (Filters: 0.00 pc) (Queues: 0.00 pc) (Sources: 0.00 pc)
EP (Filters: 55.56 pc) (Queues: 1.14 pc) (Sources: 0.00 pc)
100.00 pc - Filter:CRE EP (100000/100000)
0.66 pc - Queue:Q1From_EC_via_TCPIP (66/10000)
99.60 pc - Queue:Processor1 (249/250)
42.80 pc - Queue:AnalyticStack/NTAProcessor (107/250)
0.80 pc - Queue:AnalyticStack (2/250)
100.00 pc - Queue:EntryRouterStack (250/250)
100.00 pc - Queue:Processor2 (250/250)
In this case, the Filter: CRE EP queue is at 100%, which would mean that the Custom Rule engine is struggling to cope with the load, likely due to expensive rules.
The following documentation addresses the most common pipeline Performance issues.Expensive DSMs and CEPs:
Expensive custom rules:
Related Information
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
20 December 2022
UID
ibm16848869