IBM Support

QRadar: How does coalescing work in QRadar?

Question & Answer


Question

How does event coalescing work for log sources in QRadar? What data is kept and what is lost when events are coalesced? How are events displayed with coalescing enabled? How can I see how many events get coalesced?

Answer

Event Coalescing helps improve performance and reduce storage for non-critical security events where the full event payload does not need to be saved. As data comes in and is coalesced, a large burst of events can convert hundreds of thousands of events into only a few dozen records. This action is done while QRadar maintains the count of the number of actual events. Coalescing gives QRadar the ability to detect, enumerate, and track an attack on a huge scale. It also protects the performance of the pipeline by reducing the workload of the system, including storage requirements for those events. When events are received that match a specific criteria, QRadar can use coalescing to determine what to store from the event payload based on the log source settings in QRadar.

For example, a multitude of similar events created during a Denial Of Service attack can be converted from hundreds of thousands of events into only a few dozen records, while maintaining the count of the number of actual events received.

How does coalescing work?

Event data received by QRadar is processed in to normalized fields, along with the original payload. When coalescing is enabled the following five properties are evaluated to determine if a data source can be coalesced:
  • QRadar Identifier (QID)
  • Source IP
  • Destination IP
  • Destination port
  • Username
     
Event coalescing starts after three events have been found with matching properties within a 10 second window. Additional events that occur within the 10 second period are coalesced together, with a count of the events noted. For each record containing coalesced events, only the payload of the first coalesced event is retained.

For example, if 1,005 events are received by QRadar within a 10 second window. Each of the 1,005 events has the same  QRadar Identifier (QID), Source IP, Destination IP, Destination port, and Username and coalescing is enabled for the log source. The Log Activity tab represents these 1,005 events as follows: 
 
  • The first three events display in Log Activity as individual events with unique event payloads.
    Event 1: full event details saved.
    Event 2: full event details saved.
    Event 3: full event details saved.
  • The fourth event contains a unique payload; however, the remaining 1,001 events that also arrived within that 10 second period are coalesced and only the original payload for the fourth event is retained.
    Event 4: full event details saved.
    Event 5-1,005: event details are NOT saved to disk for each individual event. The count for the fourth event displays in the user interface as Multiple (1001). The value within parenthesis (x) indicates to the user that this log source coalesced a number of events as the  with a duplicate QRadar Identifier (QID), Source IP, Destination IP, Destination port, and Username in to the fourth event as these all occurred within the 10 second window. Events 5-1005 do not have unique event payloads stored on disk.

    Note: Rules created by users intended to count events do update the event count even if the event is coalesced. For example, a user creates a rule with the test and when at least 5 events are seen with the same Username. The events that occur within the multiple field (x) update the count tracked in QRadar and coalesced data can trigger an offense or rule response.


System-wide Event Coalescing Settings

QRadar provides the ability to disable coalescing if there is a requirement to retain all event payloads. This can be done either at system level, or on a per log source basis. 
Note:  At this time, the setting only sets this on log sources created after the change is made.   Any log sources created before the changes is made will need to be manually modified.

Procedure:

To disable coalescing at the system level:
  1. on the Admin tab, click the System Settings icon.
  2. Click Advanced.
  3. Under the System Settings heading, find the Coalescing Events setting.
  4. To disable Coalescing Events for all log sources, select No,
  5. Click Save, and close the window.
  6. From the Admin tab click Deploy Changes.
Event coalescing options for a specific log source
Coalescing can be enabled or disabled on each log source. Administrators can set coalescing when the log source is created or when editing an existing log source.

Procedure
To disable Coalescing for an existing log source:
  1. Click the Admin tab.
  2. Click the Log Sources icon.
  3. Double click on a log source to edit the configuration.
  4. Clear the Coalescing Events check box.
Event coalescing options for multiple log sources:
Coalescing can be enabled or disabled for multiple non-system log sources in the Log Source Management App.
  1. Click the Admin tab.
  2. Click the Log Sources icon.
  3. In the Filter panel on the left hand side, select a filter option.  If disabling Coalescing for all log sources is the goal, then select all of Log Source types except for "Custom rule Engine" and SIM Generic Log DSM".  Then scroll to the Coalescing Events filter and select yes.  
  4. Click the checkbox beside ID to select all of the log sources. 
  5. This will launch the blue bar.  On the blue bar, click on Edit.
  6. image-20240910144359-5
  7. This will launch the Multiple Log Sources screen.  Scroll down and select Coalescing Events.  Then switch the option to off
  8. Coalescing Events
  9. Click Save
  10. Click OK on the popup saying that the change was successful. 

What types of log sources should I consider disabling coalescing?

Log sources for DNS systems, Proxy Servers, Anti-Virus systems, Windows servers and Endpoints can be good candidates for turning coalescing off. These log sources often include additional event payload information beyond QRadar's normalized fields which can be unique to the event payload that an administrator would want captured and searchable for an investigation.

How can I see how many events get coalesced?

QRadar tracks the average and peak values of the raw and coalesced events per second in StatFilter
messages.
Jun 12 16:03:09 ::ffff:172.16.xxx.xxx [ecs-ec]
[type=com.q1labs.semsources.filters.stat.StatFilter][parent=Lab-primary.q1labs.lab:ecs-ec/EC/Processor2]]
com.q1labs.semsources.filters.stat.StatFilter: [INFO] [NOT:0000006000][172.16.xx.xx/- -] [-/- -]
Events per second: 1s:944,2591 (peak 2856,31613) (compression: 64%) 5s:1512,9355 (peak 1614,11524)
(compression: 84%) 10s:1327,7255 (peak 1569,7404) (compression: 82%) 30s:1337,5824 (peak 1493,7772)
(compression: 77%) 60s:1377,6124 (peak 1479,6986) (compression: 78%)
That was an example of a StatFilter message. A StatFilter message contains several statistics that are
averaged over various time ranges. To illustrate the information contained more clearly, this is that
same example, formatted over multiple lines:
Events per second:
1s:  944,2591  (peak 2856,31613) (compression: 64%)
5s:  1512,9355 (peak 1614,11524) (compression: 84%)
10s: 1327,7255 (peak 1569,7404)  (compression: 82%)
30s: 1337,5824 (peak 1493,7772)  (compression: 77%)
60s: 1377,6124 (peak 1479,6986)  (compression: 78%)
In each section of the message, information is displayed in the following order:
Time Period, Average Coalesced EPS, Average Raw EPS, Peak Coalesced EPS, Peak Raw EPS, Compression.
Components of StatFilter messages
Term Definition
Time Period The time period that the information is calculated over.
Average Coalesced EPS The average of the events per second calculated after coalescing. The average is calculated over readings taken during the time period.
Average Raw EPS The average of the events per second calculated before coalescing. The average is calculated over readings taken during the time period.
Peak Coalesced EPS The highest events per second seen during the time period, after coalescing.
Peak Raw EPS The highest events per second seen during the time period, before coalescing.
Compression QRadar events are compressed when stored on disk. This value is the percentage size of the compressed events compared to the original size.

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtEAAQ","label":"Log Activity"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Version(s)"}]

Document Information

Modified date:
10 September 2024

UID

swg21622709

Page Feedback