IBM Support

Event Processing Pipeline



General overview of the Event Pipeline and Processes

Resolving The Problem

QRadar Event Collection Pipeline Components:
  1. Event Collector (EC)
  2. Event Processor (EP)
  3. Magistrate Processing Core (MPC)

1. Event Collector (EC):
QRadar will accept events from a variety of sources, using syslog, syslog-tcp, snmp, etc. as well as having the ability to establish outbound connections and retrieve events, using scp, sftp, ftp, JDBC, Checkpoint OPSEC, SMB/CIFS, etc. As these events are received, they are placed into input queues for processing. The queue sizes will vary based on the protocol/method used, and from these queues, events are fed to the Event Parsers/DSMs.

Known Log Sources (based on the source IP address or hostname in the header) are parsed and coalesced into records, and unknown sources are redirected to Traffic Analysis for Log Source autodetection. If new sources are found, a configuration request message is sent back to the QRadar console to have the new source added, unless you have disabled Autodetection, or if you have reached your Log Source limit, which is controlled by your licenses.

EPS, License Rates:
Events are pulled off of the source queues at a rate dictacted by your QRadar licenses, and each event processor can have a distinct licensed event rate. If you are receiving events at higher rate than your license, they will sit in the source queues until the rate drops. However, if you continue to send at a higher rate than your license, and the queue is filled, your system will start dropping events, with a warning that you have reached your license.

Data Collection, Sources:
Each Log Source type has its own queue:
  • Syslog: 100000
  • Syslog (TCP): 100000
  • Windows Event Log: 100000
  • Windows IIS/Exchange/DHCP/etc: 10000 each
  • JDBC: 10000
  • SDEE/EStreamer: 10000 each
  • Log File Protocol (scp/sftp/ftp): 50000

The queues are processed in sequence, with parsing servicing each queue in series. For example, if a JDBC connection was to burst to 5000 eps for a few seconds, it's possible that the queue would fill and drop events, while the syslog queue, at a burst of 10000 eps for 5-10 seconds would not, since the queue size is much larger. This varying size is due to the fact that most sources come in over UDP. Note, that ALE connections will also use Syslog UDP to send events in, not the Windows Event log or other Windows methods. .

Parsing/Normalization & Coalescing:
Events are parsed and then coalesced based on common patterns across events. Once 4 events are seen with the same source ip, destination ip, destination port and username, subsequent messages for up to 10 seconds of the same pattern are coalesced together. This is done to reduce duplicate data being stored. Note, that some customers are required to disable coalescing to meet certain audit (PCI/HIPPA) requirements. In these instances, you should expect that data retention requirements will increase, as you are no longer coalescing multiple records into fewer.

Traffic Analysis (Auto Detection):
Events that come into the system from new sources that have not been previously detected are redirected to the traffic analysis (autodetection) engine. These events are run through log sources types that are defined in traffic analysis. A minimum number of messages per minute are required to create a Log Source, and this number may be different for multiple devices. A number of devices are very common from one sort to another, such as AIX, Solaris, Linux, and may not be enabled in traffic analysis. In these instances, or in cases where event rates are low from particular devices, we recommend you create the Log Sources manually.

If you are having issues where devices are getting detected incorrectly, especially if they are Log Source types not even on your network, it is possible with the assistance of IBM Support, to remove some of these devices from Traffic Analysis. If you are having this issue, please contact support.

Identity Updates:
In addition to parsing our normalized fields, some log sources and event types will also update identity. Identity records are typically generate from messages that indicate a successful connection/authentication to a resource, such as an SSH login , windows domain authentication, etc. Other events may still have a username parsed out, such as an antivirus update from a windows host, or program started from windows, but do not really indicate an authentication. These kinds of events would not generate an identity update.

Identity updates are sent from the event collectors parsing code directly back to the console and do not go through the remaining event pipeline. This is stored with the host information there, under the Asset tab of the QRadar user interface.

A few users have also asked if VA data comes through the event pipeline. This is not the case: VA/Scanner information can be retrieved and processed on any system, which is then parsed by the vis component, sent back to the console, then stored with the Asset information as well.

2. Event Processor (EP):
The Event Processor accepts normalized messages from the EC and runs them through the custom rules as defined in the Rules section of the UI. As events are processed and matched, the EP will generate notifications, syslog, SNMP, email messages, new events and offenses (on the console in MPC) as required.

Custom Rules Engine:
The Custom Rules Engine (CRE) is responsible for processing events received by QRadar and comparing them against defined rules, keeping track of systems involved in incidents over time, generating notifications to users and generating offenses. When a rule has been confirmed as match, the event processor will execute what is defined in the rule response section. Note, that if you have configured your system to generate an email or SNMP message, that these messages are delivered from the Event Processor where the events are processed, not the console itself. In this situation, you must ensure that the machine has to deliver these messages, either via an email relay or to the SNMP server being sent to.

Context & Routing:
The context and routing engine is responsible for passing events off to storage and on to MPC when required.

Event (Ariel) Storage:
Ariel Storage is where data gets written from the event pipeline out to disk. Data is written out to disk on any system that has an event processor. Ariel storage settings are configured on the console in System Settings - Ariel Database Settings. Note, that currently, ariel retention setting is a global configuration value, so that all event processors are configured with the same retention, though this may change in the future. However, QRadar will only delete data as required, to maintain 85% free disk space. The system will collect data until 85% is reached, then it begins compressing that data. As all data up to 6 hours old is compressed, it then goes back and starts deleting the oldest data, working ahead compressing and deleting until the retention threshold is hit. If you overcommit your storage, you may end up using more than 85%, and when QRadar hits 95% usage on /store/, it will shutdown and stop collecting.

3. Magistrate Processing Core (MPC):
The Magistrate Processing Core (MPC) is responsible for correlating offenses with event notifications from multiple event processors, monitoring active offenses, generating email notifications when required, and providing access to offense information to the end user via the Offenses tab.

Offense Rules:
Similar to the Event CRE, Offense rules have the ability to monitor offenses in qradar and take action, typically email messages, once they occur.

Offense Management:
(updated for 7.x) The system will monitor up to 2500 active offenses at any one time and has recall slots for 500 offenses that have been idle for more than 30 minutes. Inactive (no updates for 5 days) offenses are limited to 100,000. As offenses go periodically dormant (no updates in 30 minutes) and later updated, the offense manager will update up to 500 of these offenses.

Offense Storage:
Offense information is stored in the Postgres DB.

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"Event Pipeline","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.2;7.3","Edition":"Enterprise","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
10 May 2019