QRadar: About flows and the difference between QFlow Collector and QRadar Event Collector

QRadar collects network activity information, or what is referred to as "flow records".  Flows represent network activity by normalizing ip addresses, ports, byte and packet counts, as well as other details, into "flow", which effectively represent a session between two hosts.

For sessions that span multiple "intervals" (minutes), the pipeline reports a record at the end of each minute with the current metrics for each flow - bytes, packets etc.  For this reason, you will see multiple records (per minute) in QRadar with the same "First Packet Time", but with "Last Packet Time" values that increment through time.

A flow is different from an event, in that flows (for the most part) will have a start and end time, or a life of multiple seconds.  For example, when you connect to a website, the communication will include HTML files, images, flash files, longer file downloads, etc, and may take some time to transfer the data.  An Event, in contrast, represents a single event on the network, such as the login action of a VPN session or a firewall deny by someone trying to connect to a network.

The component in QRadar that collects and 'creates' flow information is known as "qflow".  QFlow can process flows from multiple sources.  Sources that include packet data by connecting a span/monitor port, or network tap, to a Flow collector are referred to as "internal sources". The "internal" type of collection normally requires a dedicated collection appliance (1101, 12xx, 13xx), that varies in capacity based on traffic rates. These sources provide raw packet data to a monitoring port on the QRadar flow collector, which then converts these packet details into "flow records".  QRadar does not keep the entire packet payload, but it can capture a snapshot of each "flow", which includes packets from the beginning of the communication, from the first few packets - this is known as "payload" or "content capture". Monitoring ports range from integrated network interfaces for lower data rates, up to about 200Mbps, to a dedicated monitoring card (Endace or Napatech) which can support up to 2Gbps.

Qflow can also process session or flow information from routers and other network devices, or what is called "external sources".  The formats supported are Cisco Netflow (v5, v7, v9), IPFIX, JFlow and sflow.  We also support session information from "Packeteer", which is an external source, but also includes packet payload.   External sources can be sent to a dedicated flow collector, but can also be sent to a "flow processor" (17xx appliance).  This is because 'external' sources do not require as much CPU utilization to process as we are not processing every packet to build flows.  In this configuration, a customer may have dedicated flow collector AND a flow processor, both receiving and creating flow data.   In smaller environments (less than 50Mbps), a single QRadar appliance may run all the data processing.

One of the major differences between event and network data, is that an event, which typically is a log of a particular action, happens at a single point in time, and then is complete.   A flow, in contrast, can have a life span that can last seconds, minutes, hours or days, depending on the activity within the session.  For example, a web request may pull down multiple files, images, ads, etc., and last 5-10 seconds, or a user watching a Netflix movie could have that session last for hours, depending on the length of the movie.


For more information about flows, see: https://www.securitylearningacademy.com/enrol/index.php?id=4863.