TCP Multiline Syslog protocol configuration options

The TCP Multiline Syslog protocol is a passive inbound protocol that uses regular expressions to identify the start and end pattern of multiline events.

The following example is a multiline event:
06/13/2012 08:15:15 PM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=5156
EventType=0
TaskCategory=Filtering Platform Connection
Keywords=Audit Success
Message=The Windows Filtering Platform permitted a connection.
Process ID: 4
Application Name: System
Direction: Inbound
Source Address: <IP_address>
Source Port: 80
Destination Address: <IP_address>
Destination Port:444
The following table describes the protocol-specific parameters for the TCP Multiline Syslog protocol:
Table 1. TCP Multiline Syslog protocol parameters
Parameter Description
Protocol Configuration TCP Multiline Syslog
Log Source Identifier Type an IP address or hostname to identify the log source. To use a source name instead of a log source identifier, select Use Custom Source Name and enter values for the Source Name Regex and Source Name Formatting String parameters.
Note: These parameters are only available if Show Advanced Options is set to Yes.
Listen Port The number of the port that accepts incoming TCP Multiline Syslog events. The default listen port is 12468.
To edit the port number, complete the following steps:
  1. Enter the new port number for the protocol.
  2. Click Save.
  3. On the Admin tab, click Advanced > Deploy Full Configuration.
    Attention: When administrators click Deploy Full Configuration, the system restarts all services, which can create in a gap in data collection until the deployment completes.
Aggregation Method The method that you use to aggregate your TCP Multiline Syslog data. You can choose one of the following methods:
ID-Linked Multiline
Processes multiline event logs that contain a common value at the start of each line.
Start/End Matching
Processes multiline events by specifying start and end patterns for the events.
Event Start Pattern

This parameter is available when you set the Aggregation Method parameter to Start/End Matching.

The regular expression (regex) that is required to identify the start of a TCP multiline event payload. Syslog headers typically begin with a date or timestamp. The protocol can create a single-line event that is based on solely on an event start pattern, such as a timestamp. When only a start pattern is available, the protocol captures all the information between each start value to create a valid event.
Event End Pattern

This parameter is available when you set the Aggregation Method parameter to Start/End Matching.

This regular expression (regex) that is required to identify the end of a TCP multiline event payload. If the syslog event ends with the same value, use a regular expression to determine the end of an event.

When only an Event Start Pattern is used and the sending device sends a few events (low EPS) in an interval, then the last sent event is not processed until the pipeline detects a new Event Start Pattern. For example, when a single "Test event" is sent, it might be invisible in the QRadar® Log Activity without adding an Event End Pattern. To circumvent this scenario, send 2 to 3 test events at a time. Without an Event End Pattern, the last event is not captured until a new Event Start Pattern is detected.
Message ID Pattern

This parameter is available when you set the Aggregation Method parameter to ID-Linked.

This regular expression (regex) is required to filter the event payload messages. The TCP multiline event messages must contain a common identifying value that repeats on each line of the event message.
Event Formatter Use the Windows Multiline option for multiline events that are formatted specifically for Windows.
Show Advanced Options

The default is No. Select Yes if you want to customize the event data.
Use Custom Source Name

This parameter is available when you set Show Advanced Options to Yes.

Select the checkbox if you want to customize the source name with regex.
Source Name Regex

This parameter is available when you check Use Custom Source Name.

The regular expression (regex) that captures one or more values from event payloads that are handled by this protocol. These values are used with the Source Name Formatting String parameter to set a source or origin value for each event. This source value is used to route the event to a log source with a matching Log Source Identifier value.
Source Name Formatting String

This parameter is available when you enable Use Custom Source Name.

You can use a combination of one or more of the following inputs to form a source value for event payloads that are processed by this protocol:
  • One or more capture groups from the Source Name Regex. To refer to a capture group, use \x notation where x is the index of a capture group from the Source Name Regex.
  • The IP address where the event data originated from. To refer to the packet IP, use the token $PIP$.
  • Literal text characters. The entire Source Name Formatting String can be user-provided text. For example, if the Source Name Regex is 'hostname=(.*?)' and you want to append hostname.com to the capture group 1 value, set the Source Name Formatting String to \1.hostname.com. If an event is processed that contains hostname=ibm, then the event payload's source value is set to ibm.hostname.com, and QRadar routes the event to a log source with that Log Source Identifier.
Use as a Gateway Log Source

This parameter is available when you set Show Advanced Options to Yes.

When selected, events that flow through the log source are routed to other log sources, based on the source name tagged on the events.

When this option is not selected and Use Custom Source Name is not enabled, incoming events are tagged with a source name that corresponds to the Log Source Identifier parameter.
Flatten Multiline Events into Single Line

This parameter is available when you set Show Advanced Options to Yes.

Shows an event in one single line or multiple lines.
Retain Entire Lines during Event Aggregation

This parameter is available when you set Show Advanced Options to Yes.

If you set the ID-Linked Multiline method for the Aggregation Method parameter, then this parameter can modify aggregated event data outputs.

If you enable Retain Entire Lines during Event Aggregation, then all parts of events are kept when aggregating events with the same ID pattern. If you don't enable this parameter, the part of the events before the Message ID Pattern are discarded when the events are aggregated.
Time Limit The number of seconds to wait for additional matching payloads before the event is pushed into the event pipeline. The default is 10 seconds.
Initial Number of Threads The initial number of threads to use for formatting and posting events.
Maximum Number of Threads The maximum number of threads to use for formatting and posting events. When the task queue is full, more threads are created up to the value set by the Maximum Number of Threads parameter.
Enabled

Select this checkbox to enable the log source.
Credibility

Select the credibility of the log source. The range is 0 - 10.

The credibility indicates the integrity of an event or offense as determined by the credibility rating from the source devices. Credibility increases if multiple sources report the same event. The default is 5.
Target Event Collector

Select the Event Collector in your deployment to host the TCP Multiline Syslog listener.
Coalescing Events

Select this checkbox to enable the log source to coalesce (bundle) events.

By default, automatically discovered log sources inherit the value of the Coalescing Events list from the System Settings in QRadar. When you create a log source or edit an existing configuration, you can override the default value by configuring this option for each log source.
Store Event Payload

Select this checkbox to enable the log source to store event payload information.

By default, automatically discovered log sources inherit the value of the Store Event Payload list from the System Settings in QRadar. When you create a log source or edit an existing configuration, you can override the default value by configuring this option for each log source.

TCP Multiline Syslog protocol configuration use cases

To set the TCP Multiline Syslog listener log source to collect all events that are sent from the same system, follow these steps:
  1. Leave Use As A Gateway Log Source and Use Custom Source Name cleared.
  2. Enter the IP address of the system that is sending events in the Log Source Identifier parameter.
Figure 1. A QRadar log source collects events sent from a single system to a TCP Multiline Syslog Listener
TCP Multiline Syslog protocol use case 1
If multiple systems are sending events to the TCP Multiline Syslog listener, or if one intermediary system is forwarding events from multiple systems and you want the events to be routed to separate log sources based on their syslog header or IP address, select the Use As A Gateway Log Source checkbox.
Note: QRadar checks each event for an RFC3164 or RFC5424-compliant syslog header, and if present, uses the IP or hostname from that header as the source value for the event. The event is routed to a log source with that same IP or hostname as its Log Source Identifier. If no such header is present, QRadar uses the source IP value from the network packet that the event arrived on as the source value for the event.
Figure 2. Separate QRadar log sources collect events sent from multiple systems to a TCP Multiline Listener, by using the syslog header.
TCP Multiline Syslog protocol use case 2A
Figure 3. Separate QRadar log sources collect events sent from multiple systems and forwarded through an intermediate system to a TCP Multiline Listener, by using the syslog header.
TCP Multiline Syslog protocol use case 2B

To route events to separate log sources based on a value other than the IP or hostname in their syslog header, follow these steps:

  1. Select the Use Custom Source Name checkbox.
  2. Configure a Source Name Regex and Source Name Formatting String to customize how QRadar sets a source name value for routing the received events to log sources.
Figure 4. Separate QRadar log sources collect events sent from multiple systems and forwarded through an intermediate system to a TCP Multiline Listener, by using the Source Name Regex and Source Name Formatting String.
TCP Multiline Syslog protocol use case 3