QRadar: Syslog Redirect Protocol FAQ

A. When should I use the Syslog Redirect protocol?

Syslog redirect is useful when administrators receive events sent to QRadar that include malformed headers, do not include a useful identifier for the event source, or use an internal or reserved IP address, such as 127.0.0.1 or 0.0.0.0. Syslog redirect is not intended to evaluate all events sent to a QRadar appliance as this protocol is intended to provide additional functionality for specific use cases where a legitimate identifier is not provided by the event source. Using the Syslog Redirect protocol to parse all events in QRadar that come through a Syslog forwarder or load balancer can increase the time it takes to parse an event and can lead to performance issues.

Use cases

1. The log source sends events with an non-valid syslog header.

Events from a log source that uses a RFC3164 or RFC5424 Syslog type will be correctly parsed by default. If the Syslog event payload contains a non-standard header, then Syslog Redirect can be used to substitute in a new header in front of the malformed header.

2. The log source is providing the wrong identifier in the header.
Intermediary hosts that forward events to QRadar might not include an identifiable Syslog header. In these cases, all events from the source appear to be coming in to the same log source. This is due to the event source, such as load balancers, syslog forwarders, or some form of log management system, etc.) can append their own header before forwarding events to QRadar. In some examples, events might contain a generic placeholder hostname. These events may originate from multiple sources, but will appear to have the same source in QRadar.

3. The log source interferes with internal message routing
QRadar uses 127.0.0.1 for internal auditing. A log source using 127.0.0.1 requires Syslog Redirect to replace this address with a new identifier.

B. Restrictions

The Syslog Redirect protocol is a single-threaded process in QRadar. This protocol is not intended to be used to parse all incoming events sent to a QRadar appliance, but select events that require a substitution to replace an incorrect log source identifier. Administrators are encouraged not to use Syslog Redirect as a replacement for correction of all events from a source, but leverage this protocol for event sources only where select substitutions are required.

C. How does the Syslog Redirect protocol work?

The syslog redirect protocol listens on a port (default 517) for incoming events. Using a specified regular expression (regex), the protocol will extract relevant information from the event payload and use it as the log source identifier. The substitution of the log source identifier occurs at the beginning of the event pipeline before the licensing component of ECS-EC (or ECS-EC-INGRES for QRadar 7.3.1) to ensure that events are not duplicated or counted against the deployment license twice. 

Important: Administrators must verify that they are not sending Syslog Redirect events to port 514. If your event source cannot be configured to send to port 517, you might need to use an intermediate appliance or iptables rule to properly route these events to port 517.

D. Scenarios

Assume we have events coming from multiple devices to a forwarding server. The forwarding server is configured to send events to QRadar on port 517 and a syslog redirect log source is configured to listen on port 517. Each of these events can appear as if they are being generated by the forwarding server. We want to be able to group the events coming from each unique device in to an individual log source to allow analysts to drill down to activity from specific hosts sending event data, even though they have all passed through the same forwarding server. The original source address used to identify the originating host is a name=value pair in the event payload. In the following examples, we are using 'orig=IP address' from the event payload. The regular expression used to complete the Log Source Identifier substitution is orig=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).

The syslog redirect protocol may encounter events that:

• Already have a valid syslog header
• Do not have a valid syslog header
• Match the specified regex
• Do not match the specified regex

NO VALID HEADER, REGEX MATCH
The event is forwarded to the event collector with a new packet IP. QRadar will receive this event with a packet IP of 10.199.56.21

Incoming Event
<13> 127.0.0.1 forwardinghost1 loc=10814950|time=26Mar2017 14:03:22|action=accept|orig=10.199.56.21 |i/f_dir=inbound|i/f_name=Exp1-1|has_accounting=0|uuid=<51xxxxxx,00xxxxxx,15xxxxxx,00xxxxxx>|product=VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx};mgmt=lab1;date=1364192986;policy_name=examplepolicy_cps]|src=10.203.15.141|s_port=41269|dst=10.207.36.152|service=443|proto=tcp|rule=150

Forwarded Event
<13> 127.0.0.1 forwardinghost1 loc=10814950|time=26Mar2017 14:03:22|action=accept|orig=10.199.56.21|i/f_dir=inbound|i/f_name=Exp1-1|has_accounting=0|uuid=<5151aaaa,000102eb,1538c70a,0000ffff>|product=VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx};mgmt=fm55lab01;date=1364192986;policy_name=examplepolicy_cps]|src=10.203.15.141|s_port=41269|dst=10.207.36.152|service=443|proto=tcp|rule=150

VALID HEADER, REGEX MATCH
The event is forwarded to the event collector with a new syslog header and a new packet IP
QRadar will receive this event with a packet IP of 10.199.56.21

Incoming Event
<189>Jun 29 16:30:07 Message forwarded from 10.48.225.4: loc=10814950|time=26Mar2013 14:03:22|action=accept|orig=10.199.56.21|i/f_dir=inbound|i/f_name=Exp1-1|has_accounting=0|uuid=<5151aaaa,000102eb,1538c70a,0000ffff>|product=VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx};mgmt=lab1;date=1364192986;policy_name=examplepolicy_cps]|src=10.203.15.141|s_port=41269|dst=10.207.36.152|service=443|proto=tcp|rule=150

Forwarded Event
<55>May 23 10:56:31 10.199.56.21 <189>Jun 29 16:30:07 Message forwarded from 10.48.225.4: loc=10814950|time=26Mar2013 14:03:22|action=accept|orig=10.199.56.21|i/f_dir=inbound|i/f_name=Exp1-1|has_accounting=0|uuid=<5151aaaa,000102eb,1538c70a,0000ffff>|product=VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx};mgmt=lab1;date=1364192986;policy_name=examplepolicy_cps]|src=10.203.15.141|s_port=41269|dst=10.207.36.152|service=443|proto=tcp|rule=150

VALID HEADER, NO REGEX MATCH
The event enters the event collector as-is, to be picked up as syslog. QRadar will receive this event as syslog, with a packet IP of 10.48.255.4

Incoming Event
<189>Jun 29 16:30:07 Message forwarded from 10.48.225.4: loc=10814950|time=26Mar2013 14:03:22|action=accep|i/f_dir=inbound|i/f_name=Exp1-1|has_accounting=0|uuid=<5151aaaa,000102eb,1538c70a,0000ffff>|product=
VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx};mgmt=lab1;date=1364192986|policy_name=examplepolicy_cps]|src=10.203.15.141|s_port=41269|dst=10.207.36.152|service=443|proto=tcp|rule=150

Forwarded Event
<189>Jun 29 16:30:07 Message forwarded from 10.48.225.4: loc=10814950|time=26Mar2013 14:03:22|action=accep|i/f_dir=inbound|i/f_name=Exp1-1|has_accounting=0|uuid=<5151aaaa,000102eb,1538c70a,0000ffff>|product=
VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx};mgmt=lab1;date=1364192986;policy_name=examplepolicy_cps]|src=10.203.15.141|s_port=41269|dst=10.207.36.152|service=443|proto=tcp|rule=150

E. Using Identifier Regex and Format Strings

IDENTIFIER REGEX
After you have chosen which value from the payload information is to be used as the Log Source identifier, you can create a regex to parse it out of the payload. In the above examples, we wanted the originating IP address of the logs, which looks like “|orig=10.199.56.21|”. To extract the IP, we would use orig=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) as the Log Source Identifier Regex

Note the parentheses around the portion of the pattern containing the actual address. Everything contained by the parentheses is Capture Group 1. Capture groups can be referenced by number preceded by a backslash, and used in the format string. ie, a \1 represents the part of the payload that matches \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}. \1 is also the default value.

A regex can also have more than one capture group. For example, we might wish to include the time in the log source identifier. We would use time=(\w+).*?orig=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) as the Log Source Identifier Regex.
In this case, \1 matches the first word after “time=”, and \2 matches the IP.

It is important to note that if the regex is too broad here, undesired results may be grouped under the source. Conversely, if it is too precise it my not include all of the desired values. Use discretion in choosing the regex to work best for your environment.

FORMAT STRINGS

The formatting string allows you to combine the capture groups in any order with any separator (IE: \1 \2\3, \2+\1, \1-\2, \1::\2::\3, etc) to create a Log Source Identifier from them. This will be used as the log source name following the same convention as autodiscovered devices. Device @ <format_string>.

EXAMPLES

regex: orig=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})

format: \1 LSI: Check Point @ 10.199.56.21

Alternate examples with multiple capture groups

The following examples show how capture groups can be used to format the Log Source Identifier (LSI). The format string also allows the protocol to use the packet source in the log source name by using the $PIP$ tag. This allows us to include a forwarding server address in the LSI.
 

• Regex for capture groups: time=(\w+).*?orig=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
  • Format: \1 :: \2
    Log Source Identifier as time first, then originating IP address: Check Point @ 26Mar2013 :: 10.199.56.21
  • Format: \2 :: \1
    Log Source Identifier as originating IP address first, then time: Check Point @ 10.199.56.21 :: 26Mar2013
  • Format: \1-\2-$PIP$
    Log Source Identifier as time, originating IP address, and packet address: Check Point @ 26Mar2013-10.199.56.21-10.48.225.4