Examples of parsing issues

Converting a protocol

The following example shows a typical protocol conversion that searches for TCP, UDP, ICMP, or GRE anywhere in the payload. The search pattern is surrounded by any word boundary, for example, tab, space, end of line. Also, the character case is ignored:

<pattern id="Protocol" case-insensitive="true" xmlns="">
<![CDATA[\b(TCP|UDP|ICMP|GRE)\b]]>
</pattern> 
<matcher field="Protocol" order="1" pattern-id="Protocol" capture-group="1" />

Making a single substitution

The following example shows a substitution that parses the source IP address, and then overrides the result and sets the IP address to 192.0.2.1, ignoring the IP address in the payload.

This example assumes that the source IP address matches something similar to SrcAddress=203.0.113.1 followed by a comma:

<pattern id="SourceIp_AuthenOK" xmlns=""> 
<![CDATA[SrcAddress=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}),]]>
</pattern>

<matcher field="SourceIp" order="1" pattern-id="SourceIp_AuthenOK" 
capture-group="192.0.2.1" enable-substitutions="true"/>

Generating a colon-separated MAC address

QRadar® detects MAC addresses in a colon-separated form. Because all devices might not use this form, the following example shows how to correct that situation:

<pattern id="SourceMACWithDashes" xmlns="">
    <![CDATA[SourceMAC=([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-
    ([0-9a-fA-F]{2})-([0-9a-fA-F]{2})-([0-9a-fA-F]{2})]]>
</pattern> 
 <matcher field="SourceMAC" order="1" pattern-id=" 
    SourceMACWithDashes" capture-group="\1:\2:\3:\4:\5:\6" />

In the preceding example, SourceMAC=12-34-1a-2b-3c-4d is converted to a MAC address of 12:34:1a:2b:3c:4d.

If the dashes are removed from the pattern, the pattern converts a MAC address and has no separators. If spaces are inserted, the pattern converts a space-separated MAC address.

Combining IP address and port

Typically an IP address and port are combined into one field, which is separated by a colon.

The following example uses multiple capture groups with one pattern:

pattern id="SourceIPColonPort" xmlns="">
<! [CDATA[Source=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):([\d]{1,5})]]>
</pattern> 

<matcher field="SourceIp" order="1" pattern-id="SourceIPColonPort" capture-group="1" /> 
<matcher field="SourcePort" order="1" pattern-id="SourceIPColonPort" capture-group="2" />

Modifying an Event Category

A device event category can be hardcoded, or the severity can be adjusted.

The following example adjusts the severity for a single event type:

<event-match-single event-name="TheEvent" device-event-category="Actual Category" severity="6" send-identity="UseDSMResults" />

Suppressing identity change events

A DSM might unnecessarily send identity change events.

The following examples show how to suppress identity change events from being sent from a single event type and a group of events.

// Never send identity for the event with an EventName of Authen OK 
<event-match-single event-name="Authen OK" device-event-category="ACS" 
severity="6" send-identity="OverrideAndNeverSend" />

// Never send any identity for an event with an event name starting with 7, 
followed by one to five other digits: 
<pattern id="EventNameId" xmlns=""><![CDATA[(7\d{1,5})]]>
</pattern>

<event-match-multiple pattern-id="EventNameId" capture-group-index="1" 
device-event-category="Cisco Firewall" severity="7" 
send-identity="OverrideAndNeverSend"/> 

Formatting event dates and time stamps

A log source extension can detect several different date and time stamp formats on events.

Because device manufacturers do not conform to a standard date and time stamp format, the ext-data optional parameter is included in the log source extension to allow the DeviceTime to be reformatted. The following example shows how an event can be reformatted to correct the date and time stamp formatting:

<device-extension> 
<pattern id="EventName1">(logger):</pattern> 
<pattern id="DeviceTime1">time=\[(\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2})\]</pattern> 
<pattern id="Username">(TLSv1)</pattern> 

<match-group order="1" description="Full Test"> 
   <matcher field="EventName" order="1" pattern-id="EventName1_Pattern" capture-group="1"/> 
   <matcher field="DeviceTime" order="1" pattern-id="DeviceTime1_Pattern" 
   capture-group="1" ext-data="dd/MMM/YYYY:hh:mm:ss"/> 
   <matcher field="UserName" order="1" pattern-id="Username_Pattern" capture-group="1"/>
</match-group>
</device-extension>

Multiple Log Formats in a Single Log Source

Occasionally, multiple log formats are included in a single log source.

May 20 17:15:50 kernel: DROP IN=vlan2 OUT= MAC= SRC=<Source_IP_address> 
DST=<Destination_IP_address> PROTO=UDP SPT=1900 DPT=1900
May 20 17:16:26 <server>[22331]: password auth succeeded for 'root' from <IP_address>
May 20 17:16:28 <server>[22331]: exit after auth (root): Exited normally </br>
May 20 17:16:14 <server>[22331]: bad password attempt for 'root' from <IP_address>:3364

For example, there are 2 log formats: one for firewall events, and one for authentication events. You must write multiple patterns for parsing the events. You can specify the order to be parsed. Typically, the more frequent events are parsed first, followed by the less frequent events. You can have as many patterns as required to parse all of the events. The order variable determines what order the patterns are matched in.

The following example shows multiple formats for the following fields EventName and UserName

Separate patterns are written to parse each unique log type. Both of the patterns are referenced when you assign the value to the normalized fields.

<pattern id="EventName-DDWRT-FW_Pattern" xmlns=""><![CDATA[kernel\:\s(.*?)\s]]></pattern>
<pattern id="EventName-DDWRT-Auth_Pattern" xmlns=""><![CDATA[sdrophear\[\d{1,5}\]|:\s(.*?\s.*?)\s]]>
</pattern>

<pattern id="UserName_DDWRT-Auth1__Pattern" xmlns=""><![CDATA[\sfor\s\'(.*?)\'s]]></pattern>
<pattern id="UserName_DDWRT-Auth2__Pattern" xmlns=""><![CDATA[\safter\sauth\s\((.*?)\)\:]]></pattern>

<match-group order="1" description="DD-WRT Device Extensions xmlns=""> 
   <matcher field="EventName" order="1" pattern-id="EventName-DDWRT-FW_Pattern" capture-group="1"/> 
   <matcher field="EventName" order="2" pattern-id="EventName-DDWRT-Auth_Pattern" capture-group="1"/> 
   
   <matcher field="UserName" order="1" pattern-id="UserName-DDWRT-Auth1_Pattern" capture-group="1"/> 
   <matcher field="UserName" order="2" pattern-id="UserName-DDWRT-Auth2_Pattern" capture-group="1"/> 
   

Parsing a CSV log format

To parse a log file that is in CSV format, use the Generic List expression type that is available in the DSM Editor. For more information , see Expressions in Generic List format for structured data (https://www.ibm.com/docs/en/qsip/7.5?topic=editor-expressions-in-generic-list-format-structured-data).

Event,User,Source IP,Source Port,Destination IP,Destination Port
Failed Login,<Username>,<Source_IP_address>,1024,<Destination_IP_address>,22 
Successful Login,<Username>,<Source_IP_address>,1743,<Destination_IP_address>,110 
Privilege Escalation,<Username>,<Source_IP_address>,1028,<Destination_IP_address>,23