Data Parser overview

Edit online

Instead of manually creating a data source type to fix parsing issues or extend support for new log source types, use the Data Parser. The Data Parser provides different views of your data. You use the Data Parser to extract and override fields.

The Data Parser provides the following views:

Payloads editor

The Payloads pane shows you raw event data. Use sample event payloads to test the behavior of the data source type, and then the Payloads pane shows you the data that you capture in real time.

All sample events are sent from the workspace to the data source type simulator, where properties are parsed and QID maps are looked up. The results are displayed in the parsing output table.

Paste event data into the Payloads pane or edit data directly. When you override properties on the Properties tab, matches that are in the payload are highlighted in the Payloads pane. Overridden system properties are also highlighted in the Payloads pane.

You can specify a custom delimiter that makes it easier for the QRadar® product to ingest multiline events. To ensure that your event is kept intact as a single multiline event, select the Override event delimiter checkbox to separate the individual events based on another character or sequence of characters. For example, if your configuration is ingesting multiline events, you can add a special character to the end of each distinct event in the Payloads pane, and then identify this special character as the event delimiter.

The QRadar product can suggest regular expressions (regex) when you enter event data in the Payloads pane. If you are not familiar with creating regex expressions, use one of the following methods to generate your regex.
  • Highlight the payload text that you want to capture, right click the information, and click Extract to new property or Extract to <existing_property_name>. Then, in the Properties tab, you can click Suggest regex to generate an expression.
  • Override an existing property, select Regex as the expression type, and then click Suggest regex to generate an expression.
If the QRadar product cannot generate a suitable regex for your data sample, a system message appears.
Tip: The regex generator works best for fields in well-structured event payloads. If your payload consists of complex data from natural language or unstructured events, the regex generator might not be able to parse it and does not return a result.

Parsing output table

The parsing output table simulates how the payloads in the workspace appear in the Data Explorer viewer. The Parsing Status column indicates whether your event properties are successfully parsing. Every standard property that is supported is displayed. The fields that are marked with an asterisk (*), for example, Event name, Severity, Low-level category, and QID, are populated from the QID map. Fields that are populated from the QID map cannot be parsed verbatim from the raw events data in the workspace, so they cannot be defined or edited. You can adjust their values by selecting the corresponding event ID and category combination from the Event Mappings tab. Then click Edit to remap an event to a different QID record that exists in the system or to a newly created QID.

Important: When an event is parsed successfully, that means that the Event ID and Event Category are extracted from the payload. Set an Event ID so that the system properties are parsed correctly.

Click the Customize columns icon to select which columns to show or to hide in the parsing output table and to reorder the columns.

Properties

The Properties tab contains the set of system properties that constitute a data source type configuration. You can override a property by enabling the Override option and defining the expression.
Tip: Not all properties can be overridden. If you attempt to override a property that cannot be overridden, a message displays in the Properties tab.

Matches in the payload are highlighted in the event data that is in the Payloads pane. The highlighting color depends on the type of information that you capture. For example, light green highlighting denotes a capture group match while dark green highlighting denotes a true match. Orange highlighting indicates that the match is ambiguous, which means that the parsed value is known but the location of the value in the payload is not known. When you select and manually highlight text, the highlighting is purple.

The feedback in the Payloads pane shows whether you have the correct regex. If an expression is in focus, the highlighting in the Payloads pane reflects only what that expression can match.

In the Format String field, capture groups are represented by using the $<number> notation. For example, $0 represents a true match; $1 represents the first capture group from the regex, $2 is the second capture group, and so on.

You can add multiple expressions to the same property, and you can assign precedence by dragging and dropping the expressions to the top of the list.

Event mappings tab

The Event Mappings tab displays all the event ID and category combinations that exist in the system for a selected log source type. If a new event mapping is created, it is added to the list of event ID and category combination that is displayed in the Event Mappings tab. In general, the Event Mappings tab displays all event ID and category combinations and the QID records that they are mapped to.