DSM Editor overview

Instead of manually creating a log source extension to fix parsing issues or extend support for new log source types, use the DSM Editor. The DSM Editor provides different views of your data. You use the DSM Editor to extract fields, define custom properties, categorize events, and define new QID definition.

The DSM Editor provides the following views:

Workspace

The Workspace shows you raw event data. Use sample event payloads to test the behavior of the log source type, and then the Workspace area shows you the data that you capture in real time.

All sample events are sent from the workspace to the DSM simulator, where properties are parsed and QID maps are looked up. The results are displayed in the Log Activity Preview section. Click the edit icon to open in edit mode.

In the edit mode, you paste up to 100,000 characters of event data into the workspace or edit data directly. When you edit properties on the Properties tab, matches in the payload are highlighted in the workspace. Custom properties and overridden system properties are also highlighted in the Workspace.

New in 7.4.1You can specify a custom delimiter that makes it easier for QRadar® to ingest multiline events. To ensure that your event is kept intact as a single multiline event, select the Override event delimiter checkbox to separate the individual events based on another character or sequence of characters. For example, if your configuration is ingesting multiline events, you can add a special character to the end of each distinct event in the Workspace, and then identify this special character as the event delimiter.

New in 7.4.2QRadar can suggest regular expressions (regex) when you enter event data in the Workspace. If you are not familiar with creating regex expressions, use this feature to generate your regex. Highlight the payload text that you want to capture and in the Properties tab, click Suggest Regex. The suggested expression appears in the Expression field. Alternatively, you can click the Regex button in the Workspace and select the property that you want to write an expression for. If QRadar cannot generate a suitable regex for your data sample, a system message appears.

Tip: The regex generator works best for fields in well-structured event payloads. If your payload consists of complex data from natural language or unstructured events, the regex generator might not be able to parse it and does not return a result.

Log activity preview

New in 7.4.1 The Parsing Status column was added to the Log Activity Preview.

The Log Activity Preview simulates how the payloads in the workspace appear in the Log Activity viewer. The Parsing Status column indicates whether your event properties are successfully parsing and mapping to a QID record. Every standard property that is supported is displayed. The fields that are marked with an asterisk (*), for example, Event name, Severity, Low-level category, and QID, are populated from the QID map. Fields that are populated from the QID map cannot be parsed verbatim from the raw events data in the workspace, so they cannot be defined or edited. You can adjust their values by selecting the corresponding event ID and category combination from the Event Mappings tab. Then click Edit to re-map an event to a different QID record that exists in the system or to a newly created QID.

Important: You must set an Event ID for any system properties to be parsed correctly.

Click the configure icon to select which columns to show or to hide in the Log Activity Preview window, and to reorder the columns.

Properties

The Properties tab contains the combined set of system and custom properties that constitute a DSM configuration. Configuring a system property differs from configuring a custom property. You can override a property, by selecting the Override system behaviour check box and defining the expression.

Note: If you override the Event Category property, you must also override the Event ID property.

Matches in the payload are highlighted in the event data in the workspace. The highlighting color is two-toned, depending on what you capture. For example, the orange highlighting represents the capture group value while the bright yellow highlighting represents the rest of the regex that you specified. The feedback in the workspace shows whether you have the correct regex. If an expression is in focus, the highlighting in the workspace reflects only what that expression can match. If the overall property is in focus, then the highlighting turns green and shows what the aggregate set of expressions can match, taking into account the order of precedence.

In the Format String field, capture groups are represented by using the $<number> notation. For example, $1 represents the first capture group from the regex, $2 is the second capture group, and so on.

You can add multiple expressions to the same property, and you can assign precedence by dragging and dropping the expressions to the top of the list.

A warning icon beside any of the properties indicates that no expression was added.

Event mappings tab

New in 7.4.1 Support for copying Event ID and Event Category fields was added to the Event Mapping tab.

The Event Mappings tab displays all the event ID and category combinations that exist in the system for a selected log source type. If a new event mapping is created, it is added to the list of event ID and category combination that is displayed in the Event Mappings tab. In general, the Event Mappings tab displays all event ID and category combinations and the QID records that they are mapped to.

Configuration tab

You can configure Auto Property Discovery for structured data that are in JSON format. By default, log source types have Auto Property Discovery turned off.

When you enable Auto Property Discovery on the Configuration tab, the property discovery engine automatically generates new properties to capture all fields that are present in the events that are received by a log source type. You can configure the number of consecutive events to be inspected for new properties in the Discovery Completion Threshold field. Newly discovered properties appear in the Properties tab, and are made available for use in the rules and search indexes. However, if no new properties are discovered before the threshold, the discovery process is considered complete and Auto Property Discovery for that log source type is disabled. You can manually enable the Auto Property Discovery on the Configuration tab at any time.

Note: To continuously inspect events for a log source type, you must make sure that you set the Discovery Completion Threshold value to 0.