Expensive DSM extensions were found

38750143 - Performance degradation was detected in the event pipeline. Expensive DSM extensions were found.

Explanation

A log source extension is an XML file that includes all of the regular expression patterns that are required to identify and categorize events from the event payload. Log source extensions might be referred to as device extensions in error logs and some system notifications.

During normal processing, log source extensions run in the event pipeline. The values are immediately available to the custom rules engine (CRE) and are stored on disk.

Improperly formed regular expressions (regex) can cause events to be routed directly to storage.

User response

Select one of the following options:
  • Disable any DSM extension that was recently installed.
  • Review the payload of the notification to determine which expensive DSM extension in the pipeline affects performance. If possible, improve the regex statements that are associated with the device extension.

    For example, the following payload reports that the pipeline is blocked by the Checkpoint DSM:

    Oct 23 12:32:53 ::ffff:10.1.2.4 [ecs-ec] 
[Timer-57] com.q1labs.semsources.filters.normalize.DSMFilter: [WARN]  
[NOT:0080014100][10.1.2.4/- -][-/- -]Expensive Log Source or Log Source 
Extensions Based On Average Throughput in the last 60 seconds 
(most to least expensive) - Checkpoint=0.0eps, CatOS=86.0eps, Apache=2500.0eps,
Endpointprotection=2905.0eps
  • Ensure that the log source extension is applied only to the correct log sources.

    On the Admin tab, click System Configuration > Data Sources > Log Sources. Select each log source and click Edit to verify the log source details.

  • If you are working with protocol-based log sources, reduce the event throttle to ensure that the events do not buffer to disk. The event throttle settings are part of the protocol configuration for the log source.
  • Order your log source parsers from the log sources with the most sent events to the least and disable unused parsers.
  • Verify that your Console is installed with the latest DSM versions.
  • If log sources are created for devices that aren’t in your environment, remove the log sources by using the following command:

    /opt/qradar/bin/tatoggle.pl

    If you have multiple event processors, copy the /opt/qradar/conf/TrafficAnalysisConfig.xml file to the /store/configservices/staging/globalconfig/ directory. On the Admin tab, click Deploy Full Configuration for all managed hosts to obtain the configuration file.