IBM Support

QRadar: How to create a rule to alert when the number of events routed directly to storage exceeds a configured threshold

How To


Summary

This technical note provides instructions for creating a threshold rule that triggers when the volume of events routed directly to storage exceeds a threshold configured by the administrator.

Objective

When there are performance issues in the event pipeline, the processing capacity of services like ECS-EC and ECS-EP are impacted. To prevent queues from filling up and the system from dropping events, these services bypass processing (parsing, categorization, correlation) and route data directly to disk. Thought, raw data is still collected and searchable.

Steps

Because the problem can happen on ECS-EC (parsing) or ECS-EP (Custom Rule Engine), you must create two rules to monitor both processes and two saved searches based on those properties. After you have created two Custom Event Properties to extract the number events routed directly to storage and the two saved searches based on those properties, you can use the saved searches to create the threshold rules that will alert you when the threshold is exceeded.
Note: If you already followed the steps from QRadar: How to monitor the volume of events routed directly to storage with a time series graph, you can skip to the Create a rule to alert when events routed directly to storage exceeds a configured threshold section directly.

Create two Custom Event Properties (CEPs) to extract the number of events routed to storage

This section creates two CEPs to extract the number of events routed directly to storage from the "System Notification" event.
  1. Log in to the QRadar user interface.
  2. Go to the Admin tab then open the Custom Event Properties.
  3. Click the option Add.
  4. Create the property for the ECS-EP (CRE) by using the following values:
    • Property Type Selection: Extraction Based
    • New Property: "Parsing Routed to Storage"
    • Enable for use in Rules, Forwarding Profiles and Search Indexing: enabled
    • Field Type: Numeric
    • Log Source Type: "System Notification"
    • Log Source: All
    • Event Name: Event(s) were routed directly to storage
    • Regex: Enter the following regex:
      (ThreadedEventProcessor)(.*)(Custom Rule Engine)(.*)(storage.)\s+(\d+)\s(event\(s\))
    • Capture Group: 6
      image-20230525173641-2
  5. Repeat the process to create another CEP to extract the number of events routed directly to storage from the parsing stage (ECS-EC). These fields are the only two changes for this configuration from the prior information:
    • New Property: Parsing Routed To Storage
    • Regex:
      (ThreadedEventProcessor)(.*)(Device Parsing)(.*)(storage.)\s+(\d+)\s(event\(s\))

      Result
      Two CEPs are created to extract the number of events routed directly to storage from "parsing" and the "CRE".

Create two saved search to retrieve the events routed to storage

  1. Log in to the QRadar user interface.
  2. Go to Log Activity, then click Search and select New Search.
  3. Configure the following filters:
    • Group By: Source IP
    • Columns:
      • CRE Routed to Storage (Custom) (Average)
      • Count
    • Order By: Count Desc
    • Results Limit: Leave this section blank
    • Filters:
      • QID Number is 38,750,088
      • CRE Routed to Storage (custom) is not N/A
    • Time Range: Last 5 minutes
      image-20230526171259-1
  4. Click Search and wail until the search finishes running. 
  5. Click Save Criteria. Fill these parameters:
    • Search Name: routedToStorageCRE
    • Timespan options: Recent Last 5 minutes
    • Check Share with everyone.
    • Check Include in my Dashboards.
      image-20230526172615-5
  6. Click Save.
  7. Repeat the process to create another saved search for ECS-EC. The properties are the same other than these changes:
  • In the Columns sections, replace "CRE Routed to Storage (Custom) (Average)" with "Parsing Routed to Storage (custom)"
  • In the Filters sections, replace "CRE Routed to Storage (custom) is not N/A" with "Parsing Routed to Storage (custom)"
  • Name the save search "routedToStorageParsing".

    Result
    Two saved searches to retrieve the average events routed directly to storage are created.

Create a rule to alert when events routed directly to storage exceeds a configured threshold 

  1. Log in to the QRadar user interface.
  2. Go to Log Activity, then click Search and select New Search.
  3. Find the routedToStorageCRE saved search.
  4. Click Load, and then click Search to run it.
  5. After the search completes, click Rules, add Threshold Rule.
  6. The rules wizard opens in a new window. Click Next and select Threshold Rule.
  7. Give the rule a name. In this example, we name it CRE routing to storage.
  8. Add the following condition to the rule.
    The console used on this example is handling around 5000 EPS or 300000 events per minute. The threshold was set to 60000, which means that the rule triggers when the average events routed to storage are 20% of the incoming rate during a one minute interval. You can adjust the threshold (60000 in this case) to your needs.
    • Condition: And when Parsing Routed to Storage (custom) (Average) is greater than 60000 (accumulated in 1 min intervals)
  9. Enable the Test the CRE Events Routed to Storage (custom) (Average) value of each source IP separately option.
    image-20230529184418-1
  10. Click Next and configure the rule responses and rule actions as needed. We are triggering an offense and sending an email alert:
    image-20230529184804-2
  11. Configure a proper response limiter to your needs, check Enable this rule right now, and click Finish to save the rule.
    image-20230529185033-3
  12. Repeat the process and create another rule that uses the routedToStorageParsing saved search you previously created to monitor both the Custom Rule Engine (CRE) and Device Parsing stages of the pipeline.

    Result
    The administrator has two threshold rules that alert by email and create an offense when the number of events routed to storage exceeds the configured threshold.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
21 July 2023

UID

ibm16999191