Resolving The Problem
QRadar indexes a number of properties when normalizing data, at the time it is received by QRadar and written to disk. Taking advantage of the default indexed properties is one of the best methods to improve the speed of searches in QRadar. Any search filter parameter with Indexed appended, can be leveraged for more efficient searching. By adding indexed properties, the system is able to determine which files contain the data matching that specific index, reducing the size of the data set required to find any remaining non-indexed search values.
Video 1: See the full series of videos on Searching your QRadar data efficiently in the IBM Security Learning Academy.The following fields are indexed out-of-the-box in QRadar:
|Event fields indexed by default||Image|
Note: The parameter Category [Indexed] is an index of the Low Level Category from the event data.
Note: The parameter Source or Destination IP is not defined as [Indexed], however, searches performed against this parameter use Source IP [Indexed] and Destination IP [Indexed] when selected. This field is a composite property, which is made up of two separate indexed properties.
Figure 1: Examples of default indexed event fields.
|Flows fields indexed by default||Image|
Note: Any values that display (custom) in the name are associated to Custom Event Properties or Custom Flow Properties.
Figure 2: Examples of default indexed flow fields.
Example of how filters improve results
Let's say that a user wants to find all logs in the past six months where the payload contains the text, 'The operation is not allowed'. This means that to complete a payload search on the text The operation is not allowed from the last six months, the system is forced to reread every payload value from every event or flow from the last six months to find matches. Without any filters applied, the data sets can become extremely large and impact performance when not required. QRadar is fully capable of returning this result, but the efficiency of the search is not optimal. If you can place an indexed value as a filter before you add a payload filter, for example a Log Source Type [Indexed], Username [Indexed], Event Name [Indexed], or even Source IP or Destination IP, then search for the payload containing The operation is not allowed, your results display faster. This is because the indexed filter assists the search by eliminating portions of the data set and reduces the overall data volume and number of event or flow logs that has to be searched.
Tip: Start your searches by adding a filter that includes any relevant [indexed] values to quickly locate results when looking for a specific/'sparse' result from large data sets or long timeframes. The inclusion of [indexed] filters allows the Ariel database to quickly search index files for pointers to the data files that contain your results.
How do I leverage search statistics to improve user experience?
The real power in indexing comes from understanding what users are searching for and helping them complete their searches by enabling indexes for the properties that users search frequently. In QRadar 7.1 and later, a feature was introduced that allows administrators not only to enable or disable indexes on specific fields in QRadar by using the Index Management feature, but to understand the data behind what is being searched in QRadar. The Index Management icon can be found on the Admin Tab.
Figure 3: Example of the Index Management window from the QRadar Admin tab.
Search times can be reduced when you enable indexing for properties used frequently in searches. Administrators must be mindful when you enable indexing as large numbers of enabled properties can have an effect on system performance when the data is being written and take additional disk space.
Index management allows administrators to control database indexing, which can optimize search performance for frequently searched values or criteria. As mentioned, QRadar enables some index values out-of-the-box, however, a number of values are not enabled by default. Administrators should review what types of events or flows are often searched to enable indexes that can improve search performance. The Index Management feature in QRadar can provide statistics for administrators to outline percentage values on what properties are searched and what searches are using indexes over a certain time period. Administrators can adjust the time range to understand how users are completing searches. The default statistics interval is 24 hours; however, statistics can be configured to display statistics from 1 hour ago to the last 30 days.
Index Management Columns:
- % of Searches Using Property - This value indicates what percentage of searches use (across all users) a specific property in their search criteria within the selected timeframe. High percentage values indicate properties that used in searches during the last interval.
- % of Searches Hitting Index - This value represents the percentage of searches (across all users) that used the index property in a search value.
- % of Searches Missing Index - A search that misses an index is counted each time a search includes a property, but either the property field has no index at any point during the search or when a search index is disabled.
For example, a user creates a search for a property with a time frame of 8:00 to 9:00. If the search does not have any indexes (disabled), then a miss counted. When an index is disabled, this is why QRadar can show some searches missing 100% of the index.
If an index is available for only part of the time interval, that search counts as both an "index missed" and an "index hit" value. This situation can occur when an index is only available for part of the time period, such as an index being enabled for a property at 8:15. Since the index was not available for the entire time period, both the hit and the miss are counted, which can lead to values such as Searches using Property, 1.0%. Index hit, 4.5%. Index miss, 95.5%.
- Data Written - The amount of disk space the index currently consumes.
- Database - Some common properties overlap across both the event or flow data, such as IP addresses ports, etc. This column defines whether the property is part of the event or flow database.
Are there any general rules administrators can follow for indexing?
Yes, the following table provides some general advice on how to handle search indexes. Administrators should evaluate how users are leveraging searches over several time ranges (24 hours, 7 days, and 1 week) to make informed decisions on how users are searching in QRadar before enabling or disabling indexes.
|What to look for||Time frame||Action to consider||Reason|
|Properties where the index is disabled and
the % of Searches Using Property is above 30% and the % of Searches Missing Index is above 30%
|- 24 hours
- 7 days
- 30 days
|Enable this index||If administrators see search percentages above 30% across multiple time spans, then users are leveraging this search property often and consideration should be made to enable the index. These values indicate that enabling an index can improve performance for users who search specific properties frequently.|
|Properties where the index is enabled and the % of Searches Using Property is zero.||- 30 days||Disable this index||If after 30 days the statistics show that an enabled index is used in zero % of searches, then consideration should be made to disable the indexed property.
This preserves resources for more important and actively used searches.
Was this topic helpful?
15 December 2021