Troubleshooting
Problem
How can users improve search speed using the Quick Filter feature in QRadar?
Resolving The Problem
About the Quick Filter
The Quick Filter is a search bar that is displayed on both the Log Activity and Network Activity tab in QRadar; and is one of the fastest methods for searching event or flow data. The Quick filter works similar to a 'Google-style' search where you can add in one or more terms, or use regular expressions. If the quick filter is used with other search parameters, the quick filter runs first, and the remaining search parameters are leveraged to further filter the results.
The Quick Filter requires a Payload Index was created, when data was first received by QRadar to work efficiently. If a Payload Index does not exist for the timeframe being searched, QRadar will create a Payload Index for all data, contained within the time frame, which will cause this initial search to take longer to complete. Subsequent searches against the same data, done within the same day are quicker, as the appliance can use the newly created Payload Indexes. Payload Indexes created that are outside the Payload Index Retention are removed overnight.
Figure 1: Utilizing the quick filter, we are able to search 267 MB of data in just over one second.
Location of the Quick Filter?
The Quick Filter is a search bar that is displayed on both the Log Activity and Network Activity tab in QRadar.
Figure 2: Location of the Quick Filter on the Log Activity tab in QRadar 7.2.3 and above (click to enlarge).
Figure 3: Location of the Quick Filter on the Log Activity tab in QRadar 7.1 MR2 (click to enlarge).
Payload Retention Index Settings
To adjust the Payload Index Retention settings from Console:
1) Clicking the Admin tab
2) Select System Settings
3) Locate the Database Settings section and adjusting the retention period.
4) Set the Payload Index Retention only to the timeframe typically searched, as Payload Indexes do use extra disk.
The default Payload Index Retention period is 30 days, the minimum is 1 day, and the maximum is 2 years.
Note: Administrators who want to retain payload indexes longer than the default value should be aware that extra disk space that will be used to retain the index for a longer time period. The retention values should reflect the time spans that the users or security operators are typically searching. After an increase is made to the Payload Index Retention field, administrators should monitor system notifications to ensure that they do not fill disk space by setting unnecessarily long Payload Index Retention.
Figure 4: Quick filter indexes are based on of the Payload Index Retention setting (click to enlarge).
Quick Filter - Using Advanced Search Parameters
The Quick Filter only searches raw, uncorrelated, payload data, and cannot differentiate between fields. For example, the quick filter cannot differentiate if an IP address is the source or destination.
The quick filter can be used for more than just single searches for an IP address or user names. Since the Quick Filter is based on Lucene search technology, the quick filter has the ability to do complex logical queries or include brackets, double quotation marks, AND, OR, NOT, +/-, or wildcards ( * ) operators. It is important that any word operators be uppercase (AND, OR, AND NOT) to prevent the filter from thinking the operator is a search term. All of these logical operators can be used to quickly find results from the indexes of your event or flow payloads. Leveraging operators makes even more complex Quick Filter searches very efficient in QRadar.
Example 1: How to exclude search terms from your quick filter results
To exclude search results, users can leverage the AND NOT, or a minus symbol ( - ) as a method to reduce the amount of returned results from a quick search. If you have specific text that appears with spaces, you can add double-quotes ( "term" ) to encapsulate the exact text you want the quick filter to locate. If your text was broken in to two words, such as Session Token, you could use "Session Token" in the quick filter as the search term expects the space to be present in the search when encapsulated in quotes.
or 
Figure 5 & 6: An alternate method of completing the same search result is to use a minus symbol (-) in the Quick Filter field.
Example 2: How to search for multiple terms
The following images show two examples of a combination search to locate an event that contains the term firewall or "firewall accept", but also contains the words nobody or admin.
or 
Figure 7 & 8: Using the quick filter to search multiple terms using AND/OR values.
Example 3: Using simple regex within a Lucene search
The following images show a example of using regex to search for information within the quick filter. Regular expressions must be bracketed by forward slashes, such as /my_regex_pattern/ in the quick filter. Valid regex that falls between the forward slashes is evaluated by QRadar.
Helpful implementations of regular expressions could include:
- Events with files that end in .exe or .pdf could be located with the regular expression /.*.pdf/ OR /.*.exe/
- URLs, such as /.*baddomain.com/
- Email addresses can be located with the regular expression /.*\@.*\..*/
or 
Figure 9 & 10: Values between the forward slashes can contain simple regular expression patterns.
Example 4: Using Lucene searches with special characters
The following image shows a example of using regex to search where you need to escape special characters. For examples, a username with a hyphen might need to be escaped to return an exact match. Optionally, the user could quick filter search john AND smith or john +smith, but is not as exact as searching for the exact user name of john-smith. The following special characters that are part of the search term must be escaped: + - && || ! ( ) { } [ ] ^ " * ? : \
Figure 11: Using the quick filter to search for a hyphenated user name.
Example 5: Name=Value pair searches in Lucene
To complete name=value pair searches in QRadar or look for specific terms separated by special characters, users can leverage proximity searches to locate a specific payload combination, such as a name=value pair. A proximity search looks for terms that are within a specific distance from one another. Special characters that appear within a text string can be escaped and searched; however, certain values are reserved, so a proximity search allows users to find values located next to each other or separated by mathematical operators.
Where is a proximity search useful?
Where is a proximity search useful?
A payload might contain multiple repeated values of an IP, username, port, or other relevant information. For example, typing root in the Quick Filter search will return all instances of that value, especially where an IP or username could appear repeatedly in different name=value pairs. By searching for the specific name=value pair and combining it with other search terms, users gain search flexibility and the search is extremely fast.
| Example | Quick Filter search |
|---|---|
| username=root | "username root"~1 |
| accountId=joeblack | "accountId joeblack"~1 |
| region:us-east-5 | "region us-east-5"~1 |
| detection_description=IOC ThreatHandled:false |
"detection_description IOC"~1 AND "ThreatHandled false"~1 |
Are there system resource minimums for full payload indexing?
Yes, if you plan to enable full payload indexing, your appliance requires a minimum of 24 GB of RAM. The 24 GB minimum applies to both virtual and physical appliances. However, in most cases we suggest that appliances have 48 GB of RAM when they enable this feature. The minimum and suggested RAM values for full text payload indexing applies to all systems that are processing events or flows, such as 16xx, 17xx, or 18xx appliances, as well as 31xx Consoles or All-in-One Console appliances.
Related Information
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtiAAA","label":"Performance"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
19 May 2023
UID
swg21689800