Indicators missing in results from a Threat Intelligence Insights search
The Universal Data Insights component limits the number of results that are returned from a search query. This limit can cause missing indicators in IBM® Security Threat Intelligence Insights search results.
Issue details
Universal Data Insights is designed to impose limits that protect Threat Intelligence Insights from taking too many results from individual data sources. Universal Data Insights also imposes a global maximum on results across all the data sources in a query.
User impact
When you view the indicators that are found in a search result, some indicators might be missing. This is a generic issue that can occur in any Universal Data Insights query.
Possible cause
For example, Universal Data Insights constructs a query
of many indicators that are separated by an OR clause. The Threat Intelligence
query is looking for IP address 192.0.2.01 OR 203.0.113.01. It is
possible that an individual data source result reaches the default limit and finds
192.0.2.01 on the first 10,000 records. Then, Universal Data Insights no longer retrieves data from that data
source. The consequence is that the 203.0.113.01 indicator might not show up in
results. Using the same query as an example, an alternative result is possible, across all of your
data sources the result finds 192.0.2.01 before the global maximum limit of 60,000
is retrieved. In this case, 203.0.113.01 is not found before Universal Data Insights stops querying data sources when the global
result maximum limit is reached. The data that is returned to Universal Data Insights from data sources in the time before the
limit is reached is the data that is written into the 60,000 results.
Diagnosis
In the Threat Intelligence Insights reports, you see missing indicators in the results from searches.
Remediation
Each individual data source has a setting for the maximum result size of the data that is returned for that data source. The default is 10,000 for an individual data source. The maximum result limit for an individual source can be configured in the Data Source Settings page. For more information about data source configuration, see Universal Data Insights connectors.
The default data source limit of 10,000 is set to protect data sources because some data sources do not work well above that limit. The maximum setting allowed for an individual data source setting is 60,000. The limit of 60,000 was implemented because some applications interfacing with Universal Data Insights might have issues when dealing with results above that limit.
By default, query results are limited to the maximum size of 60,000.
Postremediation validation
Run a new search with a higher result limit.