IBM Support

QRadar: Rules with email responses that leverage custom properties can cause search and ariel writer exceptions (APAR IJ21718)

Troubleshooting


Problem

This support technical article provides further guidance to administrators on the issue reported in APAR IJ21718: Ariel searches fail and events are not processes/written to disk when a concurrent modification exception occurs.

Symptom


UPDATE: For the latest information, see the Flash Notice for APAR IJ21718. Administrators must complete a Deploy Full Configuration, then install the available interim fixes. Interim fixes are available on IBM Fix Central to mitigate this issue.
Administrators or users might experience search errors in QRadar when the Event Processor or Console attempts to access an unreadable record of data for the timeframe defined in the query. As an Event Processor responds to the search request, a red alert bar is displayed with the search results to indicate one of the Event Processor appliances encountered a 'The server encountered an error reading one or more files' message. The error reading one or more files message is an indicator of APAR IJ21718.

For example:
image-20191219144221-2

Cause

APAR IJ21718 describes a situation where the ariel writer thread experiences a ConcurrentModificationException. A concurrency issue can occur when multiple users are working in QRadar and a non-optimized custom property value is being updated by a rule while another user attempts to complete a search for that same custom property at the same time. The search query for the custom property requests the data from the Event Processor, but the rule is concurrently updating the value to ensure that it iterates through the event payloads for matches.

Order of operations
  1. An administrator creates a custom event property for a log source, but does NOT select the option to parse the custom property for rules or searches.
    - In the DSM Editor, this value is not selected: Enable this Property for use in Rules and Search Indexing.
    - In the Custom Event Property interface, this value is not selected: Parse in advance for rules, reports, and searches.
  2. User A creates new rule that includes the administrator's custom property to ensure that when the rule triggers that an email response is sent.
  3. User B creates a search with the administrator's custom property and requests the data from the Event Processor appliance.
  4. As event data is evaluated by the Event Processor, rules engine needs to update the custom property map against incoming events to ensure data is collected as part of the rule test and email.
  5. The data expected in search query from user B is requested at the same moment the rule test evaluation updates the custom property map.

    Summary
    As the custom property was not configured for use in rules and searching, the search query is expecting to stream the search results from the custom property map. However, the custom rules engine increments the size of the custom property at the same moment to include the data for the rule response. When the search request and rule update occur at the same moment for the same serialized custom property data, a concurrency exception is created and the error message in the APAR is written in the logs.

     

Environment

  • QRadar 7.3.2 Patch 5 (7.3.2.20191022133252)
  • QRadar 7.3.3 (7.3.3.20191031163225)
  • QRadar 7.3.3 Patch 1 (7.3.3.20191203144110)

Diagnosing The Problem

When a concurrency issue occurs in ariel, search results can return 'The server encountered an error reading one or more files' messages as an indication of the issue as described in the Symptoms area of the technical note. An error message can also be written in /var/log/qradar.log on the appliance where the concurrency issue occurred. For example:

[ecs-ep.ecs-ep] [Ariel Writer#events] com.q1labs.frameworks.core.ThreadExceptionHandler: [ERROR] [NOT:0000003000][XX.XX.XX.XX/- -] [-/- -]Exception was uncaught in thread: Ariel Writer#events

Resolving The Problem


UPDATE: For the latest information, see the Flash Notice for APAR IJ21718. Administrators must complete a Deploy Full Configuration, then install the available interim fixes. Interim fixes are available on IBM Fix Central to mitigate this issue.

Document Location

Worldwide

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"Custom Properties","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3.2 Patch 5,\n7.3.3,\n7.3.3 Patch 1","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
07 January 2021

UID

ibm11142758