IBM Support

Updated: QRadar Custom property concurrency can cause search and ariel data loss (APAR IJ21718)

Flashes (Alerts)


Abstract

Administrators or users might encounter an Event Processor exception that can cause data loss as events are not properly written to disk. Users on impacted versions must complete a Deploy Full Configuration. An interim fix is available on IBM Fix Central to mitigate the issue on affected versions.

Content

Urgency of the issue

Critical. Administrators who encounter APAR IJ21718 can experience data loss due to a custom property concurrency issue. A detailed explanation of APAR IJ21718 is available from QRadar Support team here:  https://www.ibm.com/support/pages/node/1142758. Administrators are being alerted to this issue so they can complete a Deploy Full Configuration, then install the available interim fix.

UPDATE: The mitigation section of this article includes links and new instructions for administrators. It is expected that administrators complete the workaround, then apply available interim fixes to resolve this issue.

Affected products and versions

QRadar Event Processors (16xx or 18xx) appliances or QRadar Consoles (31xx) at the following software versions:
 
  • QRadar 7.3.2 Patch 5 (7.3.2.20191022133252)
  • QRadar 7.3.3 (7.3.3.20191031163225)
  • QRadar 7.3.3 Patch 1 (7.3.3.20191203144110)

    QRadar on Cloud Notice: QRadar on Cloud Consoles and Event Processors are being monitored for occurrences of APAR IJ21718 and this notice is informational only.  
 

How to locate QRadar Event Processors in the deployment

  • From the Admin tab, click System & License Management > Systems.
    Administrators can review the list of systems in your deployment. Event Processors are 16xx or 18xx appliances. QRadar Consoles or All-on-One appliances are identified as 31xx. Where xx is a numeric identifier for appliance capability.
  • Optional. Administrators with root access or large deployments can get a report of appliances with the following command: /opt/qradar/support/deployment_info.sh -O
    [root@qr732-3199-2553 support]# ./deployment_info.sh -O
    INFO: Gathering deployment information. This may take a while...

    Hostname             IP              HA Status    Appliance   Hardware
    qr732-3199-2553      10.10.219.230   N/A          3199        VMware Virtual Platform
    qr732-1699-2566      10.10.219.231   N/A          1699        VMware Virtual Platform
    qr732-1599-2570      10.10.219.232   N/A          1599        VMware Virtual Platform

    Results
    If you have Event Processor appliances in the network, administrators can confirm that they are not experiencing search issues.  When a concurrency issue occurs in ariel, search results from the Log Activity tab can return: 'The server encountered an error reading one or more files' error messages. An error is recorded in /var/log/qradar.log on the appliance and can be used to confirm concurrency issues. For example:

    [ecs-ep.ecs-ep] [Ariel Writer#events] com.q1labs.frameworks.core.ThreadExceptionHandler: [ERROR] [NOT:0000003000][XX.XX.XX.XX/- -] [-/- -]Exception was uncaught in thread: Ariel Writer#events

Workaround

Administrators who receive reports for search issues or notice ariel writer exceptions can complete a full deploy. A Deploy Full Configuration restarts services (ecs-ep) clears the thread exception and data is processed as expected on Event Processors. A Deploy Full Configuration is a temporary solution and the issue can occur again.
 
  1. Log in to the QRadar Console.
  2. Click the Admin tab.
  3. Click Advanced > Deploy Full Configuration.
    image-20191219144803-3
  4. Click Continue to confirm.
    image-20191219144948-4
  5. Wait for the full deploy to complete.

Mitigation

To mitigate the issue described in APAR IJ21718, administrators must to download and install an interim fix to their QRadar deployment. Interim fixes are small updates released on the latest software versions to allow administrators to resolve a specific issue. The interim fix is installed on the Console only and the updates are applied to the QRadar deployment.

 
For QRadar 7.3.2 Patch 5 (7.3.2.20191022133252)
  1. Download the QRadar 7.3.2 Patch 5 interim fix 1 from IBM Fix Central: http://www.ibm.com/support/fixcentral/swg/quickorder?parent=IBM%20Security&product=ibm/Other+software/IBM+Security+QRadar+SIEM&release=7.3.0&platform=All&function=fixId&fixids=7.3.2-QRADAR-QRSIEM-20191220232616INT&includeSupersedes=0&source=fc
  2. Install the interim fix per the release notes: https://www.ibm.com/support/pages/node/1142842

    Results
    After the interim fix is applied, the data loss issue is resolved. If you see 'The server encountered an error reading one or more files' in Log Activity, you can open a QRadar Support case. A support representative can resolve the file read error from the original incident.
For QRadar 7.3.3 (7.3.3.20191031163225)
  1. Download the QRadar 7.3.3 FP 1 from IBM Fix Central: http://www.ibm.com/support/fixcentral/swg/quickorder?parent=IBM%20Security&product=ibm/Other+software/IBM+Security+QRadar+SIEM&release=7.3.0&platform=All&function=fixId&fixids=7.3.3-QRADAR-QRSIEM-20191203144110&includeSupersedes=0&source=fc

  2. Install the update per the release notes: https://www.ibm.com/support/pages/node/1125987

  3. Download the QRadar 7.3.3 Patch 1 interim fix 1 from IBM Fix Central: http://www.ibm.com/support/fixcentral/swg/quickorder?parent=IBM%20Security&product=ibm/Other+software/IBM+Security+QRadar+SIEM&release=7.3.0&platform=All&function=fixId&fixids=7.3.3-QRADAR-QRSIEM-20191220154048INT&includeSupersedes=0&source=fc

  4. Install the interim fix per the release notes: https://www.ibm.com/support/pages/node/1142836

    Note: Administrators who are unable to patch their system due to the holiday season and staffing can contact QRadar Support. The support team can install a hotfix (jar) file to Event Processors in the deployment to assist administrators who cannot apply a fix pack (patch) and interim fix. Instructions on how to open a case for this issue are provided in this alert for administrators.
For QRadar 7.3.3  Patch 1 (7.3.3.20191203144110)
  1. Download the QRadar 7.3.3 Patch 1 interim fix 1 from IBM Fix Central: http://www.ibm.com/support/fixcentral/swg/quickorder?parent=IBM%20Security&product=ibm/Other+software/IBM+Security+QRadar+SIEM&release=7.3.0&platform=All&function=fixId&fixids=7.3.3-QRADAR-QRSIEM-20191220154048INT&includeSupersedes=0&source=fc
  2. Install the interim fix per the release notes: https://www.ibm.com/support/pages/node/1142836

    Results
    After the interim fix is applied, the data loss issue is resolved. If you see 'The server encountered an error reading one or more files' in Log Activity, you can open a QRadar Support case. A support representative can resolve the file read error from the original incident.
 

Required support case details


If you see recurring search 'The server encountered an error reading one or more files' errors messages or the logs repeatedly report Exception was uncaught in thread: Ariel Writer#events, you can open a case with QRadar Support. Administrators must include the following information in your case:
 
  • In the summary field, type: IJ21718: Exception in ariel on Event Processor
  • In your description, inform the support representative of any actions/workarounds you have completed.
  • Provide updated contact information (email and phone number).
    NOTE: This is important as we understand it is the holiday season and we want to confirm we contact the correct team members.
  • Provide logs for your QRadar Console and the Event Collector.
    NOTE: You can select multiple appliances in the Admin > System and License Management > Select multiple appliances > Actions > Collect log files in user interface. Optionally, you can use the /opt/qradar/support/get_logs.sh utility from the command-line interface of the Event Processor appliance. For more information on collecting logs in QRadar, see: https://ibm.biz/qradarlogs.




NOTE: If you are unsure of the impact to your system or if you have follow-up questions, you can open a case with the QRadar Support team or ask about questions and updates here: https://developer.ibm.com/answers/questions/525244/flash-notice-apar-ij21718-ariel-writer-concurrency.html.

 

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"Flash Notice","Platform":[{"code":"PF016","label":"Linux"}],"Version":"QRadar 7.3.2 Patch 5;7.3.3;7.3.3 Patch 1","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
21 December 2019

UID

ibm11142872

Page Feedback