News
Abstract
The QRadar auto update released on 20 July 2021 introduced a problem where the Traffic Analysis service that auto discovers and creates log sources is no longer working as expected due to a class loading issue. For customers with affected log sources configured on their QRadar appliances, the event pipeline can experience an uncaught exception, which causes events to be routed directly to storage. This technical note is updated with a simplified workaround to resolve the issue for users.
Content
Technical note updates
- 2 August 2021 (11:30 AM EDT): Added a Before you begin section for administrators to confirm the Auto Update Web Server URL field is configured to use: https://auto-update.qradar.ibmcloud.com/
- 30 July 2021 (2:00 PM EDT): Removed Juniper SRC and Itron Smart Meter from the affected products list as these DSMs are still in development and unreleased to users.
- 28 July 2021 (5:30 PM EDT): An overview of this issue was released to users as a flash notice to provide further details for the error.
- 22 July 2021 (9:30 PM EDT): New workarounds are available for users in this technical note update. A codegen.jar can be retrieved from a QRadar Auto Update or administrators can download the latest version of DSM Common RPM from IBM Fix Central (DSMCommon-7.4-20210721162935). This update allows administrators to apply a simplified workaround for IJ33892.
- 22 July 2021 (2:45 PM EDT): Updated the list of affected DSMs to remove Check Point devices from the list. After further review by the development team, the Check Point DSM is not affected by the codegen.jar issue as previous described in this technical note.
- 21 July 2021 (5:10 PM EDT): Updated the link and added an error message for users who are not in the /storetmp directory when the copy command is run.
- 21 July 2021 (4:25 PM EDT): Updated flash notice to add a workaround to restore a previous version of the DSM codegen jar file. Users can complete this workaround to resolve the issue. Created an APAR for this issue as IJ33892, which references this technical note.
- 21 July 2021 (12:00 PM EDT): Initial flash notice for administrators and users published.
Notice: An overview report for IJ33892 has been published with an analysis and preventative actions for the issue described in this technical note. For more information, see: https://www.ibm.com/support/pages/node/6476330. Administrators must confirm the Web Server for Auto Updates uses https://auto-update.qradar.ibmcloud.com/ as described in the following technical note: Important auto update server changes for administrators.
IBM® is alerting QRadar 7.4.x administrators to an auto update issue on 20 July 2021 where the DSM Common RPM can cause traffic analysis issues on managed hosts that collect events, such as Event Collectors, Data Gateways, Event Processors, or QRadar All-in-One Consoles. Managed hosts with an affected DSM configured after DSMCommon-7.4-20210624145517.noarch.rpm is installed, the event pipeline can route incoming events to storage. The DSM Common file from the 20 July 2021 auto update is removed globally from IBM servers and a new version of DSM Common is now available through auto updates or a direct download from IBM Fix Central. The updated version of the codegen JAR file from an auto update or applying DSMCommon-7.4-20210721162935 resolves the issue described in APAR IJ33892.
Important: Administrators are being notified to review for issues where new events are not properly auto discovered by QRadar. If administrators have a log source configured for an affected DSM, then incoming data for all log source types can route to storage. When events are routed to storage, incoming events for the affected DSMs are not parsed and categorized. This issue causes events to be written to disk, bypassing event parsing and correlation to write raw event data to disk. The event data for all log sources is collected and searchable through Quick Filter searches or with Advanced Searches (AQL). Affected administrators must complete the workaround described in this technical note to ensure Traffic Analysis and parsing can function as expected on QRadar 7.4.x managed hosts.
Note: QRadar on Cloud administrators have the workaround for this issue automatically applied to their managed hosts and do not need to complete the workaround described in this technical note.
QRadar Development team confirmed a workaround for this issue and released updated software. Administrators who experience ThreadException errors with events routed to storage can run an automatic update with the Get New Updates button or manually install the latest version of the DSM Common RPM (DSMCommon-7.4-20210721162935) from IBM Fix Central. If you are a new QRadar administrator and require assistance with this workaround, contact QRadar Support.
Administrators can click Get New Updates in the Auto Update user interface to download the latest version of the codegen.jar to resolve the Traffic Analysis and events routing to storage issues described in APAR IJ33872.
Before you begin
Administrators must confirm the Web Server for Auto Updates uses https://auto-update.qradar.ibmcloud.com/ as described in the following technical note: Important auto update server changes for administrators.
Procedure
Important: Administrators are being notified to review for issues where new events are not properly auto discovered by QRadar. If administrators have a log source configured for an affected DSM, then incoming data for all log source types can route to storage. When events are routed to storage, incoming events for the affected DSMs are not parsed and categorized. This issue causes events to be written to disk, bypassing event parsing and correlation to write raw event data to disk. The event data for all log sources is collected and searchable through Quick Filter searches or with Advanced Searches (AQL). Affected administrators must complete the workaround described in this technical note to ensure Traffic Analysis and parsing can function as expected on QRadar 7.4.x managed hosts.
Note: QRadar on Cloud administrators have the workaround for this issue automatically applied to their managed hosts and do not need to complete the workaround described in this technical note.
To determine whether you are experiencing this issue, administrators can:
- Review the System Notifications dashboard for QID 38750088 - Performance degradation has been detected in the event pipeline. Events were routed directly to storage. Routed to storage notifications issued in the last 24 can indicate an affected DSM configured on an appliance.
- Use the QRadar Deployment Intelligence app or search for unparsed events in the last 24 hours. Log Activity > Add Filter > Event is Unparsed = True and filter for the last 24 hours. If you see a spike in 'Stored' events, it can indicate a DSM is generating ThreadException errors. Users can also review for this issue using an Advanced Search:
select count(*) from events where devicetype = 147 and utf8(payload) like '%Exception was uncaught in thread%Event Parser%' last 7 DAYS - Review the logs on appliances. When this issue occurs, a ThreadExceptionHandler error is written to the logs followed by a NoClassDefFoundError: com/q1labs/sem/dsm/build/base/Utils message in /var/log/qradar.log. To locate these messages, administrators can grep for Event Parser messages on appliances where an affected DSM is configured. For example, administrators can grep for event parser messages in the logs:
Output error message for a configured Cisco Aironet DSM with a NoClassDefFoundError.
/opt/qradar/support/all_servers.sh -Ck "grep 'Event Parser' /var/log/qradar.log"
Example: If the returned stack trace lists a ThreadExceptions for the Event Parser with a NoClassDefFoundError, the managed host is affected.[ecs-ec.ecs-ec] [Event Parser[1]] com.q1labs.frameworks.core.ThreadExceptionHandler: [ERROR][xx.xx.xx.xx]Exception was uncaught in thread: Event Parser[1] [ecs-ec.ecs-ec] [Event Parser[1]] java.lang.NoClassDefFoundError: com/q1labs/sem/dsm/build/base/Utils [ecs-ec.ecs-ec] [Event Parser[1]] at com.q1labs.sem.dsm.cisco.aironet.Aironet.populateNevBuilder(Aironet.java:92) [ecs-ec.ecs-ec] [Event Parser[1]] at com.q1labs.sem.dsm.NevBuilderDSM.parseInternal(NevBuilderDSM.java:231) [ecs-ec.ecs-ec] [Event Parser[1]] at com.q1labs.sem.dsm.DSMBase.parse(DSMBase.java:337) [ecs-ec.ecs-ec] [Event Parser[1]] at com.q1labs.sem.dsm.DSMBase.parse(DSMBase.java:312) [ecs-ec.ecs-ec] [Event Parser[1]] at com.ibm.si.ec.filters.normalize.Processor.parse(Processor.java:315) [ecs-ec.ecs-ec] [Event Parser[1]] at com.ibm.si.ec.filters.normalize.Processor.run(Processor.java:181) [ecs-ec.ecs-ec] [Event Parser[4]] com.q1labs.frameworks.core.ThreadExceptionHandler: [ERROR][xx.xx.xx.xx]Exception was uncaught in thread: Event Parser[4]
Affected products
QRadar SIEM 7.4.x on-premise and QRadar on Cloud versions with DSMCommon-7.4-20210624145517.noarch.rpm installed from the 20 July 2021 auto update can experience this issue. Traffic Analysis and events routed to storage issues are reported for administrators with the following log source types configured in QRadar after the 20 July 2021 auto update completed:
- Array Networks SSL VPN Access Gateways
- Cisco Aironet
- CRYPTOCard CRYPTOShield
- Extreme HiGuard
- Extreme XSR Security Routers
- Fair Warning
- HP Network Automation
- IBM DB2
- IBM Informix Audit
- Juniper vGW
- Juniper Networks AVT
- McAfee Application/Change Control
- Microsoft ISA
- Motorola SymbolAP
- Redback ASE
- Sentrigo Hedgehog
- Sophos Enterprise Console
- Sophos PureMessage
- Tropos Control
Note: After further investigation by development, the Check Point DSM is not impacted by the codegen.jar issue described in this technical note. The Check Point DSM was removed from the affected products list. Removed Juniper SRC and Itron Smart Meter from the affected product list as these are in development and not available to users.
Workaround
QRadar Development team confirmed a workaround for this issue and released updated software. Administrators who experience ThreadException errors with events routed to storage can run an automatic update with the Get New Updates button or manually install the latest version of the DSM Common RPM (DSMCommon-7.4-20210721162935) from IBM Fix Central. If you are a new QRadar administrator and require assistance with this workaround, contact QRadar Support.
Option 1: Complete an auto update
Administrators can click Get New Updates in the Auto Update user interface to download the latest version of the codegen.jar to resolve the Traffic Analysis and events routing to storage issues described in APAR IJ33872.
Before you begin
Administrators must confirm the Web Server for Auto Updates uses https://auto-update.qradar.ibmcloud.com/ as described in the following technical note: Important auto update server changes for administrators.
Procedure
- Log in to QRadar as an administrator.
- On the navigation menu (
), click Admin. - In the System Configuration section, click Auto Update.
- Click Check for Updates.
- Click Get New Updates.
- Wait for the auto update to download and install.
- On the Admin tab, click Deploy Changes.
Results
After the Deploy Changes is complete, the workaround is replicated to all appliances in the deployment. Administrators can confirm events are parse and categorize as expected from the Log Activity tab. If you applied the temporary workaround to disable log source types from Traffic Analysis (TA), administrators can enable for those log sources that were previously disabled with the -e option. For more information, see QRadar: How to exclude Log Source types from being discovered by Auto Detection. If you attempted to downgrade the DSM Common RPM file and continue to experience issues after you applied the workaround documented in this technical note, contact QRadar Support.
Option 2: Manually install DSM Common on the Console
Administrators can manually install the latest version of DSMCommon-7.4-20210721162935 from IBM Fix Central for appliances that are not allowed to connect to the Internet or are in air-gapped networks. DSMCommon-7.4-20210721162935 contains the codegen.jar to resolve the Traffic Analysis and events routing to storage for affected DSMs as described in APAR IJ33892.
Procedure
To manually install an RPM file from the command line, root access to the QRadar Console is required.
Procedure
To manually install an RPM file from the command line, root access to the QRadar Console is required.
- Download DSMCommon-7.4-20210721162935 from IBM Fix Central: http://www.ibm.com/support/fixcentral/swg/quickorder?parent=IBM%20Security&product=ibm/Other+software/IBM+Security+QRadar+SIEM&release=7.4.0&platform=All&function=fixId&fixids=7.4.0-QRADAR-DSM-DSMCommon-7.4-20210721162935.noarch.rpm&includeSupersedes=0&source=fc
- Copy the RPM file to the QRadar® Console.
- Using SSH, log in to the command line as the root user.
- Go to the directory that includes the downloaded file.
- Type the following command:
yum -y install DSM-DSMCommon-7.4-20210721162935.noarch.rpmNote: The RPM installation might take several minutes. A success message is displayed to the administrator when the installation completes.
Updated: DSM-DSMCommon.noarch 0:7.4-20210721162935 Complete! - Log in to QRadar as an administrator.
- On the Admin tab, click Deploy Changes.
Results
After the Deploy Changes completes, the workaround is replicated to all appliances in the deployment. Administrators can confirm events are parsing and received as expected from the Log Activity tab. If you applied the temporary workaround to disable log source types from Traffic Analysis (TA), administrators can enable for those log sources that were previously disabled with the -e option. For more information, see QRadar: How to exclude Log Source types from being discovered by Auto Detection. If you attempted to downgrade the DSM Common RPM file and continue to experience issues after you applied the workaround documented in this technical note, contact QRadar Support.
We apologize for any inconvenience for this issue. If you have questions about the contents of this technical note, contact QRadar Support.
- QRadar Support
- QRadar Support
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtDAAQ","label":"Auto Update"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.4.0;7.4.1;7.4.2;7.4.3"}]
Was this topic helpful?
Document Information
Modified date:
02 August 2021
UID
ibm16474189