IBM Support

Notice: QRadar SIEM version 7.4.3 Fix Pack 3 removed from IBM Fix Central

News


Abstract

This technical note is an overview from the QRadar Support and Development teams regarding the removal of QRadar SIEM 7.4.3 Fix Pack 3 from IBM Fix Central. Users who downloaded this version must not install QRadar SIEM 7.4.3 Fix Pack 3. A new version is under development as QRadar SIEM 7.4.3 Fix Pack 4 to resolve two critical issues reported to the support team. This issue does not affect QRadar on Cloud users.

Content

Summary


QRadar Support is advising users to NOT upgrade to QRadar SIEM 7.4.3 Fix Pack 3 (743_QRadar_FixPack3_2020.11.3.20211021121337) as two critical issues were reported. Administrators reported network connections failing, which can cause updates to fail unexpectedly and applications that do not display after an upgrade completes. On 27 October 2021, development removed QRadar SIEM 7.4.3 Fix Pack 3 from IBM Fix Central for all users globally while a new software delivery is prepared.
 

Affected QRadar versions

 
QRadar SIEM administrators who installed QRadar SIEM 7.4.3 Fix Pack 3 (743_QRadar_FixPack3_2020.11.3.20211021121337).

Important: QRadar on Cloud users are not affected by the issues in this technical note due to mitigations by QRadar DevOps teams.

Affected services

 
Issue Summary Data loss Symptoms
Network / Tunnel Manager
SSH connections can be closed unexpected by the tunnel manager service. 
  1. Encrypted connections between appliances can be closed, causing events or flows to queue in the QRadar pipeline. 
  2. QRadar upgrades can be halted when the network connections are closed unexpectedly. This issue can leave a deployment in scenarios where only some appliances are upgraded.
  3. High Availability (HA) connections between primary and secondary appliances can become disconnected. 
  4. General SSH connectivity issues.
 No. Data is collected properly and incoming events or flows might be queued.
  • Updates failing for QRadar 7.4.3 Fix Pack 3.
  • Event or flow data being delayed in the event pipeline for encrypted managed hosts.
  • HA failover from primary to secondary appliance. A system notification '38750081 Active HA System Failure.' is displayed on the Dashboard.
  • SSH sessions being disconnected when you use the command line.
Application Framework The application framework fails to display applications in the UI on tomcat startup when the HTTPd certificate does not contain the Console IP address in the SAN list. No. Data is collected properly for incoming events and flows.
  • No application UI elements including tabs or toolbar buttons are displayed in the user interface for QRadar 7.4.3 Fix Pack 3.
 

Investigation

The following information outlines the analysis completed by QRadar Development teams.
Category Action or description
File name 743_QRadar_FixPack3_2020.11.3.20211021121337.sfs
Version QRadar SIEM 7.4.3 Fix Pack 3 (743_QRadar_FixPack3_2020.11.3.20211021121337)
Severity Critical
Issue type Network and Application Framework
Root cause New reported issue that affects network connectivity and applications in the user interface for QRadar 7.4.3 Fix Pack 3.
Services
  • Tunnel Manager
  • Application Framework
Immediate actions taken Development teams removed QRadar 7.4.3 Fix Pack 3 from IBM Fix Central.
Analysis for Tunnel Manager The tunnel manager is a service running in QRadar that simplifies the management of SSH tunnels and reduce the load on hostcontext. The tunnel manager consists of a service that creates and maintains tunnels, and a tunnel-monitor component based on a timer. The tunnel manager services runs on a 1-minutes interval on all QRadar appliances to monitor and close tunnels in a bad state.

In QRadar 7.4.3 Fix Pack 3, the service can close valid SSH sessions connections unexpectedly. A closed SSH connection between encrypted hosts can lead to failed upgrades, events or flows queued in the event pipeline (Event Collector to Event Processor handoff), or HA failovers can occur.

If you installed QRadar 7.4.3 Fix Pack 3 was installed, you can disable the tunnel-monitor service and timer as a temporary workaround.

Procedure
  1. Use SSH to log in to the QRadar Console as the root user.
  2. To stop the tunnel-monitor.service, type: 
    /opt/qradar/support/all_servers.sh -Ck "systemctl stop tunnel-monitor.service"
  3. Wait for the command prompt to return.
  4. To stop the tunnel-monitor.timer, type: 
    /opt/qradar/support/all_servers.sh -Ck "systemctl stop tunnel-monitor.timer"
  5. To disable the tunnel-monitor.timer, type: 
    /opt/qradar/support/all_servers.sh -Ck "systemctl disable tunnel-monitor.timer"
    Results
    After the command prompt returns, the tunnel manager service is disabled. The disable command can persist after an appliance reboot. If you installed QRadar 7.4.3 Fix Pack 3 and need assistance with this procedure, you can contact QRadar Support.
Analysis for Application Framework service An issue exists where applications do not display in the user interface after an upgrade to QRadar 7.4.3 Fix Pack 3. All applications report as running in the Application Framework, but the applications cannot be launched. This issue affects applications starting as expected when encryption is enabled in the QRadar deployment or if you generated your own custom certificate without the Console IP address in the SAN list.

During tomcat startup, the gui_application_startup thread attempts to run, but the SAN certificate for the HTTPd service is missing the IP address of the appliance. When this issue occurs the following error is displayed in /var/log/qradar.log: 
[tomcat.tomcat] [gui_app_startup_thread] com.q1labs.uiframeworks.startup.AppStartup
 TomcatService: [WARN] [IP ADDRESS/- -] ]Invocation of endpoint /restapi/api/
 gui_app_framework/applications failed: javax.net.ssl.SSLPeerUnverifiedException: 
 Certificate for <IPADDR> doesn't match any of the subject alternative names

Workaround
If you installed QRadar 7.4.3 Fix Pack 3, you can force stop, then force start the applications to temporarily resolve this issue. A software fix is required to permanently resolve this application framework issue.

Options
There are multiple methods to stop, then start applications in QRadar SIEM. For more information, see Starting apps that are in an ERROR state or do not display in the user interface.

1. Administrators can use the REST API to stop, then start your applications.
  OR
2. Administrators can use the qappmanager utility in /opt/qradar/support and select App instance stop, then App instance start for all applications.
 
Results
After all apps are stopped, then started, they display properly in the user interface. If you installed QRadar 7.4.3 Fix Pack 3 and need assistance with this procedure, you can contact QRadar Support.
Mitigation Do not install QRadar SIEM 7.4.3 Fix Pack 3. A new release is being prepared for all users to resolve these issues.
Mitigation delivered No, currently under development as QRadar SIEM 7.4.3 Fix Pack 4.
Preventive actions
  • Tunnel Manager
    The tunnel-monitor component must be disabled until such time as it can be verified to only close connections for which it originally created. More testing will be integrated into the development and release process to ensure the issue does not happen again.
  • Application Framework
    The Application Framework must use the Console FQDN rather than the IP address. Switching to the FQDN prevents this issue as the FQDN is present in all HTTPd certificates for QRadar, QRadar on Cloud, and any custom certificate.

[{"Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtdAAA","label":"Upgrade"}],"Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.3","Type":"MASTER"}]

Document Information

Modified date:
29 October 2021

UID

ibm16509562

Page Feedback