Release of the QRadar Incident Forensics 7.3.1 Patch 4 ISO (7.3.1.20180507202600)

Content

Important advisories in this update

This QRadar Incident Forensics software update includes an operating system mitigation for the side-channel analysis vulnerability known as Spectre (Variant 1) and Meltdown (Variant 3). Due to a potential change to search performance in Meltdown (Variant 3), a new installation prompt is displayed to administrators so they can decide how the mitigation is applied. Installation of this update is not intended to, nor will it provide mitigation measures against Spectre Variant 2 (CVE-2-17-5715). The mitigation for Variant 2/Spectre requires an OEM microcode/BIOS update on appliances. For information about other variants for Spectre and Meltdown, see the following IBM Flash Notice: http://www.ibm.com/support/docview.wss?uid=swg22012320.

This update contains a mitigation for CVE-2017-5754 Variant 3/Meltdown provided by Red Hat that can impact search performance. Administrators must read the release notes before they install this update.

Choices:
1) Enable: Turn ON the mitigation for Variant 3/Meltdown on all appliances.
2) Disable: Turn OFF the mitigation for Variant 3/Meltdown on all appliances. IF YOU CHOOSE NOT TO ENABLE THIS UPDATE TO ADDRESS CVE-2017-5754, YOU WILL NOT HAVE ANY PROTECTION AGAINST VARIANT 3/MELTDOWN.
3) Abort patch.


CVEID: CVE-2017-5753 (Variant 1/Spectre)
MITIGATION: Enabled by default during the installation of QRadar Incident Forensics 7.3.1 Patch 4 and cannot be disabled.
DESCRIPTION: Intel Haswell Xeon, AMD PRO and ARM Cortex A57 CPUs could allow a local authenticated attacker to obtain sensitive information, caused by a bounds check bypass in the CPU speculative branch instruction execution feature. By conducting targeted cache side-channel attacks, an attacker could exploit this vulnerability to cross the syscall boundary and read data from the CPU virtual memory.
IMPACT: No performance impact.

CVEID: CVE-2017-5754 (Variant 3/Meltdown)
MITIGATION: An installation prompt is provided to enable or disable this mitigation on appliances. A utility is also provided to allow administrators to enable or disable the mitigation for CVE-2017-5754 (Variant 3/Meltdown) post-installation, see the Installation wrap-up for further details. IBM cannot be held responsible for risks incurred by administrators who do not enable the mitigation of CVE-2017-5754 (Variant 3/Meltdown).
IF YOU CHOOSE NOT TO ENABLE THIS UPDATE TO ADDRESS CVE-2017-5754, YOU WILL NOT HAVE ANY PROTECTION AGAINST VARIANT 3/MELTDOWN.
DESCRIPTION: Intel Haswell Xeon, AMD PRO and ARM Cortex A57 CPUs could allow a local authenticated attacker to obtain sensitive information, caused by a rogue data cache load in the CPU speculative branch instruction execution feature. By conducting targeted cache side-channel attacks, an attacker could exploit this vulnerability to cause the CPU to read kernel memory from userspace before the permission check for accessing an address is performed.
IMPACT: Search performance degradation has been observed on appliances when the mitigation for Variant 3/Meltdown is enabled. Administrators who want to evaluate search duration on appliances before they apply the mitigation can review the following technical note: Search performance evaluation for CVE-2017-5754 (Variant 3/Meltdown).

Firmware notice
The mitigation for Variant 2/Spectre requires an OEM microcode/BIOS update on appliances. For more information on firmware releases, see: https://ibm.biz/qradarfirmware.

Performance assessment summary
Administrators can expect performance degradation after they enable the mitigation for the vulnerability.

• A 3% to 6% increase in CPU utilization has been observed across all workloads on appliances after the mitigation applied.
• Search performance for most common search types has been observed to degrade by 0% to 10%, with the following exceptions:
• Searches that use indexed criteria and match a moderate number of results (less than 10% of the total searched dataset) are expected to be degraded between 3% to 20%.
• Open-ended searches that have no limit applied to the query and return a very large number of results (30% of the total searched dataset or more) are expected to be degraded by up to 2x.
• The impact on data processing is estimated to be in the 0% to 20% range.
• High availability on 1 Gbit network is not affected. The initial high availability setup speed and catch-up replication speed after fail-over will be lower on 10 Gbit network. However, the replication rate is still in the multiple hundreds MB/s, which is sufficient for real time replication.

About this upgrade

These instructions are intended to assist administrators to update appliances from QRadar Incident Forensics 7.2.8 to 7.3.1 Patch 4 by using an ISO file. For more information, see the QRadar Incident Forensics Installation Guide.

QRadar Incident Forensics 7.3.1 Patch 4 uses an ISO file to update to the latest software version. A minimum of QRadar Incident Forensics 7.2.8 Patch 1 (or later) is required to be able to upgrade to QRadar Incident Forensics 7.3.1 Patch 4.

Administrator notes

1. You must be on QRadar Incident Forensics 7.2.8 Patch 1 or later to upgrade to QRadar Incident Forensics 7.3.1 Patch 4.
2. The upgrade from QRadar Incident Forensics 7.2.8 Patch 1 to QRadar Incident Forensics 7.3.1 Patch 4 will use an .ISO file. In the past, ISOs were for new appliance installations only, but QRadar Incident Forensics 7.3.1 Patch 4 is an exception to this rule because of the Red Hat kernel update requirements.
3. Utilities or custom scripts that power users created for their QRadar Incident Forensics deployment should be copied off the system. During the 7.3.1 Patch 4 update, a warning is displayed that only data in /store will be preserved. After the appliance reboots, any scripts, 3rd party accounts, or utilities in /tmp, or /, or /root are deleted. This does not impact any ISO files that were initially mounted by using /root because this cleanup occurs later in the installation procedure.
   

Current QRadar Incident Forensics Version	Upgrades to QRadar Incident Forensics 7.3.1 Patch 4?
QRadar Incident Forensics 7.2.6 (any patch level) or earlier	No
QRadar Incident Forensics 7.2.7 (any patch level)	No
QRadar Incident Forensics 7.2.8.0	No
QRadar Incident Forensics 7.2.8 Patch 1 or later	Yes, use these release notes to complete this process.
QRadar Incident Forensics 7.3.0 (any patch version)	See the QRadar 7.3.1 Patch 4 SFS Release Notes for more information. The QRadar SFS file can update an existing 7.3.0 Incident Forensics install to version 7.3.1 Patch 4.
QRadar Incident Forensics 7.3.1 (any patch version)	See the QRadar 7.3.1 Patch 4 SFS Release Notes for more information. The QRadar SFS file can update an existing 7.3.0 Incident Forensics install to version 7.3.1 Patch 4.

Figure 1: If you have installed QRadar 7.2.8 or later, you are not required to install each ISO release to update to QRadar 7.3.1 Patch 4.

The QRadar Incident Forensics 7.3.1 Patch 4 ISO (20180507202600) can upgrade QRadar Incident Forensics 7.2.8 Patch 1 (7.2.8.20161207001258) and later to QRadar Incident Forensics 7.3.1 Patch 4. However, this document does not cover all the installation messages and requirements, such as changes to memory requirements or browser requirements for QRadar Incident Forensics. If you are on a version of QRadar Incident Forensics earlier than QRadar Incident Forensics 7.2.8 Patch 1, you can upgrade to QRadar Incident Forensics 7.2.8 Patch 10 before proceeding to install the QRadar Incident Forensics 7.3.1 Patch 4 ISO.

Before you upgrade

Ensure that you take the following precautions:

• Back up your data before you begin any software upgrade and verify that you have recent configuration backups that match your existing Console version. If required, take an on demand configuration backup before you begin. For more information about backup and recovery, see the QRadar Administration Guide.
• All appliances in the deployment must be at the same software and patch level in the deployment.
• Verify that all changes are deployed on your appliances. The update cannot install on appliances that have changes that are not deployed.
• A QRadar Incident Forensics 7.3.1 ISO is available to upgrade from QRadar Incident Forensics 7.2.8 Patch 1 or install a new appliance or virtual machine. To complete a new install, review the QRadar Incident Forensics Installation Guide.
• To avoid access errors in your log file, close all open user interface sessions.
• If you are unsure of the IP addresses or hostnames for the appliances in the deployment, run the utility /opt/QRadar/support/deployment_info.sh to get a .CSV file that contains a list of IP addresses for every managed host.
• If you are unsure of how to proceed when reading these instructions or the documentation, it is best to ask before you start your upgrade. To ask a question in our forums, see: https://ibm.biz/qradarforums.

Part 1. Staging files and pretesting your appliance

It is important to pretest your deployment to ensure that you will not experience unexpected issues when you update to QRadar Incident Forensics 7.3.1 Patch 4. A pretest is a common precaution to locate potential issues. The pretest does not restart services and can be completed without scheduled downtime. The pretest typically takes between 3 to 5 minutes to complete on each appliance. If for some reason your SSH session is disconnected, you can reconnect to the remote host using screen.

Procedure
The pretest should be completed on all hosts before you attempt to upgrade to QRadar Incident Forensics 7.3.1 Patch 4.

1. You must install the QRadar Console with 7.3.1 before you begin this procedure.
2. Download the QRadar Incident Forensics 7.3.1 Patch 4 ISO (5.2 GB) from the IBM Fix Central website: http://www.ibm.com/support/fixcentral/swg/quickorder?parent=IBM%20Security&product=ibm/Other+software/IBM+Security+QRadar+Incident+Forensics&release=7.3.0&platform=Linux&function=fixId&fixids=7.3.1-QRADAR-QIFFULL-20180507202600&includeSupersedes=0&source=fc
   

3. Use SSH to log in to your Console as the root user.
4. Open an SSH session to the QRadar Incident Forensics appliance.
5. Type the following command: screen
6. To make the directory for the update, type: mkdir -p /media/cdrom
7. To verify you have enough space (5.2 GB) in /root or /var/log for the ISO on all appliances, type:
   df -h /root /var/log | tee diskchecks.txt
   
• Best directory option: /root
  It is available on all appliance types, is the best option to host the ISO file.
• 2nd best directory option: /var/log
  This directory is available on all appliances, but there might not be the required space available.
• DO NOT USE: /tmp, /store/tmp, or /store/transient for your ISO upgrade. These directories are partitioned as part of the upgrade; you cannot use them as storage locations or mount points for the ISO file.
  
  If the disk check command fails, retype the quotation marks from your terminal, then re-run the command. This command returns the details to both the command window and to a file on the Console named diskchecks.txt. Review this file to ensure that all appliances have at minimum 5.2 GB of space available in a directory to copy the ISO before attempting to move the file to a managed host. If required, free up disk space on any host that fails to have less that 5.2 GB available.
  
  Reminder: Any utilities or custom scripts that you have created for QRadar Incident Forensics should be copied off the system. During the 7.3.1 update, a warning is displayed that only data in /store will be preserved. Scripts, 3rd party utilities in /tmp, or /, or /root will be deleted during the upgrade.
  

8. If there is not 5.2GB of space in /root or /var/log, make directory space for the ISO file.
9. Using WinSCP or SCP, copy the ISO to the /root or /var/log directory on the QRadar Incident Forensics Console with 5.2 GB of disk space for the ISO file.
10. To mount the ISO, type the following command: mount -o loop /root/Rhe764forensics7_3_1_20180507202600.stable-7-3-1.iso /media/dvd"
11. To pretest the Console appliance, type: /media/dvd/setup -t
    The pretest output is written to the command window. Review this output after the pretest completes.
12. Use SSH to open an SSH session to the other appliances in your deployment. QRadar Incident Forensics Support recommends that you run the pretest to identify issues before the update begins.
13. To pretest the managed host, type: /media/dvd/setup -t
    
    Results
    If an appliance in your deployment fails the pretest, you can take the recommended action from the pretest utility. The issue must be resolved before the update to 7.3.1 begins to prevent downtime for specific appliances. If there are messages you do not understand or want to discuss further, you can use our forums: https://ibm.biz/qradarforums.

Part 2. Installing the QRadar Incident Forensics 7.3.1 Patch 4 ISO

These instructions guide you through the process of upgrading an existing QRadar Incident Forensics install at 7.2.8 Patch 1 or later to QRadar Incident Forensics software version 7.3.1 Patch 4. The update on the Console must be completed first before you attempt to update QRadar Incident Forensics 7.3.1 Patch 4.

Procedure
You must complete: Staging files and pretesting your deployment before you begin the installation steps listed below.

1. Use SSH to log in to the Console as the root user.
   
   NOTE: When you log in, the SSH session should display 7.3.1 as the Console's version. This verifies that the QRadar Console has been updated to 7.3.1, which is required before you update your Incident Forensics appliance.
2. Open an SSH session to the QRadar Incident Forensics appliance.
3. To run the ISO installer, type the following command: /media/dvd/setup
   
   Important: Upgrading from QRadar Incident Forensics 7.2.8 Patch 1 or later to QRadar Incident Forensics 7.3.1 Patch 4 can take 1 to 2 hours to complete on the appliance.
   

Results
A summary of the ISO installation advises you of any issues. If there are no issues, use SSH to log in to managed hosts and start the installer on each host to run the setup in parallel.

Part 3. Installation wrap-up

1. After all hosts are updated, send an email to your team to clear their browser cache before logging in to the QRadar Incident Forensics SIEM interface.
2. To unmount the /media/dvd directory, type: umount /media/dvd
3. Delete the ISO from the appliance.
4. Review any static routes or customized routing. As mentioned in the administrator notes, all routes were removed and will need to be reconfigured after the upgrade completes.
5. Review any iptable rules configured because the interface names changed in QRadar Incident Forensics 7.3.1 Patch 1 due to the Red Hat Enterprise 7 operating system updates. Any iptables rules that use Red Hat 6 interface naming conventions must be updated.
   

Part 4. Installation wrap-up

1. Due to the potential search performance change when CVE-2017-5754 (Variant 3/Meltdown) is enabled, the installation of QRadar 7.3.1 Patch 4 includes a utility to allow administrators to enable or disable the mitigation after the initial installation completes. Administrators must be aware of the security implications if they choose to use this utility to disable the mitigation for CVE-2017-5754 (Variant 3/Meltdown). IBM cannot be held responsible for risks incurred by administrators who choose to disable the mitigation for CVE-2017-5754.

• To enable the mitigation on all hosts from the QRadar Console, type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh enable-all
• To enable the mitigation on an individual appliance, SSH to the individual appliance and type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh enable 
• To disable the mitigation on all hosts from the QRadar Console, type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh disable-all
• To disable the mitigation on an individual appliance, SSH to the individual appliance and type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh disable 

1. After all hosts are updated, advise your team that they must clear their browser cache before logging in to QRadar SIEM.
2. To unmount the /media/cdrom directory on all hosts, type:
/opt/qradar/support/all_servers.sh -C -k “umount /media/cdrom"
3. Delete the ISO from all appliances.

Resolved issues for QRadar Incident Forensics 7.3.1 Patch 4

Some APAR links in the table below might take 24 hours to display properly after a software release is posted to IBM Fix Central.