Release of the QRadar Network Insights 7.3.1 Patch 4 ISO (7.3.1.20180507202600)

Content

Important advisories in this update

This QRadar Network Insights software update includes an operating system mitigation for the side-channel analysis vulnerability known as Spectre (Variant 1) and Meltdown (Variant 3). Due to a potential change to search performance in Meltdown (Variant 3), a new installation prompt is displayed to administrators so they can decide how the mitigation is applied. Installation of this update is not intended to, nor will it provide mitigation measures against Spectre Variant 2 (CVE-2-17-5715). The mitigation for Variant 2/Spectre requires an OEM microcode/BIOS update on appliances. For information about other variants for Spectre and Meltdown, see the following IBM Flash Notice: http://www.ibm.com/support/docview.wss?uid=swg22012320.

This update contains a mitigation for CVE-2017-5754 Variant 3/Meltdown provided by Red Hat that can impact search performance. Administrators must read the release notes before they install this update.

Choices:
1) Enable: Turn ON the mitigation for Variant 3/Meltdown on all appliances.
2) Disable: Turn OFF the mitigation for Variant 3/Meltdown on all appliances. IF YOU CHOOSE NOT TO ENABLE THIS UPDATE TO ADDRESS CVE-2017-5754, YOU WILL NOT HAVE ANY PROTECTION AGAINST VARIANT 3/MELTDOWN.
3) Abort patch.


CVEID: CVE-2017-5753 (Variant 1/Spectre)
MITIGATION: Enabled by default during the installation of QRadar Network Insights 7.3.1 Patch 4 and cannot be disabled.
DESCRIPTION: Intel Haswell Xeon, AMD PRO and ARM Cortex A57 CPUs could allow a local authenticated attacker to obtain sensitive information, caused by a bounds check bypass in the CPU speculative branch instruction execution feature. By conducting targeted cache side-channel attacks, an attacker could exploit this vulnerability to cross the syscall boundary and read data from the CPU virtual memory.
IMPACT: No performance impact.

CVEID: CVE-2017-5754 (Variant 3/Meltdown)
MITIGATION: An installation prompt is provided to enable or disable this mitigation on appliances. A utility is also provided to allow administrators to enable or disable the mitigation for CVE-2017-5754 (Variant 3/Meltdown) post-installation, see the Installation wrap-up for further details. IBM cannot be held responsible for risks incurred by administrators who do not enable the mitigation of CVE-2017-5754 (Variant 3/Meltdown).
IF YOU CHOOSE NOT TO ENABLE THIS UPDATE TO ADDRESS CVE-2017-5754, YOU WILL NOT HAVE ANY PROTECTION AGAINST VARIANT 3/MELTDOWN.
DESCRIPTION: Intel Haswell Xeon, AMD PRO and ARM Cortex A57 CPUs could allow a local authenticated attacker to obtain sensitive information, caused by a rogue data cache load in the CPU speculative branch instruction execution feature. By conducting targeted cache side-channel attacks, an attacker could exploit this vulnerability to cause the CPU to read kernel memory from userspace before the permission check for accessing an address is performed.
IMPACT: Search performance degradation has been observed on appliances when the mitigation for Variant 3/Meltdown is enabled. Administrators who want to evaluate search duration on appliances before they apply the mitigation can review the following technical note: Search performance evaluation for CVE-2017-5754 (Variant 3/Meltdown).

Firmware notice
The mitigation for Variant 2/Spectre requires an OEM microcode/BIOS update on appliances. For more information on firmware releases, see: https://ibm.biz/qradarfirmware.

Performance assessment summary
Administrators can expect performance degradation after they enable the mitigation for the vulnerability.

• A 3% to 6% increase in CPU utilization has been observed across all workloads on appliances after the mitigation applied.
• Search performance for most common search types has been observed to degrade by 0% to 10%, with the following exceptions:
• Searches that use indexed criteria and match a moderate number of results (less than 10% of the total searched dataset) are expected to be degraded between 3% to 20%.
• Open-ended searches that have no limit applied to the query and return a very large number of results (30% of the total searched dataset or more) are expected to be degraded by up to 2x.
• The impact on data processing is estimated to be in the 0% to 20% range.
• High availability on 1 Gbit network is not affected. The initial high availability setup speed and catch-up replication speed after fail-over will be lower on 10 Gbit network. However, the replication rate is still in the multiple hundreds MB/s, which is sufficient for real time replication.

About this upgrade

QRadar Network Insights 7.3.1 Patch 4 uses an ISO file to update QRadar Network Insights 7.2.8 hosts to the latest software version. A minimum of QRadar 7.2.8 Patch 3 (or later) is required to be able to upgrade to QRadar Network Insights 7.3.1 Patch 4. Each host must be updated individually.

Current Version	ISO Upgrades to QRadar Network Insights 7.3.1 Patch 4?
QRadar Network Insights 7.2.8 Patch 3 or later	Yes, use these release notes to complete this process.
QRadar Network Insights 7.3.0 (any patch version)	See the QRadar 7.3.1 Patch 4 SFS release notes.
QRadar Network Insights 7.3.1	See the QRadar 7.3.1 Patch 4 SFS release notes.

Figure 1: If you have installed QRadar Network Insights 7.2.8, you are not required to install each ISO release to update to the latest version.

Administrator notes

1. The QRadar Console must be updated before you update your QRadar Network Insights appliance.
2. If you have a software installation, you need your Red Hat Enterprise ISO and the QRadar Network Insights ISO. Administrators with software installations on their own hardware must read the QRadar Network Insights Install Guide to understand how to partition their systems appropriately.
3. QRadar Network Insights must be on QRadar 7.2.8 Patch 4 or later to upgrade.
4. The upgrade from QRadar 7.2.8 Patch 3 to QRadar Network Insights 7.3.1 Patch 4 will use a .ISO file. Previously, ISOs were for new appliance installations only, but QRadar 7.3.1 Patch 4 is an exception to this rule because of the Red Hat kernel update requirements.
5. The ISO must be mounted to the appliance and run locally on each host.
6. The 7.3.1 Patch 4 upgrade will take longer than expected due to the kernel changes to Red Hat 7 Enterprise. Early upgrade customers are reporting 2 to 2.5 hours to upgrade the Console appliance. You should be aware of this longer time frame and plan your maintenance accordingly.
7. If power users created utilities or custom scripts for their QRadar deployment, these files should be copied off the system. During the 7.3.1 Patch 4 update, a warning is displayed that only data in /store is going to be preserved. After the appliance reboots, any scripts, 3rd party accounts, or utilities in /tmp, or /, or /root will be deleted. This does not impact ISO files mounted initially using /root because this cleanup only occurs later in the installation procedure.
   

Before you upgrade

Ensure that you take the following precautions:

• Back up your data before you begin any software upgrade and verify that you have recent configuration backups that match your existing Console version. If required, take an on demand configuration backup before you begin. For more information about backup and recovery, see the IBM QRadar Administration Guide.
• All appliances in the deployment must be at the same software and patch level in the deployment.
• Verify that all changes are deployed on your appliances. The update cannot install on appliances that have undeployed changes.
• A QRadar Network Insights 7.3.1 Patch 4 ISO is available to upgrade from QRadar 7.2.8 Patch 4 or install a new appliance or virtual machine. To complete a new installation, review the QRadar Network Insights Installation Guide.
• If you are unsure of the IP addresses or hostnames for the appliances in the deployment, run the utility /opt/qradar/support/deployment_info.sh to get a .CSV file with information about the QRadar deployment. The CSV file will contain a list of IP addresses for each managed host.
• If you are unsure of how to proceed when reading these instructions or the documentation, it is best to ask before starting your upgrade. To ask a question in our forums, see: http://ibm.biz/qradarforums.

Part 1. Staging files and pretesting your deployment (required)

It is important that you pretest your deployment to ensure that you will not experience unexpected issues when you update to QRadar 7.3.1 Patch 4. A pretest is a common precaution that should be taken before you install an update to locate potential issues. The pretest does not restart services and can be completed without scheduled downtime. The pretest typically takes between 3 to 5 minutes to complete on each appliance. If for some reason your SSH session is disconnected, you can reconnect to the remote host using screen.

Procedure
The pretest should be completed on all hosts by the administrator before you attempt to upgrade to QRadar 7.3.1 Patch 4.

1. Download the QRadar Network Insights 7.3.1 Patch 4 ISO (5.2 GB) from the IBM Fix Central website: https://www-945.ibm.com/support/fixcentral/swg/selectFixes?product=ibm%2FOther+software%2FIBM+Security+QRadar+Incident+Forensics&fixids=7.3.1-QRADAR-QIFFULL-20180507202600&source=SAR&function=fixId&parent=IBM%20Security
   
   NOTE: QRadar Network Insights and QRadar Network Insights use the same ISO for both products to upgrade from 7.2.8 to 7.3.1 Patch 4.
   

2. Use SSH to log in to your Console as the root user.
3. Type the following command: screen
4. To make the directory for the update, type: mkdir -p /media/cdrom
5. To verify that you have enough space (5.2 GB), type: df -h /root /var/log | tee diskchecks.txt
   
• Best directory option: /root
  It is available on all appliance types, and is the best option to host the ISO file.
• 2nd best directory option: /var/log
  This directory is available on all appliances, but there might not be the required space available.
• DO NOT USE: /tmp, /store/tmp, or /store/transient for your ISO upgrade. These directories are partitioned as part of the upgrade and administrators cannot use them as storage locations or mount points for the ISO file.
  
  If the disk check command fails, retype the quotation marks from your terminal, and then re-run the command. This command returns the details to both the command window and to a file on the Console named diskchecks.txt. Review this file to ensure that all appliances have a minimum of 5.2 GB of space available in a directory to copy the ISO before attempting to move the file to a managed host. If required, free up disk space on any host that fails to have less than 5.2 GB available.
  
  Reminder: Utilities or custom scripts that administrators have created for QRadar should be copied off the system. During the 7.3.1 update a warning is displayed that only data in /store will be preserved. Therefore, scripts, 3rd party utilities in /tmp, or /, or /root will be deleted during the upgrade.
  

6. If there is not 5.2GB of space in /root or /var/log, the administrator must make directory space for the ISO file.
7. Use WinSCP or SCP to copy the ISO to the /root or /var/log directory on the QRadar Network Insights appliance.
8. To mount the ISO on all appliances, type mount -o loop /root/Rhe764QRadar7_3_1_20180507202600.stable-7-3-1.iso /media/cdrom
9. To pretest the Console appliance, type: /media/cdrom/setup -t
   The pretest output is written to the command window. Review this output after the pretest completes.
10. Use SSH to open an SSH session to the other appliances in your deployment. QRadar Support recommends that all administrators run the pretest on each host to identify issues before the update begins.
11. To pretest the managed host, type: /media/cdrom/setup -t
    
    Results
    If an appliance in your deployment fails the pretest, take the recommended action from the pretest utility. The issue must be resolved before the update to 7.3.1 Patch 4 begins to prevent downtime for specific appliances. If there are messages you do not understand or want to discuss further, you can use our forums http://ibm.biz/qradarforums to get advice. Administrators can open a ticket directly with QRadar Support (http://ibm.biz/qradarsupport).

Part 2. Installing the QRadar Network Insights 7.3.1 Patch 4 ISO

These instructions guide administrators through the process of upgrading an existing QRadar Network Insights install at 7.2.8 Patch 1 or later to QRadar 7.3.1 Patch 4. The update on the Console must be completed before you attempt to update QRadar Network Insights.

Procedure

1. Use SSH to log in to the Console as the root user.
2. Use SSH to log in to the QRadar Network Insights appliance.
3. To run the ISO installer on the Console, type the following command: /media/cdrom/setup
   
   Important: Upgrading from QRadar Network Insights 7.2.8 Patch 1 or later to QRadar Network Insights 7.3.1 Patch 4 can take approximately 1 to 2 hours.
   

Results
A summary of the ISO installation advises you of any issues. If there are no issues, administrators can use SSH to log in to managed hosts and start the installer on each host to run the setup in parallel.

Part 3. Installation wrap-up

1. After all hosts are updated, send an email to your team to inform them that they will need to clear their browser cache before they log in to QRadar SIEM.
2. To unmount the /media/cdrom directory on all hosts, type: umount /media/cdrom"
3. Administrators can delete the ISO from all appliances.
4. Review any static routes or customized routing. As mentioned in the administrator notes, all routes were removed and must be reconfigured after the upgrade completes.
5. Review all configured iptable rules because the interface names have changed in QRadar 7.3.1 Patch 4 due to the Red Hat Enterprise 7 operating system updates. Any iptables rules that use the Red Hat 6 interface naming conventions must be updated.

Part 4. Installation wrap-up

1. Due to the potential search performance change when CVE-2017-5754 (Variant 3/Meltdown) is enabled, the installation of QRadar 7.3.1 Patch 4 includes a utility to allow administrators to enable or disable the mitigation after the initial installation completes. Administrators must be aware of the security implications if they choose to use this utility to disable the mitigation for CVE-2017-5754 (Variant 3/Meltdown). IBM cannot be held responsible for risks incurred by administrators who choose to disable the mitigation for CVE-2017-5754.
• To enable the mitigation on all hosts from the QRadar Console, type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh enable-all
• To enable the mitigation on an individual appliance, SSH to the individual appliance and type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh enable 
• To disable the mitigation on all hosts from the QRadar Console, type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh disable-all
• To disable the mitigation on an individual appliance, SSH to the individual appliance and type: /opt/qradar/bin/configure-spectre-meltdown-fixes.sh disable 
2. After all hosts are updated, advise your team that they must clear their browser cache before logging in to QRadar SIEM.
3. To unmount the /media/cdrom directory on all hosts, type:
   /opt/qradar/support/all_servers.sh -C -k “umount /media/cdrom"
4. Delete the ISO from all appliances.