IBM Support

QRadar: How to Update Appliances in Parallel

Question & Answer


Question

Updating in parallel allows adminsitrators to save on downtime by first patching the Console, then applying the update to all other appliances simultaneously. This article walks through process of how to update appliances in parallel.

Answer

 

What are the update methods supported by QRadar?


QRadar has two methods of completing updates using SFS files:
 
  • Patch all - Patch all is an easy method of upgrading QRadar as administrators can mount and run the installer only on the console and select the Patch All option. This option upgrades the Console appliance first, then copies the necessary files to each managed host and starts the install in a serial method.
    - Pro: Easy to deploy, just run from the Console and walk away. A summary of the update is provided to the administrator on the Console.
    - Pro: Appliances are only down one at a time, so you are only not collecting data while that one appliance is patching.
    - Con: This upgrade method can take a long time for larger deployments.
    - Con: The all option uses deployment information to update appliances and there is no method of applying an order to how appliances are updated. The system walks through the deployment and there is no way to tell which appliance is going to be updated next.
     
  • Updating in Parallel - Parallel updates take more manual intervention from an administrator, but can drastically reduce the amount of time required to update a large deployment to the latest version. In this scenario, parallel updates allow administrators to control the order that the appliances are patched and can start a number of updates simultaneously to prevent extended downtime. For example, the administrator might choose to simultaneously update EP1, EP2, and EP3, and set the load balancer to forward all data to EP4.
    - Pro: The fastest method of upgrading a deployment.
    - Pro: - Fairly easy to distribute the SFS and mount files using the all_servers.sh support tool.
    - Pro: - Administrators can decide the order in which appliances are updated or start updates in parallel for groups or regions to reduce downtime.
    - Con: When you update in parallel, all appliances you update simultaneously are not collecting data.


Here is a visual example of the difference in process:
Installation type Step 1 Step 2 Step 3 Step 4 Step 5
Patch all (incremental) Mount file and install on the Console (1h - 1.25 hours) File copied automatically to EP1 and EP1 installs (45m) File copied automatically to EP2 and EP2 installs (45m) File copied automatically to EP3 and EP3 installs (45m) File copied automatically to EP4 and EP4 installs (45m)
Updating in Parallel Mount file and install on the Console (1h - 1.25 hours) Copy files to all hosts and start updates simultaneously for all managed hosts (45m)  
Time savings: Over 2 hours


IMPORTANT: The Console expects all appliances to be on the same version. Meaning that they can stay at a different version temporarily, but administrators are expected to ensure that the entire deployment is updated. Users concerned about downtime should patch their QRadar managed hosts in parallel.
When you upgrade a QRadar deployment, all appliances are intended to be at the same software version. If you update the Console and leave EPs in the network at the old version they will still collect data, however, the deployment might not work as expected. For example, offenses sent from the EP to the Console might not trigger rules properly, searches might not complete as expected, and several other issues can occur. 



Restrictions
This procedure does not apply to HA secondary appliances, as these systems are updated by the primary host. This procedure also does not apply to QRadar Packet Capture appliances, as they use a different SFS file for their appliance updates.



Where to ask questions
Let us know if you have questions before you start your update. It is best to talk through any issues or concerns first so we can review and assist where possible. Especially if you have HA or questions about patching with HA as that was not discussed in this post. Our forums are a great place to ask a question before you begin a patch to a new version. To participate in our forums, see: http://ibm.biz/qradarforums.



Before you begin
Ensure that you take the following precautions:

  • Back up your data before you begin any software upgrade. For more information about backup and recovery, see the IBM Security QRadar Administration Guide.
  • To avoid access errors in your log file, close all open QRadar sessions.
  • Verify there are no Deploy Changes pending on the Console.
  • The file referred to in this procedure is an .SFS patch file which is only capable of performing upgrades/patches to existing QRadar installations. Administrators who want to complete a new appliance install of QRadar need to review the QRadar Installation Guide.
  • HA secondaries are upgraded by the primary host. There is not need to copy or run the patch on an HA secondary; however, you should confirm that primary HA hosts are active and secondaries are in Standby mode and not offline before upgrading.

How to upgrade a QRadar deployment in parallel


Part 1: Update the Console first
The instructions guide administrators through the process of upgrading an existing QRadar version at 7.2.4 or later to the newest software version.
 
  • Procedure
    1. Download the fix pack from the IBM Fix Central website:
    2. Using SSH, log in to your QRadar Console as the root user.
    3. Copy the fix pack to a directory on the QRadar Console that has sufficient space for the SFS file.

      For QRadar 7.2.x: The best directories for the SFS file are as follows:
      1. /tmp (best) 
      If you leave a file in /tmp for 10 days without completing the SFS update, it is likely that the Red Hat's tmpwatch cron job will run and the file will be removed from the appliance.
      2. /store/tmp (better)
      3. /store/transient (okay)

      QRadar 7.3.x: The best directories for the SFS file are as follows:

      1. /store/tmp (best)
      The storetmp directory is available on all appliance types. This directory can be cleaned by tmpwatch the tmpwatch cron job if you postpone your update.
      2. /tmp (better)
      This directory is available on all appliances, but in 7.3.0 versions is significantly smaller and moving a file here can cause services to stop. If you leave a file in /tmp for 10 days without completing the SFS update, it might get cleaned up by Red Hat's tmpwatch cron job.
      3. /store/transient
      The store/transient directory was introduced in QRadar 7.2.1 and is allocated 10% of the overall /store directory. However, this directory does not exist on all appliances, such as QFlow or QRadar Network Insights and might not be an actual partition on all appliances.
       
    4. To create the /media/updates directory, type the following command: mkdir -p /media/updates 
    5. Change to the directory where you copied the patch file.
    6. To mount the patch file to the /media/updates directory, type one of the following commands:

      - For QRadar 7.2.x: mount -o loop -t squashfs /tmp/728_QRadar_patchupdate-(version).sfs /media/updates
      - For QRadar 7.3.x: mount -o loop -t squashfs /store/tmp/728_QRadar_patchupdate-(version).sfs /media/updates
       
    7. To run the patch installer, type the following command: /media/updates/installer.
      NOTE: The first time that you run the fix pack, there might be a delay before the fix pack installation menu is displayed.
    8. Select the option to patch the Console.
      NOTE: If the SSH session is disconnected, reconnect to the host you are upgrading and re-run the installer. The system will take you back to the current step in the patch process.
    9. After the patch completes and you have exited the installer, type the following command: umount /media/updates 
    10. Administrators and users should clear their browser cache before logging in to the Console.

      Results
      The Console appliance is updated. Administrators can now proceed to Part 2 to learn how to distribute the SFS and mount files using the all_servers support tool.



  •  
 

Part 2: Update your managed hosts


This section informs administrators how to parallel patch a QRadar deployment. It is critical that customers verify they have space to install the fix pack. You can use the Console to stage the SFS file to all managed hosts and verify that you have disk space available on all hosts.

As a general rule of thumb, a system should have enough space equivalent to twice the size of the fix pack in the root directory. If the system does not have enough disk space to install the fix pack, the appliance is bypassed and a summary details which managed hosts were installed successfully and which were unsuccessful.
  1. Using SSH, log in to the QRadar Console as the root user.
  2. Verify that none of your appliances are out of disk space before you move a large file. If you want, you can run /opt/qradar/support/all_servers.sh -k "df -h /tmp /store/tmp /store/transient" | tee diskchecks.txt
    This command will export a list of partitions and the free space.
  3. From the command line of the Console, type the following command to copy the patch file (SFS) to all managed hosts:

    For QRadar 7.2.x: /opt/qradar/support/all_servers.sh -p <QRadar file name>.sfs

    For QRadar 7.3.x: /opt/qradar/support/all_servers.sh -p /store/tmp/<QRadar file name>.sfs -r /store/tmp 

    The -r flag is required for QRadar 7.3.x systems as by default all_servers will put the SFS file in the /tmp directory
  4. Type the following command to ensure the target directory exists on all managed hosts: /opt/qradar/support/all_servers.sh "mkdir -p /media/updates && umount /media/updates"
  5. Mount all of the files on managed hosts using all_servers by typing:/opt/qradar/support/all_servers.sh "mount -o loop -t squashfs /tmp/728_QRadar_patchupdate-(version).sfs /media/updates"
  6. SSH to each managed host individually and type: /media/updates/installer

    Important: Do not launch the installer using the all_servers command. Attempting to run the installer using all_servers.sh will prevent errors from being displayed to the end user and can prevent the pretest from running properly.
  7. This will start a screen session and start the installer menu.
  8. Select Yes to update the managed host.
  9. Repeat step 6 to 8 for each managed host in the deployment.

    NOTE: If the SSH session is disconnected, reconnect to the host you are upgrading and re-run the installer. The system will take you back to the current step in the update process.
  10. Wait for each appliance to finish as they all update in parallel. To check on the status of an individual host, you can SSH to that host directly and run the installer to reconnect, if disconnected.
 Results
A summary messages will be displayed as each managed host completes the update or any errors are displayed. If you experience an upgrade error, you should contact QRadar Support (http://ibm.biz/qradarsupport).


Where do you find more information?
  QRadar Documentation QRadar Forums QRadar Knowledge QRadar Software QRadar Notifications QRadar Support QRadar YouTube

[{"Product":{"code":"SSBQAC","label":"IBM QRadar SIEM"},"Business Unit":{"code":"BU008","label":"Security"},"Component":"Upgrade","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.2;7.3","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
02 December 2019

UID

swg21998517