IBM Support

QRadar: Replacing a QRadar Managed Host (16xx, 17xx, 18xx appliance) in your deployment

Troubleshooting


Problem

This technote describes the process that can be used to migrate data from an older QRadar managed host (16xx, 17xx, or 18xx) appliance to newer hardware. This instruction is intended for non-HA appliances.

Resolving The Problem

The procedure outline is intended to get appliances replaced in the network quickly with minimum downtime so they can continue to receive events.

Overview and check list
 

  Before you begin

  Step 1: Preparing your new hardware

  Step 2: Removing the old appliance from the deployment

  Step 3: Reassigning IP addresses on appliances

  Step 4: Adding the new appliance to the deployment

  Step 5: Moving certificates and any custom generated key pairs

  Step 6: How to transfer event and flow data to the new hardware

  Step 7: Optional. How to transfer data for protocols

Before you begin

  • Write down the network information for the old appliance. The administrator needs this information to complete the network configuration for the new appliance.
  • The software version of the new appliance must match the software version of the Console. QRadar does not allow appliances at different software versions in the deployment. Administrators might be required to reinstall an ISO for the appliance to downgrade or use a Fix Pack (SFS) to upgrade on the new appliance.
  • Administrators must back up data to prevent potential data loss.

Step 1: Preparing your new hardware

During this step, the administrator must review and potentially install QRadar on the managed host with the software version that matches the Console. The installation of the new appliance uses a temporary IP address until the old hardware is removed from the deployment later on in this document.

Procedure
 
  1. Rack the appliance.
  2. Plug-in power and network connections.
  3. Review the paperwork for your appliance. The paperwork contains the current software version installed on your new hardware.
  4. Confirm the QRadar software version installed in your deployment:
    • If your new appliance software is older than the software version on the Console, you must upgrade the new appliance. A fix pack (SFS) can upgrade your appliance software to match the version of the Console. Fix pack (SFS) files are available for download on IBM Fix Central.
    • If your appliance software version is newer than the software version on the Console, administrators have two options:
      1. Downgrade the new appliance version with an ISO file, then apply any fix packs to match the Console software version.
      2. Upgrade the Console with a fix pack to bring the entire deployment to the same version as the new appliance. Fix packs have a patch all option to allow the Console to upgrade the entire deployment.
  5. Mount and install QRadar.
  6. Select the appliance type and continue through the configuration wizard.
  7. Type a temporary IP address and network information for the new hardware.
  8. Type a root password for the appliance.
  9. Follow the installation wizard to complete the installation.

    Results
    You are now ready to remove the old appliance from the QRadar deployment


Step 2: Removing the old appliance from the deployment

This procedure has the administrator review any special configuration options for the existing host before you remove it from the deployment.

Procedure
  1. Log in to the QRadar Console as an administrator.
  2. Click the Admin tab and select the System and License Management icon.
  3. From the System list of appliances, select the old QRadar appliance.
  4. Click Deployment Actions > Remove Host.
  5. When prompted, click Remove to confirm the removal of the host from the deployment.
    Important: Deploy Full Configuration results in services being restarted. Reports and searches in progress manually need to be restarted. 
    Schedule a maintenance window before you perform the next step.
  6. Click Advanced > Deploy Full Configuration.

    Results
    You are now ready to reassign IP addresses for the old appliance and the new replacement appliance.


Step 3: Reassigning IP addresses on appliances

After the managed host is removed from the deployment, administrators can change the IP address of the old managed host to an unused or decommissioned IP range. Reassigning the IP address or hostname ensures that if the decommissioned appliance is ever powered back on that it does not cause packet collisions in the network.

Note: Appliance IP addresses changes are not allowed over SSH connections to change. An IP address change can be completed by remotely connecting with IMM or from a physical keyboard to prevent connection and lockout issues.

3a. Reassign the IP address of the old appliance to a decommissioned range
This procedure reassigns the IP address of the old appliance to a new address space or a decommissioned address space to free up the existing IP address for the new hardware.
  1. Connect with IMM for remote access or a local keyboard.
  2. Log in to the old appliance as the root user.
  3. To reassign the IP address of the old appliance, type /opt/qradar/bin/qchange_netsetup to change IP address to a decommissioned IP range.

3b. Setting the IP address for the new hardware
This procedure assigns the existing IP address to the new hardware so the new appliance can be added to the deployment with the old hardware's IP address.
  1. Connect with IMM for remote access or a local keyboard.
  2. Log in to the new appliance as the root user.
  3. From the command line of the new appliance, type /opt/qradar/bin/qchange_netsetup to use same hostname and IP address as old server.

    Results
    The IP addresses and hostnames are updated for both appliances. The administrator can now add the new hardware to the QRadar deployment. Administrators must not unrack the old hardware yet, as data still exists on the old appliance that needs to be transferred to the new appliance.

Step 4: Adding the new appliance to the deployment

The new appliance must be at the same software version of QRadar Console or the appliance being replaced.
Procedure
  1. Log in to the QRadar Console as an administrator.
  2. Click the Admin tab and select the System and License Management icon.
  3. Click Deployment Actions > Add Host.
  4. If you receive a prompt to add components from the deployment to the host, select Yes.
    Any deployed components that were on the old appliance are reassociated to this host. Any protocol-based configurations are automatically enabled and migrated to the new appliance.
  5. Click Save.
  6. Click Close.
  7. From the Admin tab, click the Deploy Changes icon.
  8. Verify that event or flow sources that were reporting to the original host are being processed in the QRadar user interface.

    Results
    After the host is added back to the QRadar deployment, the deploy process will ensure that required configuration is regenerated on the new appliance. The administrator can verify that Log Source data is being pulled and that flow data is being received by the new hardware. Any log sources that are not collecting data might require certificates to be moved to the new host.
 

Step 5: Moving certificates and custom generated key pairs (as required)

Appliances that manage scanners and log sources that authenticate must copy the certificates from the old appliance to the new appliance to ensure that log sources and scanners can connect to remote sources. Administrators who use custom generated private keys need to migrate the keys by transferring the /etc/ssh and /root/.ssh directories.
 
Procedure
  1. Log in to the QRadar old managed host as the root user.
  2. To copy the data from the old hardware to the new appliance (IPAddress), type the following command:
    For certificates,
      rsync -avz /opt/qradar/conf/trusted_certificates/ root@IPAddress:/opt/qradar/conf/trusted_certificates
    NOTE: If a crossover cable is plugged in to complete the data transfer, use the rsync -av option.

    For SSH keys,
      rsync -avz /etc/ssh/ root@IPAddress:/etc/ssh
      rsync -avz /root/.ssh/ root@IPAddress:/root
  3. Wait for the transfer to complete.

    Results
    The required certificate and custom key files are transferred to the managed host. The administrator is now ready for the final step, which is to migrate event and flow data from the old appliance to the new hardware.
 

Step 6: How to transfer event and flow data to the new hardware

The attached utility syncAriel.sh was designed to facilitate moving data from /store/ariel of an old appliance to a new appliance. Data is moved in one month intervals to keep performance impact at a minimum. This utility does not move certificates or configurations, only data in /store/ariel. SSH traffic on TCP port 22 must be allowed to migrate the data. The administrator might be required to accept SSH keys and provide root password for the target server to stat the transfer.

IMPORTANT: If the administrator does not transfer private keys between hosts as outlined in a previous step, they are prompted to type a password each time the syncAriel.sh utility attempts to move a directory. The data transfer can be a lengthy process. Appliances can use cross-over cables when located in the same data center to expedite the data transfer.
 
  1. To copy data from the old host to the new appliance, download the syncAriel.sh utility: syncAriel.sh
  2. Use SSH to log in to the old host as the root user.
  3. Copy the file to the old host, for example the /tmp directory.
  4. Navigate to the directory with the syncAriel utility and type:
      chmod +x syncAriel.sh
  5. To start a screen session before your data transfer, type:
      screen
    NOTE: It is recommended that the administrator starts a screen session to reestablish the connection in case of a minor network outage. To detach the session so you can log out, type Ctrl-a and press d or use Ctrl+a, then Ctrl+d and use screen -r to reattach to the screen session.
  6. To run the utility, type:
      sh syncAriel.sh -i IPAddress
    Where IPAddress is the address of the new appliance. For a list of other commands, run the script without any parameters.
  7. Wait for the transfer to complete.
  8. Close the screen session.

    Results
    Data is migrated from /store/ariel of the old managed host to the new managed host appliance. The transfer can take a long time to complete based on how much data needs to be transferred.
 

Step 7: Optional. How to transfer data for protocols

This section applies to administrators who have appliances that remotely poll for data, such as AWS, JDBC, Netskope, Google Gsuite, and other REST API protocol types. Appliances that remotely poll for data can create marker files to track the last timestamp. These files include information on polling position for each protocol on the local appliance in /store/ec. Administrators who fail to move the data in /store/ec might poll and reprocess old events.

For example,
  /store/ec/googlegsuiteactivityreportsrestapi/configId-5.properties
Tue Jan 21 09:33:02 EST 2020  
drive_LastQueryTime=2018-01-13T21\:24\:31.700Z  
user_accounts_LastQueryTime=2018-12-04T19\:33\:09.640Z  
admin_LastQueryTime=2018-01-13T21\:09\:53.205Z  
login_LastQueryTime=2018-01-13T20\:23\:57.708Z

Procedure
  1. Log in to the old Event Collector appliance as the root user.
  2. Stop ecs-ec-ingress on the old appliance by typing the following command:
      systemctl stop ecs-ec-ingress
  3. Log in to the new appliance as the root user.
  4. Create a file on the new appliance to prevent ecs-ec-ingress from automatically restarting by typing the following command:
      touch /storetmp/ecs-ec-ingress.ecs-ec-ingress.manually_stopped
  5. Stop ecs-ec-ingress on the new appliance by typing the following command:
      systemctl stop ecs-ec-ingress
  6. Copy the data from /store/ec on the old appliance to /store/ec on the new appliance.
  7. Remove the file created in substep d from the new appliance by typing the following command:
      rm -f /storetmp/ecs-ec-ingress.ecs-ec-ingress.manually_stopped
  8. Start ecs-ec-ingress on the new appliance by typing the following command:
      systemctl start ecs-ec-ingress
  9. Wait for the data migration to complete.
  10. Start the iptables services, type:
      systemctl start iptables

     
What to do next
The administrator can decommission the old appliance and unrack the obsolete hardware.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtdAAA","label":"Upgrade"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
01 November 2021

UID

swg21983313