QRadar console-only disaster failover

Set up a remote destination site console and switch deployment from the main site to a destination site for deployments with managed hosts where console disaster recovery resiliency is required. All in one deployment can be resolved by using the Data Synchronization app.

For these procedures, the main site is the original site and the destination site is the disaster recovery site.

Use Cases

These procedures are only for the following scenarios:

An actual disaster recovery in which only the Console is unavailable but other deployment hosts are still running.
A disaster recovery exercise in which the main site is still available during the process.

Switching deployment control from the main site console to the destination site console

Before you begin

Ensure that following prerequisites are met before you begin.

Ensure that both IBM® QRadar® consoles are installed with the same software version. Follow the installation wizard to complete the installation on the destination site.
1. Type a root password for the appliance.
2. Type in an IP address and network information for the hardware.
  Important: The destination site IP address must be different that the main site IP address.
3. Log in as a root user and select the appliance type during the installation process.
To ensure consistent performance, ensure that both QRadar consoles have identical hardware resources and network bandwidth.
The remote QRadar console on the destination site requires a different license that is known as a DR license. Contact your sales representative regarding the licensing.
You must back up the QRadar console on the main site. For example, the certificates, configuration, and the data on the main site. To obtain a backup, complete the following task. Creating an on-demand configuration backup archive.
Tip: It is a best practice to back up configuration data, every night, and any custom certificates or custom ssh keys regularly. Store these in a safe location to avoid data loss when there is a main site outage.

Backups can be stored on either the destination site or another location that is accessible by both the main site and destination site.
The backup, that will be restored, must be on the destination site. Place the backup archive into the destination site console’s /store/backupHost/inbound/ directory by typing the following command:
```
scp <backup_location> root@<target_IP_address>:<target_backup_location>
```
For more information, see Importing a backup archive.
If High availability (HA) is configured, it must be removed from both the main and destination site consoles. For more information, see Disconnecting an HA cluster.
If you need to transfer the Ariel data from the main site to the destination site, ensure that both sites are available and have a unique IP address.

About this task

This task might take a considerable amount of time to complete.

Offenses that are restored from the main site console to the destination site console are restored in a closed state. New offenses that are generated on the destination site console, following this procedure, are generated in the normal fashion.

Important:

Do not attach a new data node-managed host to either console during this procedure.

Procedure

If the main site is still available, remove HA on the main site console, if HA is configured.
Stop the IP table service on each managed host in your deployment.
1. Using SSH, log in to the managed host as the root user.
2. For managed hosts except App hosts, type the following command:
```
service iptables stop
```
3. For App Host, type the following commands:
```
systemctl stop docker_iptables_monitor.timer
```
```
systemctl stop iptables
```
4. Repeat for all managed hosts in your deployment.
If the main site is still available, reassign the IP address or hostname on the main site console.
Administrators can change the IP address or hostname of the main site console to an IP address different from the original main or destination IPs. Reassigning the IP address or hostname ensures that if the main site console is ever powered back on that it does not cause packet collisions in the network.

Important: Appliance IP addresses changes are not allowed over SSH connections. You must complete changes by using an Integrated Management Module (IMM), or from the local console keyboard.
1. Using IMM for remote access or the local console keyboard, log in to the command line of the main site console as the root user.
2. Verify which network interface is the management interface by typing the following command:
```
cat /etc/management_interface
```
3. Change the directory to /etc/sysconfig/network-scripts/.
4. Open the ifcfg-<name> file that is listed in the /etc/management_interface file.
5. Change the IP address to an unused or decommissioned range by editing the IPADDR= line, and then save the file.
6. Restart the network by typing the following command:
```
systemctl restart network
```
7. Stop the following services on the main console by typing the following commands:
```
systemctl stop hostcontext
```
```
systemctl stop tomcat
```
```
systemctl stop hostservices
```
```
systemctl stop tunnel_manager
```
```
systemctl stop syslog-ng
```
```
systemctl stop ecs-ec-ingress
```
```
systemctl disable hostcontext tomcat hostservices tunnel_manager syslog-ng ecs-ec-ingress
```
Review your software version. Ensure that the software version on the destination site console matches the one that was on the main. If they do not match, ensure you install the matching on your destination site or patch to that version.

Important: You must not have any managed hosts or HA appliances that are attached to the destination site when you perform the step below.
1. On the destination site run qchange_netsetup.
  Ensure that you enter the main site hostname, the new IP address (initial main site IP address before it was changed) and network information for the hardware.
  
  Important: You might have to update your DNS servers to reflect the change since the IP address for the destination site hostname has changed.
If the main site QRadar console is available, login as the root user and copy the certificates and the custom-generated key pairs from the main site to the destination site. Otherwise, retrieve the certificates and the custom-generated key pairs from the location where the backups of these files are kept.
1. Copy the certificates and keys from the main site to the destination site by using the rsync command to copy the certificate, as in the following example:
  rsync -avz /opt/qradar/conf/trusted_certificates/ root@destination_site:/opt/qradar/conf/trusted_certificates/
  Tip: For better performance when you use a crossover cable solution, use rsync -av instead of rsync -avz.
2. Wait for the transfer to complete.
3. If you are using custom SSL certificates, make sure they are valid for both hosts, by following these steps:
  1. Copy the certificate or intermediate certificate from the /etc/httpd/conf/certs directory on the main site console to the /tmp directory or your preferred location on the destination site console.
    Important: Do not copy the certificate to the /etc/httpd/conf/certs directory on a destination site console.
  2. Install the SSL certificate that you copied on the destination site console by running the command:
```
/opt/qradar/bin/install-ssl-cert.sh -i
```
    and following the instructions.
  The wizard prompts you for a private key. You might have to copy the private key to the server, if it is not stored in the /etc/httpd/conf/certs/ directory. It is a best practice not to store the private key on the server.
Important: If the console on your destination site has a different certificate authority (CA) certificate than the console on your main site, place the CA from your main site in /etc/pki/ca-trust/source/anchors directory and run $ update-ca-trust.
Restore the backup archive on the destination console by completing the following steps:
1. From the Admin tab in the System Configuration section, click Backup and Recovery.
2. Select the relevant archive and click Restore.
3. In the Restore a Backup window, ensure both Select All Configuration Items and Select All Data Items are selected and click Restore.
  For more information on the different items that are restored in these selections, see Restoring a backup archive
4. In the Restore a Backup window. If you are prompted to test host access, click Test Hosts Access.
  1. After testing is complete for all managed hosts, verify that the status in the Access Status column indicates a status of OK.
  2. If the Access Status column indicates a No Access status for a host, follow Step 2 of this procedure to stop IP tables, and then click Test Host Access again to attempt a connection.
5. Click Restore.
6. Click OK to log in to the QRadar Console.
  - If the user interface was closed during the user restore process, open a web browser and log in to QRadar.
  - If the interface was not closed, the login window is displayed. Log in to QRadar.
7. Review the results of the restore process and follow the instructions to resolve any errors.
Refresh your web browser window.
From the Admin tab, select Advanced > Deploy Full Configuration.
QRadar continues to collect events when you deploy the full configuration, but QRadar does not restart the event collection service automatically. Restart the event collection service at the end of the process to make sure all changes are implemented. You can cancel the deployment and restart the services at a more convenient time.
To start the IP tables for an App Host, type the following command:
```
 systemctl start docker_iptables_monitor.timer
```
Transfer any event and flow data to the destination site:
- From the main site console, if available.
- If the main site console is unavailable, transfer the data from the nightly data backups that are stored offsite.
For more information, see Restoring data.
Important: This step can be completed only if the main site and destination site consoles have different IP addresses and the main site console is still available.
The data transfer can be a lengthy process. You can use cross-over cables to quicken the transfer of event and flow information if your appliances are located in the same data center. Data is moved in one month intervals to keep the performance impact at a minimum. The syncAriel.sh utility does not move certificates or configurations, only data that is stored in the /store/ariel/ directory. You must use SSH traffic to migrate the data. You might be required to accept SSH keys and provide the root password for the target server to start the transfer.
1. Download syncAriel.sh from step 7 in the following technote.
  https://www.ibm.com/support/pages/qradar-replacing-console-appliance-deployment-using-same-ip-address-or-hostname-updated#dataxfer
2. Log in to the main site QRadar console as the root user.
3. Navigate to the directory with the syncAriel.sh utility and transfer syncAerial.sh to the main site console and then type the following commands:
```
chmod +x syncAriel.sh
```
```
screen
```
  Tip: To transfer data in a minor network outage, start a screen session to reestablish the connection. To detach the session so that you can log out, type Ctrl+A and press D or use Ctrl+D, then type Ctrl+D and use screen -r to reattach to the screen session.
4. Run the utility by typing the following command:
```
sh syncAriel.sh -I <destination_site_consoles_IPAddress>
```
5. Wait for the transfer to complete, then close the screen session.
  If your connection dropped or a network outage occurred, you can run the syncAriel.sh utility again to migrate data. The syncAriel.sh utility tracks the files that are synchronized to the new appliance and data that was transferred is not copied a second time. If the transfer fails or encounters errors, transfer the data manually by using SCP, SFTP, or another file transfer method.
  Warning:
  The syncAriel.sh script is not supported by IBM. You must resolve any errors that might be contained within the script yourself. The script is provided for convenience.
Shut down the main site.
Install the latest Weekly Auto Update on the destination site console.
For more information about automatic updates and scheduling updates, see Automatic updates.
To verify that your migration is successful, log in as an administrator, click the Log Activity tab and search to see whether events are flowing. Then, click the Network Activity tab and search to see whether flows are being processed.

If the migration is not successful, try the procedure again or contact IBM Support.

Switching deployment control back to the main site from the destination site

When your main site is ready switch deployment control back to the main site from the destination site.

Before you begin

Ensure that all data was transferred and restored correctly before beginning the fail back process.

To complete this procedure, you must first create a backup from the destination site QRadar console. For more information, see Creating an on-demand configuration backup archive.

Upload the backup to the main site QRadar console by placing the backup archive into the main site console’s /store/backupHost/inbound/ directory. Run the following command:

scp <backup_location> root@<destination_site_IP_address>:<target_backup_location>

This should be done regularly. For more information, see Importing a backup archive

Procedure

Rebuild the main site console by following the procedure Reinstalling QRadar from media.

Important: Due to the known issue APAR IJ39521 https://www.ibm.com/support/pages/apar/IJ39521, you must install your new console by using the 7.4.3 GA ISO and then upgrade to the needed 7.5.0 version that matches the destination console.

Rebuilding the main site prevents it from communicating with the managed hosts that reside on the destination site console. The Retain option keeps all data from the main site console. Ensure that you validate all data was transferred to the destination site and data backups exist before proceeding.
1. Select the Retain option, if it is an available option during the rebuild to reduce data transfer.
2. If Retain is not available, choose Flatten if you are certain all data is backed up and on the destination site.
3. Configure the main site console with the IP address that was used as a temporary IP address during the migration to the destination site. The IP address must not be the same as the IP address the main site had originally.
  
  Important: If you plan to configure the main site with HA, add HA to the main site console only at the end of this procedure, and after control is switched back to the main site.
If HA is configured on the destination site console, remove it. For more information, see Disconnecting an HA cluster.
Copy the certificates and custom-generated key pairs from the destination site to the main site to ensure that log sources and scanners can connect to remote sources.

Important: If changes were made to certificates since the destination site became active, ensure you copy them to the main site console.
1. Copy the certificates and keys from the destination site to the main site by using the rsync as in the following example:
```
rsync -avz /opt/qradar/conf/trusted_certificates/ root@destination_site:/opt/qradar/conf/trusted_certificates/
```
  Check the sha256sum of the certificates to make sure they are not corrupted.
  
  Tip: For better performance when you use a cross-over cable solution, use rsync -av instead of rsync -avz.
2. Wait for the transfer to complete.
3. If you are using custom SSL certificates, they are valid for both hosts, following these steps:
  1. Copy the certificate or intermediate certificate from the /etc/httpd/conf/certs directory on the destination site console to the /tmp directory or your preferred location on the main site console.
    Important:
    
    Store these certificates in a location external to QRadar so that they can always be recovered.
    
    Do not copy the certificate to the /etc/httpd/conf/certs directory on a main site console.
  2. Install the SSL certificate that you copied on the destination site console by running the command:
```
/opt/qradar/bin/install-ssl-cert.sh -i
```
    and following the instructions.
  The wizard prompts you for a private key. You might have to copy the private key to the server if it is not stored in the /etc/httpd/conf/certs/ directory. It is a best practice not to store the private key on the server.
  Important: If the console on your main site has a different certificate authority (CA) certificate than the console on your destination site, the CA from your destination site should be placed under the /etc/pki/ca-trust/source/anchors directory and run $ update-ca-trust command.
Stop the IP table service on each managed host in your deployment.
1. Using SSH, log in to the managed host as the root user.
2. For managed hosts except App hosts, type the following command:
```
service iptables stop
```
3. For App Host, type the following commands:
```
systemctl stop docker_iptables_monitor.timer
```
```
systemctl stop iptables
```
4. Repeat for all managed hosts in your deployment.
Reassign the IP address or hostname on the destination site console.
Administrators can change the IP address or hostname of the destination site console to an IP address different from the original main or destination IPs. Reassigning the IP address or hostname ensures that if the destination site console is ever powered back on that it does not cause packet collisions in the network.

Important: Appliance IP addresses changes are not allowed over SSH connections. You must complete changes by using an Integrated Management Module (IMM), or from a physical keyboard.
1. Using IMM for remote access or the local console keyboard, log in to the command line of the destination site console as the root user.
2. Verify which network interface is the management interface by typing the following command:
```
cat /etc/management_interface
```
3. Change the directory to /etc/sysconfig/network-scripts/.
4. Open the ifcfg-<name> file that is listed in the /etc/management_interface file.
5. Change the IP address to an unused or decommissioned range by editing the IPADDR= line and then save the changes to the file.
6. Restart networking by typing the following command:
```
systemctl restart network
```
7. Stop the following services on the destination site console by typing the following commands:
```
systemctl stop hostcontext
```
```
systemctl stop tomcat
```
```
systemctl stop hostservices
```
```
systemctl stop tunnel_manager
```
```
systemctl stop syslog-ng
```
```
systemctl stop ecs-ec-ingress
```
```
systemctl disable hostcontext tomcat hostservices tunnel_manager syslog-ng ecs-ec-ingress
```
Review your software version. Ensure the software version main site consoles matches the one that was on the destination site. If they do not match, install the matching version on your main site or patch to that version.

Important: You must not have any managed hosts or HA appliances that are attached to the main site when you complete the following steps.
1. On the main site run qchange_netsetup.
  Be sure you put in the main site hostname, the new IP (different than the destination site's IP address), and network information for the hardware.
  
  Important: You might have to update your DNS servers to reflect the change since the IP address for the main hostname changed.
Complete the restore on the main console by completing the following steps:
1. From the Admin tab in the System Configuration section, click Backup and Recovery.
2. Select the relevant archive and click Restore.
3. In the Restore a Backup window, ensure both Select All Configuration Items and Select All Data Items are selected and click Restore.
  For more information on the different items that are restored in these selections, see Restoring a backup archive
4. In the Restore a Backup window. If you are prompted to test host access, click Test Hosts Access.
  1. After testing is complete for all managed hosts, verify that the status in the Access Status column indicates a status of OK.
  2. If the Access Status column indicates a No Access status for a host, follow Step 4 of this procedure to stop IP tables, and then click Test Host Access again to attempt a connection.
5. Click Restore.
6. Click OK to log in to the QRadar Console and then open the Web browser and clear the cache.
  - If the user interface was closed during the user restore process, open a web browser and log in to QRadar.
  - If the interface was not closed, the login window is displayed. Log in to QRadar.
7. Review the results of the restore process and follow the instructions to resolve any errors.
Refresh your web browser window.
From the Admin tab, select Advanced > Deploy Full Configuration.
QRadar continues to collect events when you deploy the full configuration, but QRadar does not restart the event collection service automatically. Restart event collection service at the end of the process to make sure all changes are implemented. You can cancel the deployment and restart the services at a more convenient time.
To start the IP tables for an App Host, type the following command:
```
 systemctl start docker_iptables_monitor.timer
```
If you need HA on the main site, it can be added now.
Transfer any event and flow data to the main site.
Important: This step can be completed only if the main site and destination site consoles have different IP addresses.
The data transfer can be a lengthy process. You can use cross over cables to quicken the transfer of event and flow information if your appliances are located in the same data center. Data is moved in one month intervals to keep the performance impact at a minimum. The syncAriel.sh utility does not move certificates or configurations, only data that is stored in the /store/ariel/ directory. You must use SSH traffic to migrate the data. You might be required to accept SSH keys and provide the root password for the target server to start the transfer.
1. Download syncAriel.sh from step 7 in the following technote.
  https://www.ibm.com/support/pages/qradar-replacing-console-appliance-deployment-using-same-ip-address-or-hostname-updated#dataxfer
2. Log in to the destination site QRadar console as the root user.
3. Navigate to the directory with the syncAriel.sh utility and transfer the syncAriel.sh utility to the destination site console and then type the following commands:
```
chmod +x syncAriel.sh
```
```
screen
```
  Tip: To transfer data in a minor network outage, start a screen session to reestablish the connection. To detach the session so that you can log out, type Ctrl+A and press D or use Ctrl+D, then type Ctrl+D and use screen -r to reattach to the screen session.
4. Navigate to the directory with the syncAriel.sh utility and type the following command:
```
chmod +x syncAriel.sh
```
5. Type the following command:
```
screen
```
  Tip: To transfer data in a minor network outage, start a screen session to reestablish the connection. To detach the session so that you can log out, type Ctrl+A and press D or use Ctrl+D, then type Ctrl+D and use screen -r to reattach to the screen session.
6. Run the utility by typing the following command:
```
sh syncAriel.sh -I <destination_site_consoles_IPAddress>
```
7. Wait for the transfer to complete, then close the screen session.
  If your connection dropped or a network outage occurred, you can run the syncAriel.sh utility again to migrate data. The syncAriel.sh utility tracks the files that are synchronized to the new appliance and data that was already transferred is not copied a second time. If the transfer fails or encounters errors, transfer the data manually by using SCP, SFTP, or another file transfer method.
  Warning:
  The syncAriel.sh script is not supported by IBM. You must resolve any errors that might be contained within the script yourself. The script is provided for convenience.
Install the latest Weekly Auto Update on the main site console.
To verify that your migration is successful, log in as an administrator, click the Log Activity tab and search to see whether events are flowing. Then, click the Network Activity tab and search to see whether flows are being processed.

If the migration is not successful, try the procedure again or contact IBM Support.

What to do next

Important: Due to the known issue APAR IJ39521 https://www.ibm.com/support/pages/apar/IJ39521, you must install your new console by using the 7.4.3 GA ISO and then upgrade to the needed 7.5.0 version that matches the destination console.

After control is returned to the main site console and all data is successfully validated on the main site, the destination site console should be rebuilt. This can be done by following the procedure Reinstalling QRadar from media. Ensure you select the Flatten option during the rebuild.

To verify that your deployment is healthy, see https://www.ibm.com/support/pages/qradar-general-health-checklist.