QRadar: How to move ariel event and flow data between QRadar appliances

How To

Summary

This article describes steps for copying event and flow data from between QRadar hosts. The most common method for users to copy data between appliances is when they are migrating to new hardware. This article describes how to move data with the support tool syncAriel.sh or manually moving data with rsync, then regenerate indexes on the new appliance.

Environment

You can copy event or flow data between QRadar hosts as long as they are (or were) part of the same deployment. The QRadar version does not need to match between the source data and the destination appliance.

Important: The procedure outlined in this technical note expects that completed the appliance migration procedure outlined in the QRadar Administration Guide. You must complete the migration steps, such as restoring a config backup and moving SSH keys to the new server before you use the syncAriel.sh utility to copy your events and flows. For more information, see Migration procedures.

Steps

Method A: Use the syncAriel.sh utility (recommended)

You can use the syncAriel.sh script to copy Ariel data from one QRadar host to another. This method copies all event and flow data from /store/ariel on the source host to the destination. The utility tracks progress for files moved by month. If the script gets interrupted it can pick up where it left off, which is a major advantage if there is a large amount of data to copy. The script is not included in QRadar installs by default. This script copies data from the source host to the destination, with no files or directories at the destination overwritten, and also leaving the source data intact. For a list of other commands, run the script without any parameters.

Before you begin

Ensure you restored your latest nightly configuration backup from your old appliance to your new appliance before you attempt to migrate any data on disk. For more information, see Migration procedures.
Download the syncAriel.sh utility.

1. Set up iptables rules on the destination

To transfer data to a host that is attached to the QRadar deployment, you must create an iptables rule to open port 22 on the destination system. If the destination host is a Console or Log Manager, this procedure is not required as the Console has port 22 open to all hosts.

Use SSH to log in to the QRadar Console as the root user.
Open an SSH session to the managed host that is the destination for the event and flow data (new appliance).
On the destination host, type:
```
vi /opt/qradar/conf/iptables.pre
```
Add the following iptables rule and save your changes:
```
-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>
```
Note: In this example, the data source is the QRadar host you are replacing that contains the data to migrate to a new appliance.
Save your changes to the file.
Run the iptables update script to commit the change.
```
/opt/qradar/bin/iptables_update.pl
```
Note: Any mistake with the rule syntax can cause iptables to fail, use caution when you update rules. The output of running the iptables_update.pl script confirms whether or not iptables was reloaded successfully. If the script reports errors, revert changes and check the syntax of the rule before you try again.
To list iptables rules, type:
```
iptables -L -v -n
```
Review the INPUT chain to confirm the new rule is listed for port 22 to the source host.
You can now test connectivity from the source host CLI by running:
```
ssh <destination IP>
```
For more information, see How to edit iptables rules in QRadar.

Results
You are now ready to set up a nonpassword authentication key (public key) as described in the next step. As multiple SSH sessions are opened, the nonpassword authentication procedure allows the migration to continue without prompting for the root password during the data migration.

2. Setup nonpassword authentication (public key)

The syncAriel.sh script opens multiple SSH sessions during the process. Unless you have nonpassword authentication setup between the source and destination, you are required to enter the password multiple times during the data migration. You need to enable nonpassword authentication (public key) to allow the source to copy the data to the destination without prompting for a password from an administrator.

Procedure

To copy the SSH ID, type:
```
ssh-copy-id <Destination IP>
```
Type the root password for the destination.
This command updates the authorized_keys file of the destination to match the id_rsa.pub key from the source.
Test the connection to ensure no password is needed when you SSH from the source host CLI to the destination IP.
1. If the destination prompts for a password, see Step 4. In recent QRadar versions the SSH configuration is more strict. Administrators might need to generate n rsa-sha2-512 public key before you run the ssh-copy-id command.
2. If you successfully connect without a password prompt, see Using syncAriel.sh to copy data to a destination.
If your initial attempt to run the ssh-copy-id command failed to allow nonpassword authentication, type:
```
ssh-keygen -t rsa -b 2048 -E sha512
```

When prompted, overwrite the existing id_rsa key. For example,

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

Repeat this procedure to copy the SSH ID and verify whether you can successfully SSH without a password prompt as described in step 3b.

3. Using syncAriel.sh to copy data to a destination

The syncAriel utility copies data from the source host to the destination. No files or directories at the destination are overwritten and the source data is left intact on the source host.

Copy the syncAriel.sh script to the source host's "/root" directory
The set permissions on the utility, type:
```
chmod +x syncAriel.sh
```
To start a screen session, type:
```
screen -S syncariel
```
The screen command creates keeps sessions alive regardless of your connection to the host. The screen command is useful for scenarios where your SSH session might be interrupted by a network issue.

Note: If your session is interrupted, SSH to the source host. You can reconnect to the screen session with the command screen -r syncariel.
To sync data between QRadar hosts, type:
```
sh syncAriel.sh -i <destination IP>
```
Wait for the transfer to complete. Depending on the amount of data or network bandwidth limitations, the transfer might take hours or days to complete. The command prompt returns when the data migration is complete.

Note: If you experience 'No such file or directory' errors while rsync runs, the error message is caused when files and folders are no longer present. The most likely cause of this error is when data is removed by event or flow retention before the rsync was able to move the data. You can ignore these error messages.
After your rsync is complete, type the following command to close the screen session:
```
exit
```
Open an SSH session to the destination host.

Remove the iptables rule for the legacy server:

-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>

Save your changes to the file.
To update iptables and commit your change, type:
```
/opt/qradar/bin/iptables_update.pl
```
To confirm the iptables rules for your old appliance is removed, type:
```
iptables -L -v -n
```
Results
After the data is migrated to the new appliance, you must reindex the data to ensure searches perform as expected.

Method B: Use rsync to manually copy data (alternate)

You can use rsync to copy Ariel data from one host to another, similar to the syncAriel.sh script. QRadar Support recommends administrators use the syncAriel.sh utility to move data in bulk between appliances. If you need to copy a small amount of data or specific directories, then rsync can be used. There are disadvantages to using rsync:

Before you begin

Ensure you restored your latest nightly configuration backup from you old appliance to your new appliance before you attempt to migrate any data on disk. For more information, see Migration procedures.
If the copy process is interrupted rsync does not track what files were moved. The entire process must then be restarted.
You must ensure that your rsync command is correct or your data could end up in the wrong directory.

1. Set up iptables rules on the destination

Use SSH to log in to the QRadar Console as the root user.
Open an SSH session to the managed host that is the destination for the event and flow data (new appliance):
On the destination host, type:
```
vi /opt/qradar/conf/iptables.pre
```

Add the following iptables rule and save your changes:

-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>

To update iptables and commit your change, type:
```
/opt/qradar/bin/iptables_update.pl
```
Note: Any mistake with the rule syntax can cause iptables to fail, use caution when you update rules. The output of running the iptables_update.pl script confirms whether or not iptables was reloaded successfully. If the script reports errors, revert changes and check the syntax of the rule before you try again.
To confirm the iptables rules for your old appliance is removed, type:
```
iptables -L -v -n
```
Review the INPUT chain to confirm the new rule is listed for port 22 for the source host's IP address.
You can now test connectivity from the source host CLI by running:
```
ssh <destination IP>
```
For more information, see How to edit iptables rules in QRadar.

2. Copying data with rsync

Use SSH to log in to the QRadar Console as the root user.
Open an SSH session to the legacy host that has the data to migrate.
To start a screen session, type:
screen -S rsync
Note: If your session is interrupted, SSH to the source host. You can reconnect to the screen session with the command screen -r rsync.
Use rsync to move the required files to the new appliance. For example,
- Use rsync to copy all data (not recommended, see syncAriel.sh)
  If administrators want to use rsync to copy all data from one appliance to another, type the following command:
```
rsync -avz /store/ariel/ root@<destination IP>:/store/ariel
```
  Note: If you experience 'No such file or directory' errors while rsync runs, the error message is caused when files and folders are no longer present. The most likely cause of this error is when data is removed by event or flow retention before the rsync was able to move the data. You can ignore these error messages.
- How to copy a specific event or flow directory with rsync
  To rsync a specific directory in /store/ariel, administrators must copy both payloads and records directories. For example, to copy events and flows from Aug 2021 to a new appliance, type the following commands:
```
rsync -avz /store/ariel/events/payloads/2021/8/ root@<destination IP>:/store/ariel/events/payloads/2021/8
```
```
rsync -avz /store/ariel/events/records/2021/8/ root@<destination IP>:/store/ariel/events/records/2021/8
```
  The same logic applies when you copy flow records. When you copy flows, there is only the record that needs to be migrated. If you need to migrate a large amount of data, it is best to use the syncAriel.sh utility.
```
rsync -avz /store/ariel/flows/records/2021/8/ root@<destination IP>:/store/ariel/flows/records/2021/8
```
  Note: If you experience 'No such file or directory' errors while rsync runs, the error message is caused when files and folders are no longer present. The most likely cause of this error occurs when data is removed by event or flow retention before rsync was able to move the data. You can ignore these error messages.
- Moving certificates
  Appliances that collect events need to have their trusted certificates migrated to the new appliance. If this directory is not copied, then log sources on the appliance that attempt to connect to a remote host can fail to handshake or collect data.
```
rsync -avz /opt/qradar/conf/trusted_certificates/ root@<destination IP>:/opt/qradar/conf/trusted_certificates
    
```
  Note: If you experience 'No such file or directory' errors while rsync runs, the error message is caused when files and folders are no longer present. The most likely cause of this error is data being removed by event or flow retention before rsync was able to move the data. You can ignore these error messages.
- Crossover cables and rsync
  If crossover cables are configured between the two appliances, use rsync -av instead of rsync -avz.
```
rsync -av /store/ariel/ root@<destination IP>:/store/ariel
```
  Note: If you experience 'No such file or directory' errors while rsync runs, the error message is caused when files and folders are no longer present. The most likely cause of this error is data being removed by event or flow retention before rsync was able to move the data. You can ignore these error messages.
After your rsync is complete, type the following command to close the screen session:
```
exit
```
Open an SSH session to the destination host.

Remove the iptables rule for the legacy server:

-A INPUT -p tcp --dport 22 -j ACCEPT --src <IP Address (data source)>

Save your changes to the file.
To update iptables to commit the change, type:
```
/opt/qradar/bin/iptables_update.pl
```
To confirm the iptables rules for your old appliance is removed, type:
```
iptables -L -v -n
```
Results
After the data is migrated to the new appliance, you might need to reindex the data to ensure searches perform as expected.

Post-migration: Reindexing your data

Each QRadar appliance that stores event or flow data creates local index files on the appliance to improve search speed. When you move /store/ariel data manually between appliances, reindexing is necessary to ensure old indexes are removed and updated. Indexes allow QRadar running on the host to determine where on disk the data resides so results return quickly. When indexes are not available, a direct scan of the raw data is performed, which can create unnecessary disk (I/O) and CPU load and degrade search speed.

Reindexing your data is required in the following scenarios:

If data migrated for a timeframe that already has data on the destination host.
If data migrated from multiple hosts to a single host where the data has an overlapping time frame.

NOTE: Depending on the amount of data on the host, reindexing data might take a considerable amount of time. It is recommended to use the "screen" command as noted in previous steps to avoid interruptions related to network issues.

Procedure

The time period in the example procedure updates the last two days of data to ensure recent searches are quick for recently migrated data. Administrators must update the date and time in the example commands to ensure the offline indexer utility re-creates indexes.

Use SSH to log in to the QRadar Console as the root user.
Open an SSH session to the host that contains your copied data.
To start a screen session, type:
```
screen -S indexer
```
Important: Update the date and time in the following example commands. QRadar Support typically recommends that administrators re-create indexes for the latest two days of data migrated to estimate the time to complete. Administrators can reindex all of the data, but reindexing can take significant amount of time to complete. Starting with two days can provide a baseline for the administrator.

Remove any 1-minute indexes.

/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/10/25 00:00' -d 2880 -a -r

To rebuild 1-minute indexes for your events, type:

/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/10/25 00:00' -d 2880 -a

Remove the super indexes for the same period:

/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/10/25 00:00' -d 2880 -s -r

To rebuild your super indexes, type:
```
/opt/qradar/bin/ariel_offline_indexer.sh -n events -v -t '2021/08/09 00:00' -d 2880 -s
```
Results
After the indexes are rebuilt, the data migration is complete.

Related Information

QRadar SIEM Hardware Migration Scenarios

Restoring Event Data

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt8AAA","label":"Ariel"},{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Tips