Switching deployment control back to the main site from the destination site

When your main site is ready, switch deployment control back to the main site from the destination site.

Before you begin

CAUTION:

Consider the following scenarios before you proceed to the failback operation.

After the main site is recovered and becomes operational, both the main and destination sites display all referenced managed hosts. However, the main site shows these managed hosts in an 'unknown' state. This occurs because the managed hosts were re-homed to the destination console during the activation process. Complete the following tasks to resolve this issue.
1. Remove the managed hosts that are in the Unknown state from the main site before you run the failback operation on the main site.
2. If the main site MH is configured with HA and the main site is down at the time of activation, it’s required to perform extra steps after powering on the main site to remove HA which is in the Unknown state. If you still cannot remove HA hosts from the console by following the steps in Troubleshooting QRadar HA deployments, contact IBM Support.
3. If all apps are hosted in the AppHost that connects to the main site, you cannot remove the Unknown app host from the main site console. You need to migrate all apps (including Data Synchronization App) configuration and volume data to the main site console from the AppHost, because the current app host is re-homed to the Destination site. Data Synchronization App must work normally on the main site before you run the failback operation. See QRadar: How to force the applications to run on the Console when the App Host is unrecoverable to force the apps to run on the main site console with the Unknown state, and then you can remove AppHost from the main site console.
4. If any managed hosts are still showing as Active, please open a support ticket to identify root cause and and to remove the same from main site console.
When the main site is up and running, but the pairing connection between the main site and the destination site suddenly disconnects. Complete the following tasks to reestablish the pairing connection.
1. From the QRadar Console on the main site, run the following command:
```
/opt/ibm/si/dr/bin/dr_create_ssh.sh -i <destination_site_ip>
```
2. From the QRadar Console on the destination site, run the following command:
```
/opt/ibm/si/dr/bin/dr_create_ssh.sh -i <main_site_ip>
```

You must log in as an admin user to take a fresh backups on both the main site console and the destination site console. After a backup is generated, the system transfers the generated backup to another site. Open the Backup and Recovery screen to check whether the transferred backup is visible. If the transferred backup is not visible, refresh the Backup and Recovery screen.

Failback is the restoration operation to the main site when you no longer require the destination site to replace the main site's operation. When you fail back to the main site, the Ariel copy process copies the Ariel data that was collected. The data comes from the time that the destination site was activated until the end of the hour when the failback process initiated.

The time that the failback process takes can vary based on the following conditions:

The length of time the destination site was active.
The amount of data that was collected on the destination site while it was active.
The bandwidth available between the destination site and the main site.

Ariel copy accepts only time to the nearest hour. The time when failback starts is rounded down to the nearest hour. 9:15 AM is saved as 9:00 AM in the Ariel copy profiles. Ariel copy synchronizes that entire hours worth of data; for example, from 9:00 AM to 9:59 AM.

In this failback process, Ariel sync is completed and receives a notification that the Ariel copy is complete on the destination site. After Ariel sync completion, restoration automatically starts on the main site with the latest backup that was taken after activation on the destination site and transferred to the main site. After the main site restoration completes, the restoration starts on the destination site.

Tip:

If the main site wasn’t available during activation (activation scenario 1), the latest backup from the destination site that the admin initiated before activation is restored on the destination site during the failback process.

For apps restoration, the latest app volume backup from the destination site that the admin initiated before activation is restored on the destination site during the failback process.

If the main site was available during activation (activation scenario 2), the latest main site backup that was transferred to the destination site is restored on the destination site during the failback process.

For apps restoration, the latest app volume backup from main site that was transferred to the destination site is restored on the destination site during the failback process.

Remember:

Apps that are installed on console is only supported during failover and failback operations. If apps are installed on AppHost, then apps are not restored or migrated during the failover and failback operations.

Apps volume backup are being transferred automatically as per daily schedule. However it is advised to take latest volume backup.

To take an app volume backup from the destination site console:
- Apps that run on the console
  1. See Backing up and restoring app data to back up an app volume data from the destination site console.
  2. Transfer the app volume backup from the destination site console to the main site console by running the following command on the destination site console.
```
systemctl start app_sync
```
  3. Verify the transfer on the main site console directory (/store/app_sync/backups). If the transfer is unsuccessful or with issues, copy the app volume backup from the destination site console (/store/apps/backup) directory to the main site console (/store/app_sync/backups) directory.
- Apps that run on AppHost
  1. Move all installed apps to the destination site console
  2. See Backing up and restoring app data to back up an app volume data from the destination site console.
  3. Transfer the app volume backup from the destination site console to the main site console by running the following command on the destination site console.
```
systemctl start app_sync
```
  4. Verify the transfer on the main site console directory (/store/app_sync/backups). If the transfer is unsuccessful or with issues, copy the app volume backup from the destination site console (/store/apps/backup) directory to the main site console (/store/app_sync/backups) directory.
To take an app volume backup from the main site console (Apps that run on the console):
1. See Backing up and restoring app data to back up app volume data from the main site console.
2. Transfer app volume backup data from the main site console (/store/app_sync/backups) to the destination site console (/store/app_sync/backups) directory. This step is required only for the main site that was available during activation scenario 2.
3. Transfer the app volume backup data from the main site console to the destination site console by running the following command on the main site console.
```
systemctl start app_sync
```
4. Verify the transfer on the destination site console directory (/store/app_sync/backups). If the transfer is unsuccessful or with issues, copy the app volume backup from the main site console (/store/apps/backup) directory to the destination site console (/store/app_sync/backups) directory.

Procedure

Initiate the failback operation from the destination site.
1. In the QRadar® Console on the destination site, click Admin > Data Synchronization app.
2. Open the app menu and select Failback to main site.
3. Click Perform failback and then confirm.
4. Point all data sources that were directed to the destination site back to the main site.
5. When the restoration processes are complete on both the main and the destination site consoles:
  1. Perform the full deployment on the main site.
  2. Perform the full deployment on the destination site.
  Note: The full deployment sequence must be maintained to ensure managed hosts are re-homed properly on the main site.
6. On the main site QRadar Console, run the script /opt/ibm/si/dr/bin/dr_clear_seal_files.sh on the hour that the failback was completed to allow for data to be properly synced between sites. See Resynchronizing data that was previously copied on the main site for details.
Reactivate the main site.
1. In the QRadar Console on the main site, click Admin > Data Synchronization app.
2. Open the app menu and select Reactivate main site.
3. Click Reactivate and then Next .
4. On the Admin tab for both the main site and destination site, click Deploy Changes. The main site is now active.

What to do next

After completion of activation, the pairing connection between both the sites is removed. To establish the pairing connection again, you must run the following pairing commands from both the sites:
1. In the QRadar Console on the main site, run the following script:
```
/opt/ibm/si/dr/bin/dr_create_ssh.sh -i <destination_site_ip>
```
  .
2. In the QRadar Console on the destination site, run the following script:
```
/opt/ibm/si/dr/bin/dr_create_ssh.sh -i <main_site_ip>
```
  .
Apps that are installed on AppHost are not restored or migrated during the failover and failback operations. To restore the destination site apps on the main console, take the following steps
1. If a user at the Main Site needs to access an application that was available on the Destination Site but is not accessible from the Main Site, it should be reinstalled by using Main Console site -> IBM QRadar Hub (formerly known as IBM QRadar Assistant) -> Applications -> Installed Extensions section.
  For a successful restore, the app versions must be identical on both the main site and the destination site.
2. Back up volume data of the existing apps on the main site console before you proceed to restoration operations.
  - Ensure that the correct app volume backups are available on the main site console. To restore transferred apps volume backups, copy the apps volume backup data from /store/app_sync/backups to /store/apps/backup.
  - Restore only the necessary apps and the apps of smaller sizes. To restore more apps on the destination site or to keep the apps on the DR site for a longer time, you can migrate the apps from the destination site console to AppHost first before you proceed to the restoration procedure.
  - See Backing up and restoring app data to restore app volume data. The standard practice is to use UUID to and not to restore the Data Synchronization app volume on the destination site console.
  - Data Synchronization app is necessary to maintain its own state and to run failback operation as to activate the main site.
3. If any apps are found in an Error state after restoration is complete or after the failover or failback operation, restart the apps by using the qappmanager utility (/opt/qradar/support/qappmanager).
In Console-Only setup, during failover and failback, only the license key information is restored. The managed host retains the corresponding nonConsoleEventLimit or flowLimit parameters that are defined within the license key. You need to manually reconfigure license pool allocations by using Console Admin -> System and License Management -> Change Display Drop down: Licenses -> License Pool Management.