IBM Support

QRadar: Paired Hosts in Error state in Data Synchronization App

Troubleshooting


Problem

Data Synchronization App UI reports a paired hosts synchronization status of "Error" and the following error repeating in /var/log/qradar/qdr/qdr.log:
[SEVERE ] Disaster Recovery: ArielSync Rsync command failed.
: /store/ariel/events/records/2022/11/21/18 SSH connection Error Code: 255

Symptom

The Data Synchronization Status column in the UI reports an Error for one or more paired hosts:
error status column
Accessing the History tab for the pair shows a list of data synchronization failures:
Data Sync App Error status

Cause

SSH Keys between the host pairs are not valid, missing, or changed. 
Administrators are advised to read the QRadar: Data Redundancy (DR) and support policies, QRadar: Data Synchronization App FAQ and IBM® QRadar Data Synchronization app Documentation to familiarize themselves with these deployments.

Diagnosing The Problem

Confirm the paired hosts authenticate to each other without asking for passwords by using SSH.
  1. Use SSH to log in to the QRadar Console as the root user.
  2. Optional. SSH to the affected managed host.
  3. Attempt to SSH from the affected managed host to the paired host: 
    ssh <PAIRED_HOST_IP>
    Result
    If the previous command request a password or returns an error, close the session and follow the steps in the Resolving The Problem section.

Resolving The Problem

  1. Use SSH to log in to the QRadar Console as the root user.
  2. Optional. SSH to the affected managed host.
  3. Correct the SSH key mismatch by running this command in the affected main site and destination console or managed host: 
    ssh-copy-id <PAIRED_HOST_IP>
    The expected result looks like this: 
    [root@<hostname> ~]# ssh-copy-id -f <PAIRED_HOST_IP>
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
root@<PAIRED_HOST_IP>'s password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '<PAIRED_HOST_IP'"
and check to make sure that only the key(s) you wanted were added.

[root@<hostname> ~]#
  4. Run the dr_create_ssh.sh script between the paired hosts: 
    /opt/ibm/si/dr/bin/dr_create_ssh.sh -i <PAIRED_HOST_IP>
    The expected result looks like this: 
    [root@<hostname>~]# /opt/ibm/si/dr/bin/dr_create_ssh.sh -i <PAIRED_HOST_IP>
DR SSH keys successfully created in /store/ssh/disaster_recovery/.ssh/
Adding <PAIRED_HOST_IP> to .ssh/known_hosts
# <PAIRED_HOST_IP>:22 SSH-2.0-OpenSSH_7.4
# <PAIRED_HOST_IP>:22 SSH-2.0-OpenSSH_7.4
# <PAIRED_HOST_IP>:22 SSH-2.0-OpenSSH_7.4
Preparing to copy '/store/ssh/disaster_recovery/.ssh/ariel_copy' key over to host <PAIRED_HOST_IP>
Preparing to copy '/store/ssh/disaster_recovery/.ssh/config_backup' key over to host <PAIRED_HOST_IP>
DR SSH keys successfully copied to host <PAIRED_HOST_IP>.
  5. Confirm data sync resumes successfully by running: 
    tail /var/log/qradar/qdr/qdr.log
    Output example: 
    [2022-11-21 19:59:38] [INFO   ] Disaster Recovery: Ariel Sync commencing...
[2022-11-21 19:59:38] [INFO   ] Disaster Recovery: Ariel Sync complete for : /store/ariel/events/records/2022/11/21/19 to x.x.x.x. Ariel folder still active. Not sealed.
[2022-11-21 19:59:39] [INFO   ] Disaster Recovery: Ariel Sync complete for : /store/ariel/events/payloads/2022/11/21/19 to x.x.x.x. Ariel folder still active. Not sealed.
[2022-11-21 19:59:39] [INFO   ] Disaster Recovery: Ariel Sync complete for : /store/ariel/flows/records/2022/11/21/19 to x.x.x.x. Ariel folder still active. Not sealed.
[2022-11-21 19:59:39] [INFO   ] Disaster Recovery: Ariel Sync complete.
    Result
    The data synchronization between the paired hosts resumes. The reported status in the Data Synchronization App UI shows "OK":
    image-20221122150527-1
    If the status remains as "Error", contact QRadar Support for assistance.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt3AAA","label":"QRadar Apps"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
30 November 2022

UID

ibm16840879

Page Feedback