IBM Support

QRadar: Replication bandwidth requirements and verifying speed between console and managed host

Troubleshooting


Problem

This document discusses some pitfalls of having a slower connection between the console and a managed host, with details on how to test the network speed.

Symptom

You might find that a particular host (or hosts) in your environment frequently fail to deploy changes due to insufficient network speed between the console and the host.

Diagnosing The Problem

One indication that the bandwidth between your console and your managed host is too slow is to look at how long replication takes to complete. To view how long replication takes, SSH to the console and then SSH to the managed host in question. On the managed host, run the following:
grep "Replication download" /var/log/qradar.log | tail

Example output:

[root@HOSTNAME ~]# grep "Replication download" /var/log/qradar.log | tail
Jun 29 15:41:08 HOSTNAME replication[13866]: Replication download timing: Downloading: 427 ms Overall: 453 ms DB transaction: 20 ms Transaction verification: 6 ms
Jun 29 15:42:08 HOSTNAME replication[724]: Replication download timing: Downloading: 399 ms Overall: 428 ms DB transaction: 21 ms Transaction verification: 6 ms
Jun 29 15:43:08 HOSTNAME replication[19562]: Replication download timing: Downloading: 403 ms Overall: 431 ms DB transaction: 22 ms Transaction verification: 6 ms
Jun 29 15:44:08 HOSTNAME replication[5264]: Replication download timing: Downloading: 399 ms Overall: 426 ms DB transaction: 20 ms Transaction verification: 6 ms
Your results display the 10 most recent replication downloads, occurring each minute. Find the "Downloading:" part of the messages and note how long it took to complete. Ideally, replication needs to be under 3000 ms. If these are taking longer than 5000 ms to complete, you likely have some bandwidth issues. In the last example, each replication all completing around 1/2 a second and indicate no problem.
If you do find that replication is taking longer than 5 seconds, you can test the network speed by transmitting some data between the console and the managed host. One way of testing speed is using the dd command and moving the data to /dev/null, so no hard disk space is used. An example of the command:
dd if=/dev/zero bs=1024 count=1048576 | ssh root@x,x,x,x 'cat > /dev/null'

This example transfers approximately 1GB of information over the network. Once completed, your output is similar to the following:

[root@HOSTNAME ~]# dd if=/dev/zero bs=1024 count=1048576 | ssh root@192.0.2.10 'cat > /dev/null'
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 6.6233 s, 162 MB/s
Note: If the transfer takes longer than 30 seconds to complete, enter Ctrl+c to end the transfer. It displays how much data transfers in that time and the average speed of the transfer.
This example shows that the transfer speed was 162 MB/s, which is sufficient. If your results are less than 10MB/s (roughly 100Mb/s after overhead), then this slower network link could impact the ability to stay in sync, causing replication or deployments to fail.

Resolving The Problem

If your network speed between the console and the managed hosts are consistently below the requirement of 100Mb/s, work with your network team on the possibility of increasing the speed to meet the requirements. Basic network troubleshooting needs done first to confirm there is not a configuration issue somewhere on the network potentially causing issues.

Here are some troubleshooting steps to help isolate the issue
  1. Check you network workflow. Refer to Technote 2001955 - QRadar: Basic network troubleshooting workflow.
  2. Run from an SSH session the command traceroute between manage hosts switches and routers to isolate latency issues. An example of traceroute:
    [root@HOSTNAME ~]#  traceroute 192.168.0.84
    traceroute to 192.168.0.84 (192.168.0.84), 30 hops max, 60 byte packets
    732APPhost (192.168.0.84)  0.809 ms  0.708 ms  0.638 ms
  3. Use tools such as ifconfig, ip ethtool or netstat. Refer to this article on how to use these tools:
    Technote 1997106 - QRadar: Using Linux Networking Tools to troubleshoot Interfaces
  4. Check all Switches and routers between QRadar console and managed hosts to ensure that they are working at network speeds compatible to QRadar.
  5. If you are doing intensive searches, managing large numbers of assets, log sources, or Offenses, the preferred network requirement is 1 gigabit per second.
Results
Isolation of network issues ensures all QRadar components replicate and perform optimally.

Document Location

Worldwide

[{"Business Unit":{"code":"BU008","label":"Security"},"Product":{"code":"SSBQAC","label":"IBM QRadar SIEM"},"Component":"Deployment","Platform":[{"code":"PF043","label":"Red Hat"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
26 January 2021

UID

ibm10957897