There are several possible symptoms that can point to a tunnel issue:
- Issuing a Deploy Changes or a Full Deploy from the Console can timeout on a Managed Host.
- A managed host will show in an Unknown status in the Console.
- Searches performed in the Console might fail with error " An IO error occurred on server(s) hostname. Please try again."
- One of the following errors can be seen in the qradar.log:
Setup process setuptunnel.host_114tunnelevent
stream has failed to start for 22 intervals. Continuing to try to start...
127.0.0.1 [ProcessMonitor] com.q1labs.hostcontext.processmonitor.ProcessManager: [ERROR] [NOT:0150114103][192.0.2.10/- -] [-/- -]Setup process setuptunnel.host_104tunnelrdate has failed to start for 276 intervals. Continuing to try to start..
[QRadar]  qflow0: [WARNING] Lost connection to 192.0.2.10:32010
Here are several possible causes for a down tunnel:
- SSH connectivity issues: Technote 10960870 - QRadar: Checking SSH connectivity to ensure a connection can be formed
- SSH connectivity validation: Technote 10960862 - QRadar: Validating SSH from the Console to a managed host is connecting
- Troubleshooting SSH connections: Technote 10960868 - QRadar: Troubleshooting SSH when connections cannot be established
- Bandwidth issues between the Console and Managed host that could cause the tunnel to time out or fail at times: Technote 10957897 - QRadar: Replication bandwidth requirements and verifying speed between console and managed host
- Version differences between the Console and the Managed Host: Technote 10960936 - QRadar: All hosts in your deployment must be at the same version
About encrypted connections 'tunnels' in QRadar
All the ports that are used by the QRadar Console to communicate with managed hosts can be encrypted using tunnels. Tunneled connections between the Console and managed hosts are done over SSH, using TCP port 22. QRadar allows administrators to use both encrypted and unencrypted connections for a managed host that is connected to the Console. The settings to encrypt communication between a Console and managed hosts are found on the Admin tab > System and License Management > Deployment Actions > Edit Managed Host > Encrypt Host Connections menu option. As managed hosts are added or edited in QRadar using the Deployment Options, Administrators can choose the option to encrypt the connection based on the location of the appliance.
For security reasons, you cannot set up an SSH tunnel from the managed host to the Console, but you can set up an SSH tunnel from the Console to the managed host. The managed host's public key is not added to the Console's authorized keys file. These SSH sessions are initiated from the Console to provide data to the managed host. For example, the QRadar Console can initiate multiple SSH sessions to the Event Processor Appliances for secure communication. This communication can include tunneled ports over SSH, such as HTTPS data for port 443 and Ariel query data for port 32006. QRadar QFlow Collectors that use encryption can initiate SSH sessions to Flow Processor appliances that require data.
Using Tunnels adds additional layers to QRadar and can impact performance. If you are on a closed network, tunnels may not be the best solution. To improve performance, you might need to also enable Encryption compression. If you require encryption and the tunnel fails to add, look at the suggestions below to determine if you see similar error messages or issues with SSH.
NOTE: Administrators are not able to SSH between managed hosts. SSH sessions must originate from the Console, or a root password is required when you SSH from the managed host to the Console. This is Intentional, and IP tables are configured in QRadar to prevent users from moving between managed hosts freely as part of our security protocols. One exception is that you can SSH from a QFlow to a Flow Processor. The flow will create the tunnel to a Flow Processor so that it can communicate with it.
Diagnosing The Problem
Other Troubleshooting steps for QRadar 7.3.x versions
email@example.com loaded failed failed QRadar Tunnel tunnel
Other troubleshooting from System and License Management verify that the host is not in an unknown state.
Check to make sure the Host is online or there is not a network issue.
Resolving The Problem
From the example above in diagnosing the issue, the tunnel failed was
firstname.lastname@example.org.Restart the tunnel service using the command:
systemctl restart email@example.com
- Should these steps not resolve your issue, collect logs from the Console and the managed host with a failed tunnel. See Technote 16266887 - Getting Help: What information should be submitted with a QRadar service request? on collecting logs.
- Open a case with IBM QRadar support
- For more information on troubleshooting SSH connections, see Technote 960602 - QRadar: Troubleshooting SSH connections and tunnels issues
08 January 2021