Troubleshooting
Problem
Symptom
There are several possible symptoms that can point to a tunnel issue:
- Issuing a Deploy Changes or a Full Deploy from the Console can time out on a Managed Host.
- A managed host shows in an Unknown status in the Console.
- Searches performed in the Console might fail with error " An IO error occurred on server(s) hostname. Please try again."
- One of the following errors can be seen in the qradar.log:
Setup process setuptunnel.host_114tunneleventstream has failed to start for 22 intervals. Continuing to try to start... 127.0.0.1 [ProcessMonitor] com.q1labs.hostcontext.processmonitor.ProcessManager: [ERROR] [NOT:0150114103][192.0.2.10/- -] [-/- -]Setup process setuptunnel.host_104tunnelrdate has failed to start for 276 intervals. Continuing to try to start..
[QRadar] [3330] qflow0: [WARNING] Lost connection to 192.0.2.10:32010
Cause
Here are several possible causes for a down tunnel:
- SSH connectivity issues: QRadar: Checking SSH connectivity to ensure a connection can be formed
- SSH connectivity validation: QRadar: Validating SSH from the Console to a managed host is connecting
- Troubleshooting SSH connections: QRadar: Troubleshooting SSH when connections cannot be established
- Bandwidth issues between the Console and Managed host that could cause the tunnel to time out or fail at times: QRadar: Replication bandwidth requirements and verifying speed between console and managed host
- Version differences between the Console and the Managed Host: QRadar: All hosts in your deployment must be at the same version
Environment
About encrypted connections 'tunnels' in QRadar
All the ports that are used by the QRadar Console to communicate with managed hosts can be encrypted in tunnels. Tunneled connections between the Console and managed hosts are done over SSH, on TCP port 22. QRadar allows administrators to use both encrypted and unencrypted connections for a managed host that is connected to the Console. The settings to encrypt communication between a Console and managed hosts are found on the Admin tab > System and License Management > Deployment Actions > Edit Managed Host > Encrypt Host Connections menu option. As managed hosts are added or edited in QRadar in the Deployment Options, Administrators can choose the option to encrypt the connection based on the location of the appliance.
For security reasons, you cannot set up an SSH tunnel from the managed host to the Console, but you can set up an SSH tunnel from the Console to the managed host. The managed host's public key is not added to the Console's authorized keys file. These SSH sessions are initiated from the Console to provide data to the managed host. For example, the QRadar Console can initiate multiple SSH sessions to the Event Processor Appliances for secure communication. This communication can include tunneled ports over SSH, such as HTTPS data for port 443 and Ariel query data for port 32006. QRadar QFlow Collectors that use encryption can initiate SSH sessions to Flow Processor appliances that require data.
Using Tunnels adds extra layers to QRadar and can impact performance. If you are on a closed network, tunnels might not be the best solution. To improve performance, you might need to also enable Encryption compression. If you require encryption and the tunnel fails to add, look at the following suggestions to determine whether you see similar error messages or issues with SSH.
NOTE: Administrators are not able to SSH between managed hosts. SSH sessions must originate from the Console, or a root password is required when you SSH from the managed host to the Console. SSH from the Console and not a managed host is intentional, and IP tables are configured in QRadar to prevent users from moving between managed hosts freely as part of our security protocols. One exception is that you can SSH from a QFlow to a Flow Processor. The flow creates the tunnel to a Flow Processor so that it can communicate with it.
Diagnosing The Problem
Other Troubleshooting steps for QRadar 7.3.x versions
systemctl list-units --type=service | grep tunnel
tunnel@tunnel2.service loaded failed failed QRadar Tunnel tunnel
Other troubleshooting from System and License Management verifies that the host is not in an unknown state.
Check to make sure that the Host is online or there is not a network issue.
Resolving The Problem
Restarting tunnels
-
From the example used in diagnosing the issue, the tunnel failed was
tunnel@tunnel2.service.Restart the tunnel service by using the command:
systemctl restart tunnel@tunnel2.service - If not resolved, collect logs from the Console and the managed host with a failed tunnel. To collect logs, see Getting Help: What information should be submitted with a QRadar service request?
- Open a case with IBM QRadar support
- For more information on troubleshooting SSH connections, see QRadar: Troubleshooting SSH connections and tunnels issues
Related Information
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
25 October 2023
UID
ibm10959347