IBM Support

QRadar: Troubleshooting tunnel issues

Troubleshooting


Problem

This article discusses encrypted managed host connections "tunnels" and common troubleshooting tips.

Symptom

There are several possible symptoms that can point to a tunnel issue:

  1. Issuing a Deploy Changes or a Full Deploy from the Console can timeout on a Managed Host.
  2. A managed host will show in an Unknown status in the Console.
  3. Searches performed in the Console might fail with error " An IO error occurred on server(s) hostname. Please try again."
  4. One of the following errors can be seen in the qradar.log:

    Setup process setuptunnel.host_114tunneleventstream has failed to start for 22 intervals. Continuing to try to start...
    127.0.0.1 [ProcessMonitor] com.q1labs.hostcontext.processmonitor.ProcessManager: [ERROR] [NOT:0150114103][192.0.2.10/- -] [-/- -]Setup process setuptunnel.host_104tunnelrdate has failed to start for 276 intervals. Continuing to try to start..
    [QRadar] [3330] qflow0: [WARNING] Lost connection to 192.0.2.10:32010

Cause

Here are several possible causes for a down tunnel:

Environment

About encrypted connections 'tunnels' in QRadar


All the ports that are used by the QRadar Console to communicate with managed hosts can be encrypted using tunnels. Tunneled connections between the Console and managed hosts are done over SSH, using TCP port 22. QRadar allows administrators to use both encrypted and unencrypted connections for a managed host that is connected to the Console. The settings to encrypt communication between a Console and managed hosts are found on the Admin tab > System and License Management > Deployment Actions > Edit Managed Host > Encrypt Host Connections menu option. As managed hosts are added or edited in QRadar using the Deployment Options, Administrators can choose the option to encrypt the connection based on the location of the appliance.


image-20190911105727-2

For security reasons, you cannot set up an SSH tunnel from the managed host to the Console, but you can set up an SSH tunnel from the Console to the managed host. The managed host's public key is not added to the Console's authorized keys file. These SSH sessions are initiated from the Console to provide data to the managed host. For example, the QRadar Console can initiate multiple SSH sessions to the Event Processor Appliances for secure communication. This communication can include tunneled ports over SSH, such as HTTPS data for port 443 and Ariel query data for port 32006. QRadar QFlow Collectors that use encryption can initiate SSH sessions to Flow Processor appliances that require data.

Using Tunnels adds additional layers to QRadar and can impact performance. If you are on a closed network, tunnels may not be the best solution. To improve performance, you might need to also enable Encryption compression. If you require encryption and the tunnel fails to add, look at the suggestions below to determine if you see similar error messages or issues with SSH.

NOTE: Administrators are not able to SSH between managed hosts. SSH sessions must originate from the Console, or a root password is required when you SSH from the managed host to the Console. This is Intentional, and IP tables are configured in QRadar to prevent users from moving between managed hosts freely as part of our security protocols. One exception is that you can SSH from a QFlow to a Flow Processor. The flow will create the tunnel to a Flow Processor so that it can communicate with it.

Diagnosing The Problem

Other Troubleshooting steps for QRadar 7.3.x versions

To locate issues with these services type from the Console the command:
 
 systemctl list-units --type=service | grep tunnel
All services should be active and the output should be similar to:
 
image-20190805110518-2
Should a tunnel not be started it will be listed as failed.

tunnel@tunnel2.service                             loaded failed failed QRadar Tunnel tunnel

Other troubleshooting from System and License Management verify that the host is not in an unknown state.

image-20190805124011-1

Check to make sure the Host is online or there is not a network issue.

Resolving The Problem

Restarting tunnels

  1. From the example above in diagnosing the issue, the tunnel failed was tunnel@tunnel2.service. Restart the tunnel service using the command:

    systemctl restart tunnel@tunnel2.service

  2. Should these steps not resolve your issue, collect logs from the Console and the managed host with a failed tunnel. See Technote 16266887 - Getting Help: What information should be submitted with a QRadar service request?  on collecting logs.
     
  3. Open a case with IBM QRadar support
     
  4. For more information on troubleshooting SSH connections, see Technote 960602 - QRadar: Troubleshooting SSH connections and tunnels issues

Document Location

Worldwide

[{"Business Unit":{"code":"BU008","label":"Security"},"Product":{"code":"SSBQAC","label":"IBM QRadar SIEM"},"Component":"Encryption","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
08 January 2021

UID

ibm10959347