IBM Support

QRadar: Troubleshooting tunnels and SSH issues in QRadar 7.2.5 and later

Troubleshooting


Problem

This article discusses encrypted host connections "tunnels" and how to troubleshooting SSH connections that can prevent the Console from creating a tunnel to a host and common troubleshooting tips.

Resolving The Problem


About encrypted connections 'tunnels' in QRadar


All the ports that are used by the QRadar Console to communicate with managed hosts can be encrypted using tunnels. Tunneled connections between the Console and managed hosts are done over SSH, using TCP port 22. QRadar allows administrators to have both encrypted and decrypted appliance that is connected to the Console in their deployment. The settings to encrypt communication between a Console and managed hosts are found on the Admin tab in the System and License Management screens. As managed hosts are added or edited in QRadar using the Deployment Options, Administrators can choose the option to encrypt the connection based on the location of the appliance.


For security reasons, you cannot set up an SSH tunnel from the managedhost to the Console, but you can set up an SSH tunnel from the Console to the managedhost. The managedhost's public key is not added to the Console's authorized keys file. These SSH sessions are initiated from the Console to provide data to the managedhost. For example, the QRadar Console can initiate multiple SSH sessions to the Event Processor Appliances for secure communication. This communication can include tunneled ports over SSH, such as HTTPS data for port 443 and Ariel query data for port 32006. QRadar QFlow Collectors that use encryption can initiate SSH sessions to Flow Processor appliances that require data.

Using Tunnels adds additional layers to QRadar and can impact performance. If you are on a closed
network Tunnels may not be the best solution. If you require them and they fail to add look at the suggestions below to determine if you see any of these conditions.

NOTE: Administrators are not able to SSH between managed hosts. SSH sessions must originate from the Console or a root password is required. This is intended and IP tables are configured in QRadar to prevent users moving between managed hosts freely as part of our security protocols. The exception is you should be able to SSH from a QFlow to a Flow Processor. The QFlow will create the tunnel to a Flow Processor so that it can communicate with it.

What error messages indicate a tunnel issue in QRadar?


Administrators can review the logs in /var/log/qradar.log for error messages that are similar to the error text below:

127.0.0.1 [ProcessMonitor] com.q1labs.hostcontext.processmonitor.ProcessManager: [ERROR] [NOT:0150114103][192.0.2.10/- -] [-/- -]Setup process setuptunnel.host_104tunnelrdate has failed to start for 276 intervals. Continuing to try to start..

QRadar-3100 [QRadar] [3330] qflow0: [WARNING] Lost connection to 192.0.2.10:32010



Validating SSH from the Console to a managed host is connecting


The following examples show what an Administrator would see when attempting to SSH or telnet to a remote host. Using SSH or telnet are good methods of validating that a tunneled connection is working as expected.

  1. First successful SSH connection
    In this example, you can see what a successful SSH connection would look like on your first attempt. You can see that you are prompted to accept the RSA key on the first connection.

    [root@Console-1]# ssh 192.0.2.11
    The authenticity of host '192.0.2.11 (192.0.2.11)' can't be established.
    RSA key fingerprint is bd:36:16:a8:00:2a:c9:56:6d:e2:26:eb:8d:66:3f:d5.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added '192.0.2.11' (RSA) to the list of known hosts.
    This server was upgraded to QRadar 7.2.6.20151107134559 on Thu Apr 14 21:42:58 EDT 2016.


     
  2. SSH banner when network is not blocked
    [root@Console-1 ~]# telnet 192.0.2.11 22
    Trying 192.0.2.11...
    Connected to 192.0.2.11.
    Escape character is '^]'.SSH-2.0-OpenSSH_5.3


Troubleshooting SSH connections when tunnels cannot be established


If you cannot SSH from the Console, the examples below will provide Administrators an overview of what can prevent a tunneled connection in QRadar with potential workarounds'.

  1. Managed Tunnel Key directory (Console only)
    If tunnels are being created properly between appliances, the Console should have one key for each encrypted host. Administrators can review their system to determine that each host has a key and that the ownership is as defined in the example.

    /store/configservices/staging/globalconfig/ssh_public_keys/
    [root@Console-1 ~]# ls -al /store/configservices/staging/globalconfig/ssh_public_keys/
    drwxr-xr-x 2 nobody nobody 62 May 2 20:19 .
    rwxrwxr-x 9 nobody nobody 24576 May 3 10:10 ..
    -rw-r--r-- 1 nobody nobody 406 May 2 18:25 Console_key
    -rw-r--r-- 1 nobody nobody 409 May 2 18:27 host_103_key
    rw-r--r-- 1 nobody nobody 409 May 2 20:19 host_104_key


     
  2. Review the permissions within the /root/.ssh directory (Console & managed hosts)
    Permissions on the .ssh directory should be 700 and the files within the directory should be 600. For Console appliances and managed hosts, the files to review are in the /root/.ssh directory. Administrators check and fix permissions on directory /root/.ssh/ and SSH files.

    [root@Console-1 .ssh]# ls -la .ssh/
    total 24
    drwx------ 2 root root 4096 May 2 18:35 .
    dr-xr-x---. 4 root root 4096 May 2 18:38 ..
    -rw------- 1 root nobody 426 May 2 18:35 authorized_keys
    -rw------- 1 root nobody 1675 May 2 18:25 id_rsa
    -rw------- 1 root nobody 406 May 2 18:25 id_rsa.pub
    -rw------- 1 root root 788 May 2 18:25 known_hosts


    Workaround
    If permissions are incorrect, then Administrators should update permissions as instructed below:

    [root@Console-1~]# chmod 700 /root/.ssh
    [root@Console-1~]# chmod 600 /root/.ssh/*


     
  3. Review the Console's public key file is present on the managedhost
    If the Console's id_rsa.pub is not in remote host's /root/.ssh/authorized_keys or there are no local public/private keys, then the SSH session will request a password. This password request due to a missing authorized key can prevent the tunnel from being created properly.

    [root@Console-1 ~]# ssh 192.0.2.11
    root@192.0.2.11's password:


    In 7.2.5 and later:
    - SSH public key login from managed hosts to the Console is no longer automatic.
    - SSH public key login from a QFlow to a Flow Processor is still automatic.

    Workaround
    Manually copy the Consoles id_rsa.pub contents to remote host and append it to the end of the file ~/ssh/authorized_keys
    Example: id_rsa.pub from the Console is in /root
    cat /root/id_rsa.pub >> /root/.ssh/authorized_keys

    Note: Do not copy the Console's id_rsa.pub to the /root/.ssh directory as the file must reside in the authorized_keys folder.

     
  4. Remote hosts SSH public key is wrong in local hosts /root/.ssh/known_hosts.
    If the public key is incorrect on a host, this could be the cause of why an SSH connection cannot be established. When a standard SSH session cannot be established, then attempting to add a managedhost with a tunnel would also fail to be added properly. The information below outlines how to review known SSH hosts issues.

    In the example below, attempting to open a standard SSH session from the Console to a managedhost cannot complete due to key not being found in the known_hosts list.


    [root@QRadar-3100 .ssh]# ssh 192.168.0.77
    Last login: Tue May 3 16:32:30 2016 from 192.168.0.75
    This server was upgraded to QRadar 7.2.6.20151107134559 on Thu Apr 7 16:05:15 EDT 2016
    with patch 7.2.6.20160405164932 applied on Mon Apr 11 14:00:17 EDT 2016

    [root@Qradar726-1201 ~]# ssh 192.168.0.76
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    bd:36:16:a8:00:2a:c9:56:6d:e2:26:eb:8d:66:3f:d5.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:2
    RSA host key for 192.168.0.76 has changed and you have requested strict checking.
    Host key verification failed.


    Explanation
    The error message 'Offending key in /root/.ssh/known_hosts: 2' indicates to the Administrator that line #2 in the known_hosts file is incorrect. Administrators can either compare the fingerprint.

    Workaround
    Administrators can use the command below and the a file editor to correct or remove line #2. After line #2 is removed or corrected, Administrators can attempt to connect by using SSH to get a prompt to update the known_hosts list or if the key was corrected, then the SSH session would be established without any prompts to the user.  Alternatively you can use the ssh-keygen -R command and remove the offending IP of the host.

    Options:
    1) Type vim ~/.ssh/known_hosts
        Type
    sed -i '`2d' ~/.ssh/known_hosts
        Where the number 2 is the offending line.
    2) Type ssh-keygen -R <IP of host>
     


     


Checking SSH connectivity to ensure a tunneled connection can be formed

The following error messages when attempting to create SSH connections indicate network issues, NIC configuration problems, firewall configuration issues, or hosts that are down within the network.

  1. SSH is not responding or packets dropped by network devices (firewalls)
    [root@QRadar-3100 ~]# telnet Qradar726-1201 22
    Trying 192.168.0.77...
    telnet: connect to address 192.168.0.77: Connection timed out.


    Explanation
    Possible symptom is problems with a NIC interface, switch port, or LAN cables. Check with your Administrator to verify these are working properly.

     
  2. SSH connection refused or being actively blocked by a firewall
    [root@QRadar-3100 ~]# telnet Qradar726-1201 22
    Trying 192.168.0.77...
    telnet: connect to address 192.168.0.77: Connection refused.


    Explanation
    Possible symptom is firewall is blocking port 22. Check with your firewall Administrator to verify port 22 is open.

     
  3. SSH issue 'No route to host' or 'Host not available'
    [root@QRadar-3100 ~]# telnet Qradar726-1201 22|
    Trying 192.168.0.77...
    telnet: connect to address 192.168.0.77: No route to host


    Explanation
    Possible symptom the host is down. Verify with the Data Center Administrator that the host is online.

     

[{"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Operating System","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.2","Edition":"Advanced","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
07 January 2021

UID

swg21981436