IBM Support

QRadar: SSH connection or tunnel fails due to SSH cipher mismatch

Troubleshooting


Problem

SSH missing cipher causes the SSH connection or tunnel to fail

Symptom

Searching Log Activity in the Console UI fails with error "An IO error occurred on server(s) localhost:32008. Please try again."

The "localhost:32008" entry is mapped in the QRadar configuration files to a remote managed host ariel_proxy_server service.

Cause

The SSH protocol fails to connect to the managed host or the tunnel fails to start when the managed host connection is encrypted. 

This document discusses the missing SSH ciphers issue on the Console or the managed host.

For additional information on these type of connectivity issues, see Technote #10960602 - QRadar: Troubleshooting SSH connections and tunnels issues.

Diagnosing The Problem

You can search the qradar.log for communication messages:

# less -i /var/logs/qradar.log | grep -i communication

An example output would look similar to this:

[ariel_proxy.ariel_proxy_server] [aqw_remote_15:subsearch1554634781] com.q1labs.ariel.searches.tasks.ArielQueryTaskBase: [ERROR] [NOT:0000003000][10.10.10.10/- -] [-
/- -]Can't communicate to server [localhost:32008] executing query:Id:subsearch1554634781, DB:<events@/store/ariel/events/records, /store/ariel/events/payloads>, Time:<19-04-07,16:27:00 to 19-04-07,16
:29:00>, last 120 secs, Criteria=(<DeviceId:pr=SensorDeviceName():ILike('.*.*health.*.*')> AND <Metric ID:pr=SingleArgScalarFunctionAdapter():[boolean(true)]>), MappingFactory=com.q1labs.core.types.ev
ent.mapping.NormalizedEventMappingFactory@4ee, prio=NORMAL
Apr  7 16:29:41 ::ffff:10.10.10.10 [ariel_proxy.ariel_proxy_server] [aqw_remote_15:subsearch1554634781] java.net.ConnectException: Connection refused

If the managed host connection is not encrypted, you can troubleshoot SSH connectivity issue from the technote mentioned in the Cause section. If the managed host connection is encrypted, you can verify the status of the tunnel services via command:

# systemctl list-units | grep -i tunnel

A sample output would look like this:

setuptunnel@host_106tunneleventstream.service                                                    loaded failed failed    QRadar Setup Tunnel host_106tunneleventstream
setuptunnel@host_106tunnelflowstream.service                                                     loaded failed failed    QRadar Setup Tunnel host_106tunnelflowstream
setuptunnel@host_106tunnelhttps.service                                                          loaded failed failed    QRadar Setup Tunnel host_106tunnelhttps
setuptunnel@host_106tunnelsyslog.service                                                         loaded failed failed    QRadar Setup Tunnel 

Next, the reason why the tunnel service fails to start can be determined via the journalctl command:

# journalctl -xe | grep -i tunnel

A sample output will look like this:

tunnel_host_106tunnelhttps[433507]: Unable to negotiate with 10.10.10.10 port 22: no matching cipher found. Their offer: aes128-cbc,aes256-cbc

The cipher types available for use on both the Console and the managed host can be seen in the /etc/ssh/sshd_config file.

The cipher requested by the SSH session initiator (console in our case) can be seen via the SSH connection debug switches (where "10.10.10.10 is the managed host IP address):

# ssh -vvv 10.10.10.10

You can try to enforce the use of a certain cipher for SSH via this command:

# ssh -c aes256-ctr root@10.10.10.10

If the managed host is missing that cipher, you will see a similar output:

Unable to negotiate with 10.10.10.10 port 22: no matching cipher found. Their offer: aes128-cbc,aes256-cbc

That output points to a missing "aes256-ctr" cipher in the managed host SSH configuration file. You can compare the SSH configuration files via the cat command (you might have to use the IMM or DRAC session for the managed host):

# cat /etc/ssh/sshd_config | grep Ciphers

If there is a mismatch, see the steps in the "Resolving The Problem" section on how to resolve the issue.

Resolving The Problem

For example, if the "Ciphers" line in the Console /etc/ssh/sshd_config configuration file is:

# cat /etc/ssh/sshd_config | grep Ciphers
# Ciphers and keying
Ciphers aes128-ctr,aes192-ctr,aes256-ctr

And, if the "Ciphers" line in the Managed Host /etc/ssh/sshd_config configuration file is:

# cat /etc/ssh/sshd_config | grep Ciphers
# Ciphers and keying
Ciphers aes128-cbc,aes192-cbc,aes256-cbc

You will have to modify the Managed host /etc/ssh/sshd_config configuration file to match the Console via the vi command or your preferred file editor.

Note: Backup the /etc/ssh/sshd_config file prior to modification by typing:

# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup

In our case, we changed the entry "aes256-cbc"  on the managed host to "aes256-ctr" so it can match the console. The "Ciphers" line now will display:

# cat /etc/ssh/sshd_config | grep Ciphers
# Ciphers and keying
Ciphers aes128-ctr,aes192-ctr,aes256-ctr

Restart the sshd service on the managed host system:

# systemctl restart sshd

Results
The tunnel on the Console should restart automatically, and searches can be performed again.

Document Location

Worldwide

[{"Business Unit":{"code":"BU008","label":"Security"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"Component":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]

Document Information

Modified date:
31 March 2020

UID

ibm11033252