Troubleshooting
Problem
Symptom
Searching Log Activity in the Console UI fails with error "An IO error occurred on server(s) localhost:32008. Please try again."
The "localhost:32008" entry is mapped in the QRadar configuration files to a remote managed host ariel_proxy_server service.
Cause
The SSH protocol fails to connect to the managed host or the tunnel fails to start when the managed host connection is encrypted.
This document discusses the missing SSH ciphers issue on the Console or the managed host.
For additional information on these type of connectivity issues, see Technote #10960602 - QRadar: Troubleshooting SSH connections and tunnels issues.
Diagnosing The Problem
You can search the qradar.log for communication messages:
# less -i /var/logs/qradar.log | grep -i communication
An example output would look similar to this:
[ariel_proxy.ariel_proxy_server] [aqw_remote_15:subsearch1554634781] com.q1labs.ariel.searches.tasks.ArielQueryTaskBase: [ERROR] [NOT:0000003000][10.10.10.10/- -] [-
/- -]Can't communicate to server [localhost:32008] executing query:Id:subsearch1554634781, DB:<events@/store/ariel/events/records, /store/ariel/events/payloads>, Time:<19-04-07,16:27:00 to 19-04-07,16
:29:00>, last 120 secs, Criteria=(<DeviceId:pr=SensorDeviceName():ILike('.*.*health.*.*')> AND <Metric ID:pr=SingleArgScalarFunctionAdapter():[boolean(true)]>), MappingFactory=com.q1labs.core.types.ev
ent.mapping.NormalizedEventMappingFactory@4ee, prio=NORMAL
Apr 7 16:29:41 ::ffff:10.10.10.10 [ariel_proxy.ariel_proxy_server] [aqw_remote_15:subsearch1554634781] java.net.ConnectException: Connection refused
If the managed host connection is not encrypted, you can troubleshoot SSH connectivity issue from the technote mentioned in the Cause section. If the managed host connection is encrypted, you can verify the status of the tunnel services via command:
# systemctl list-units | grep -i tunnel
A sample output would look like this:
setuptunnel@host_106tunneleventstream.service loaded failed failed QRadar Setup Tunnel host_106tunneleventstream
setuptunnel@host_106tunnelflowstream.service loaded failed failed QRadar Setup Tunnel host_106tunnelflowstream
setuptunnel@host_106tunnelhttps.service loaded failed failed QRadar Setup Tunnel host_106tunnelhttps
setuptunnel@host_106tunnelsyslog.service loaded failed failed QRadar Setup Tunnel
Next, the reason why the tunnel service fails to start can be determined via the journalctl command:
# journalctl -xe | grep -i tunnel
A sample output will look like this:
tunnel_host_106tunnelhttps[433507]: Unable to negotiate with 10.10.10.10 port 22: no matching cipher found. Their offer: aes128-cbc,aes256-cbc
The cipher types available for use on both the Console and the managed host can be seen in the /etc/ssh/sshd_config file.
The cipher requested by the SSH session initiator (console in our case) can be seen via the SSH connection debug switches (where "10.10.10.10 is the managed host IP address):
# ssh -vvv 10.10.10.10
You can try to enforce the use of a certain cipher for SSH via this command:
# ssh -c aes256-ctr root@10.10.10.10
If the managed host is missing that cipher, you will see a similar output:
Unable to negotiate with 10.10.10.10 port 22: no matching cipher found. Their offer: aes128-cbc,aes256-cbc
That output points to a missing "aes256-ctr" cipher in the managed host SSH configuration file. You can compare the SSH configuration files via the cat command (you might have to use the IMM or DRAC session for the managed host):
# cat /etc/ssh/sshd_config | grep Ciphers
If there is a mismatch, see the steps in the "Resolving The Problem" section on how to resolve the issue.
Resolving The Problem
For example, if the "Ciphers" line in the Console /etc/ssh/sshd_config configuration file is:
# cat /etc/ssh/sshd_config | grep Ciphers
# Ciphers and keying
Ciphers aes128-ctr,aes192-ctr,aes256-ctr
And, if the "Ciphers" line in the Managed Host /etc/ssh/sshd_config configuration file is:
# cat /etc/ssh/sshd_config | grep Ciphers
# Ciphers and keying
Ciphers aes128-cbc,aes192-cbc,aes256-cbc
You will have to modify the Managed host /etc/ssh/sshd_config configuration file to match the Console via the vi command or your preferred file editor.
Note: Backup the /etc/ssh/sshd_config file prior to modification by typing:
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup
In our case, we changed the entry "aes256-cbc" on the managed host to "aes256-ctr" so it can match the console. The "Ciphers" line now will display:
# cat /etc/ssh/sshd_config | grep Ciphers
# Ciphers and keying
Ciphers aes128-ctr,aes192-ctr,aes256-ctr
Restart the sshd service on the managed host system:
# systemctl restart sshd
Results
The tunnel on the Console should restart automatically, and searches can be performed again.
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
31 March 2020
UID
ibm11033252