QRadar: Events fail to show in the Log Activity tab after pointing an Event Collector to a different Event Processor

Troubleshooting

Problem

You might find that after an Event Collector (EC) connection is modified to point to a different Event Processor (EP), the events from that EC stop showing in the Log Activity tab.

Symptom

No events are received in the Log Activity tab when a filter to show the events received from the Event Collector is used.

The following message can be seen in the /var/log/qradar.log file of the Event Collector:

[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] com.ibm.si.ec.destinations.StoreForwardDestination(ecs-ec/EC/TCP_TO_EP): [WARN] [NOT:0000004000] [-/- -]Unable to connect to server
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] java.net.ConnectException: Connection refused
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:731)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:123)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at com.q1labs.frameworks.nio.network.CommunicatorBase.connect(CommunicatorBase.java:211)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at com.q1labs.frameworks.nio.network.Communicator.connect(Communicator.java:276)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at com.q1labs.frameworks.nio.network.Communicator.<init>(Communicator.java:53)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at com.q1labs.frameworks.nio.network.Communicator.createClient(Communicator.java:95)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at com.q1labs.frameworks.nio.network.Communicator.createClient(Communicator.java:79)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at com.ibm.si.ec.destinations.StoreForwardDestination._connect(StoreForwardDestination.java:494)
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue]    at com.ibm.si.ec.destinations.StoreForwardDestination$TakeFromQueueJob.run_internal(StoreForwardDestination.java:192)

Cause

The connection between the EC and the new EP is getting dropped due to an internal network device, for example, an IPS or firewall.
The Remote Tunnel Initiation setting is not enabled for the EC. Some network devices allow only one-way communication. By default, the EC initiates the tunnel to the EP.

Environment

QRadar® environments that use encryption and running QRadar® 7.4 and later as version.

Diagnosing The Problem

Scenario #1 - Network infrastructure is blocking the connection:

Stop the iptables service temporarily in the EC and the EP.
```
# systemctl stop iptables
```

Attempt an SSH connection between the EP and the EC and vice-versa. The connection cannot be established.
Connection attempt from the EC to the EP

[root@qradar-ec01# ssh -v <IP of Event Processor>
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to <IP of Event Processor> [<IP of Event Processor>] port 22.
debug1: connect to address <IP of Event Processor> port 22: Connection timed out
ssh: connect to host <IP of Event Processor> port 22: Connection timed out

Connection attempt from the EP to the EC

[root@qradar-ep01# ssh -v <IP of Event Collector>
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to <IP of Event Collector> [<IP of Event Collector>] port 22.
debug1: connect to address <IP of Event Collector> port 22: Connection timed out
ssh: connect to host <IP of Event Collector> port 22: Connection timed out

Scenario #2 - Host is not configured for Remote Tunnel Initiation:

In the managed host originating the tunnel (by default the EC), the status is in SYN/SENT status:

# netstat -nap | grep 22| grep '<IP of Event Processor>'

tcp  0  1  <IP of Event Collector>:3954   <IP of Event Processor>:22    SYN_SENT    8397/ssh

The Remote Tunnel Initiation box is not checked.

Resolving The Problem

Scenario #1:

When this scenario occurs, the administrator must reach out to their Network team and request to allow the connection between the Event Collector and the Event Processor.

Scenario #2:

Edit the Event Collector host connection and enable the Remote Tunnel Initiation by following these steps:
1. Log in to the QRadar Console as an administrator.
2. Click the Admin tab.
3. In the System Configuration section, click System and License Management.
4. In the Display list, select Systems.
5. Select the Event Collector in the host table, and on the Deployment Actions menu, click Edit Host.
6. Select the Remote Tunnel Initiation and click Save.
From the Admin tab, click Deploy Changes.

Verify the connection between the ecs-ec and ecs-ep components in the file /var/log/qradar.log of the EC shows: TCP Destination [ecs-ec/EC/TCP_TO_EP] is alive.

[ecs-ec.ecs-ec] [ECS Runtime Thread] com.ibm.si.ec.destinations.StoreForwardDestination(ecs-ec/EC/TCP_TO_EP): [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]Compression initialized to: true
[ecs-ec.ecs-ec] [ECS Runtime Thread] com.q1labs.sem.monitors.PipelineStatusMonitor: [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]   EC-Queue TCP_TO_EP  registered.
[ecs-ec.ecs-ec] [ECS Runtime Thread] com.ibm.si.ec.destinations.StoreForwardDestination(ecs-ec/EC/TCP_TO_EP): [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]Parameters - server: localhost:32005, write timeout: 10000, retry attempts: 3
[ecs-ec.ecs-ec] [ECS Runtime Thread] com.q1labs.sem.monitors.TCPDestinationStatusMonitor: [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]TCP Destination [ecs-ec/EC/TCP_TO_EP] registered.
[ecs-ec.ecs-ec] [ECS Runtime Thread] com.q1labs.frameworks.queue.FileQueueInfo: [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]Read CfgFile ecs-ec_EC_TCP_TO_EP: head/tailFile 0/0, spillCount 0, head/TailOffset 0/0, clean true
[ecs-ec.ecs-ec] [ECS Runtime Thread] com.eventgnosis.ecs: [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]"qradar-ec01.test.local:ecs-ec/EC/TCP_TO_EP" THREAD started.
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] com.q1labs.frameworks.nio.network.protocol.ProtocolProcessor: [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]Protocol compression turned off. 24 threads, 48 tasks are used by communicator
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] com.q1labs.sem.monitors.TCPDestinationStatusMonitor: [INFO] [NOT:0000006000][<IP of Event Collector>/- -] [-/- -]TCP Destination [ecs-ec/EC/TCP_TO_EP] is alive.

Verify the events display in the Log Activity tab.

Related Information

QRadar: Events from Event Collectors are not displayed in the Log Activity due …

QRadar: Tunnel services in version 7.4.x

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"TS004821862","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.4.0;7.4.1;7.4.2;7.4.3;7.5.0"}]

Tips