IBM Support

QRadar: Events from Event Collectors are not displayed in the Log Activity due to missing connection

Troubleshooting


Problem

Administrators might find that events received successfully by an Event Collector (EC) do not display in the Log Activity tab despite the host is reachable and when a Deploy Changes completes. If the Event Collector cannot open a server port to the Event Processor in the next stage of the event pipeline, events buffer on the Event Collector while it waits for a server port. If you do not see events that are received by the Event Collector when you search from the Console, you can confirm if the following error occurred: java.lang.RuntimeException: Server port is not specified.

Symptom

The following are common symptoms when the issue occurs:
  1. The events no longer display in the Log Activity tab for events received by an Event Collector.
  2. The Event Collector's persistent queue keeps growing as the received events are stored temporarily in there. This condition triggers a disk space alert for the Event Collector's /store partition when it passes the warning threshold.
    Note: When no action is done by the administrator, the /store partition grows up until it reaches the 95% and critical services are stopped. 
    # du -ch /store/persistent_queue/
325G    /store/persistent_queue/ecs-ec-ingress.ecs-ec-ingress
4.0K     /store/persistent_queue/ecs-ec.ecs-ec
  3. In /var/log/qradar.log on the Event Collector appliance, the following error is displayed: 
    [ecs-ec.ecs-ec] [ECS Runtime Thread] java.lang.RuntimeException: Error attempting to load {Event Collector Hostname}/
EC/TCP_TO_EP  Error : java.lang.RuntimeException: Server port is not specified
-- Output Snipped --
[ecs-ec.ecs-ec] [ECS Runtime Thread] Caused by: java.lang.RuntimeException: Server port is not specified
    Note: To quickly locate the error message, type: tail -f /var/log/qradar.log | grep 'TCP_TO_EP'

Cause

The Event Collector is missing the connection required to establish the connection between the ecs-ec service in the EC and the ecs-ep service in the Console or Event processor.

Environment

Event Collectors in a QRadar distributed deployment.

Diagnosing The Problem

The following steps guide administrators to determine when the processor appliance is not set as the destination connection.
  1. Log in to the QRadar® user interface as an administrator.
  2. Click the Admin tab.
  3. In the System Configuration section, click System and license Management.
  4. Expand the display list and select Systems.
  5. Select the Event Collector in the host table.
  6. Click the Deployment Actions menu, and click Edit Host.
  7. Expand the list menu and verify no host is set as destination.
    Note: When a host is set a destination connection, it has an asterisk (*) next to it.

    Figure01

Resolving The Problem

To resolve the problem, administrators must select the preferred processor appliance as the destination connection.
  1. Log in to the QRadar user interface as an administrator.
  2. Modify the target processor host in the Event Collector.
    1. Click the Admin tab.
    2. In the System Configuration section, click System and license Management.
    3. Expand the display list and select Systems.
    4. Select the Event Collector in the host table.
    5. Click the Deployment Actions menu, and click Edit Host.
    6. Select the target processor appliance that receives the events of the conflicting EC.
      Note: The target processor host can be the Console or an Event processor.
    7. Verify the target processor host now has the asterisk (*) next to it, which indicates the host is set as the destination connection.
      Note: In the following screen capture, the Console was set as the destination connection.

      Figure02
  3. Click the Admin tab and click Deploy Changes.
  4. Wait until the configuration changes process finishes.
  5. Verify the connection between the ecs-ec and ecs-ep components in the file /var/log/qradar.log of the EC shows: TCP Destination [ecs-ec/EC/TCP_TO_EP] is alive.
      
    [ecs-ec.ecs-ec] [ECS Runtime Thread] com.ibm.si.ec.destinations.StoreForwardDestination(ecs-ec/EC/TCP_TO_EP): [INFO] 
[<IP of Event Collector>/- -] [-/- -]Parameters - server: localhost:32005, write timeout: 10000, retry attempts: 3
[ecs-ec.ecs-ec] [ECS Runtime Thread] com.q1labs.sem.monitors.TCPDestinationStatusMonitor: [INFO] 
[<IP of Event Collector>/- -] [-/- -]TCP Destination [ecs-ec/EC/TCP_TO_EP] registered.
[ecs-ec.ecs-ec] [ECS Runtime Thread] com.eventgnosis.ecs: [INFO]
[<IP of Event Collector>/- -] [-/- -]"qradar-ec01.test.local:ecs-ec/EC/TCP_TO_EP" THREAD started.
[ecs-ec.ecs-ec] [ecs-ec/EC/TCP_TO_EP:TakeFromQueue] com.q1labs.sem.monitors.TCPDestinationStatusMonitor: [INFO]
[-/- -]TCP Destination [ecs-ec/EC/TCP_TO_EP] is alive.
Result
Administrators can now click the Log Activity and see the events received by the conflicting Event Collector displayed. If the events are not displayed, a restart of the ecs-ec-ingress and ecs-ec services in the Event Collector might be required. For more information about the impact of restarting these services, see this technical note. If the problem persists after the restarts, contact QRadar Support for assistance.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
30 June 2022

UID

ibm16598661

Page Feedback