IBM Support

QRadar: Troubleshooting your DLC - health metrics or other events not received in QRadar

Troubleshooting


Problem

This article helps you troubleshoot scenarios with missing events from DLCs.

Symptom

In this article we cover two main use cases, missing health metrics and missing all events. If you are seeing some incoming traffic, but not any traffic from DLCs, then the issue might be with the log source. If you are missing traffic from one or some DLCs, then you need to know how to pinpoint the issue - whether the issue is with the DLC, the network, or something else.
  • No health metrics (HM) events seen from a DLC.
    In this scenario we go through:
    1. How to look for HM events in QRadar.
    2. The three main configuration settings on the DLC.
    3. How to establish whether the HM events are being sent to QRadar.
    4. How to establish whether the HM events are being received in QRadar.
       
  • No events received in QRadar from a DLC.
    In this scenario we go through:
    1. How to look for events from a specific DLC in QRadar Log Activity.
    2. The main configuration settings on the DLC.
    3. Are events being sent to the DLC?
    4. How to establish whether any events from a specific DLC are being sent to QRadar.
    5. How to establish whether any events from a specific DLC are being received in QRadar.

Resolving The Problem

Follow the troubleshooting steps under the No health metrics events seen from a DLC or No events at all from DLC use case depending on which situation applies to you. If none of those steps resolve the issue, complete the steps in the Additional troubleshooting section.

Use case: No health metrics events seen from a DLC

Complete each step until you diagnose your issue:

  • 1. How to look for HM events in QRadar

    In Log Activity, you can use filters like:

    • Log Source [Indexed] Equals Log Source Filter: Type ibm dlc, then select from the Log Source list, and click Add Filter.
    • Event Name [Indexed] Equals any of, click Browse, QID/Name: Type dlc metrics, click Search. You get one to three alternatives. Select one event in the search window, go back to the previous window, and add the event name with the + button. Repeat for all the events you want to include. Click Add Filter.
    • Payload Contains is Value: Type DLCMetrics. Click Add Filter.
    With these types of filters, you can add Grouping by either Log Source or by Source IP, whichever is more useful. If you can't find the DLC HM events that you are looking for, proceed to check the configuration.

  • 2. The three main configuration settings on the DLC
    Are health metrics enabled on the DLC - and are the destination IP and destination port set correctly? 
    [root@CentOS7-4938 ~]# cat /opt/ibm/si/services/dlc/conf/config.json
{
        "Destination":{
               "destination.type":"UDP",
               "destination.ip":"10.10.218.220",
               "destination.port":"32500"
       },
       "TLS":{
               "tls.keystorefilepath":"/opt/ibm/si/services/dlc/keystore/",
               "tls.keystorepassword":"",
               "tls.keystoreexpirywindow":"14"
       },
       "EPS":5000,
       "DLCMetricsEventsEnabled":"false",
       "TOPIC":""
}
[root@CentOS7-4938 ~]#
    In this example it seems that DLCMetricsEventsEnabled are set to "false", which means that the health metrics are not enabled. Edit the file, save and exit, and restart the DLC service (systemctl restart dlc).

  • 3. Are health metrics messages being sent from the DLC?

    To verify, you can use tcpdump from the DLC CLI.
    Command: tcpdump -nnvvAs0 -i any dst <QRadarEventCollectorIP>
    Example Output (IP 10.10.218.229 is the DLC, 10.10.218.220 is the QRadar host):

    12:43:00.429790 IP (tos 0x0, ttl 64, id 18962, offset 0, flags [DF], proto UDP (17), length 489)
    10.10.218.229.55590 > 10.10.218.220.32500: [bad udp cksum 0xca16 -> 0x8d24!] UDP, length 461
E...J.@.@.&.    7..     7...&~..........D....b...................dlc-CentOS7-4938......N.)....R.....2.....7...1<134>1 2022-07-15T12:43:00.326+01:00 ]... DLC 1354 - - [NOT:0....6000][10.10.218.229/- -] [-....LEEF:1.0|IBM|DLC|1.7.2
..Metrics|src=?..6      InstanceID=4656a954-ac98-4997-a492-3897f8a0acf0 ComponentType=queues....Name=EventProcessingFilterQ*..  ....ID=SpillFilesCount      Value=1
n..................0..
.7....ecs-dlc_...TCP_TO_QRADAR..)....5."...]sourc*..S... Monitor...B.CRate..?0.0...P-4938
    If the connection to QRadar is encrypted, you can't see readable payloads. If you want to see the payloads, make sure that in file: /opt/ibm/si/services/dlc/conf/config.json
    This value exists:  
    "destination.type":"UDP",
    For more information about these commands, see the article on Changing the QRadar server destination port.

    If you are unable to see any outgoing health metrics events, you need to examine /var/log/dlc/dlc.error for any messages that might give a clue to the cause. You can try:
    • Restarting the rsyslog or syslog-ng service on the DLC host.
    • Check whether the network interface working.
    • Restarting the DLC service.
    • Rebooting the DLC host.
    • Check whether you have a supported version of Java™ (version 8 or later is required).
      How to check your Java version on your DLC host - look for the build version (in this case 8.0.7.5): 
      [root@CentOS7-4938 ~]# /opt/ibm/java-x86_64-80/jre/bin/java -version
java version "1.8.0_321"
Java(TM) SE Runtime Environment (build 8.0.7.5 - pxa6480sr7fp5-20220208_01(SR7 FP5))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20220104_19630 (JIT enabled, AOT enabled)
OpenJ9   - 2d4c7d9
OMR      - 59845b7
IBM      - 3c151c1)
JCL - 20220120_01 based on Oracle jdk8u321-b07
[root@CentOS7-4938 ~]# 
      If you do not have a supported version, update it. 
    • Check whether you have the latest version of the DLC RPM from IBM Fix Central.
      To check your installed version on your DLC host - the "current" directory is linked to the directory showing the dlc version: 
      [root@CentOS7-4938 ~]# ls -ld /opt/ibm/si/services/dlc/current
lrwxrwxrwx. 1 root root 30 Mar  9 15:32 /opt/ibm/si/services/dlc/current -> /opt/ibm/si/services/dlc/1.7.2
[root@CentOS7-4938 ~]#
      If you do not, update it.

  • 4. Are health metrics events received on the QRadar event collector?

    To verify, there are two alternative methods that you can use:

    1. Use tcpdump from the QRadar CLI.

      Command: tcpdump -nnvvAs0 -i any src <dlcIP>
      Example Output (IP 10.10.218.229 is the DLC, 10.10.218.220 is the QRadar host):

      13:52:01.811480 IP (tos 0x0, ttl 64, id 42880, offset 0, flags [DF], proto UDP (17), length 474)
    10.10.218.229.55590 > 10.10.218.220.32500: [udp sum ok] UDP, length 446
E.....@.@..b    7..     7...&~..........D....b...................dlc-CentOS7-4938.......))....R.....2.....7...1<134>1 2022-07-15T13:52:01.705+01:00 ]... DLC 1354 - - [NOT:0....6000][9.55.218.229/- -] [-....LEEF:1.0|IBM|DLC|1.7.2
..Metrics|src=?..6      InstanceID=4656a954-ac98-4997-a492-3897f8a0acf0 ComponentType=queues....Name=EventProcessingFilterQ*..  ....ID=SpillFilesCount      Value=1
n......5.0....ecs-dlc_...TCP_TO_QRADAR..)....5."...]sourc*..S... Monitor...B.CRate..?0.0...P-4938
    2. If the traffic is visible in a packet capture, investigate whether the health metrics events are visible in the Debug logging output. 
      To enable Debug logging on QRadar: 
      /opt/qradar/support/mod_log4j.pl -w yourName -al com.q1labs.semsources.sources.ibmqradardlc -duration 30m
      The Debug output is in /var/log/qradar.java.debug.
      To manually disable Debug logging: 
      /opt/qradar/support/mod_log4j.pl -w yourName -r
      Debug logging prints messages like this:
      Jul 19 15:09:00 ::ffff:10.10.218.220 [ecs-ec-ingress.ecs-ec-ingress] [IBMQRadarDLC Protocol Provider Thread: class com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider6] com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider: [DEBUG] PAYLOAD: <134>1 2022-07-19T15:09:00.464+01:00 CentOS7-4938 DLC 1357 - - [NOT:0000006000][10.10.218.229/- -] [-/- -]LEEF:1.0|IBM|DLC|1.7.2|DLCMetrics|src=10.10.218.229       InstanceID=4656a954-ac98-4997-a492-3897f8a0acf0 ComponentType=queues    ComponentName=ecs-dlc_dlc_TCP_TO_QRADAR MetricID=SpillFilesCount        Value=1
Jul 19 15:09:00 ::ffff:10.10.218.220 [ecs-ec-ingress.ecs-ec-ingress] [IBMQRadarDLC Protocol Provider Thread: class com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider6] com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider: [DEBUG] POSTED EVENT: Event: <?xml version="1.0" encoding="UTF-8"?><event xmlns:ev="http://www.eventgnosis.com/">
    <ev:host>127.0.0.1</ev:host>
    <ev:app>Syslog</ev:app>
    <ev:log>none</ev:log>
</event>
      These two lines tell us that one event was received, and the same event was ingested in the QRadar pipeline ("posted").

Use case: No events at all from DLC

The prerequisites for ingesting events in from a DLC in QRadar are:

  • A log source that acts as a listener on port 32500 on a specific QRadar host.
  • Knowledge of whether we are using protocol UDP or (TCP over) TLS to send the events with.

Complete each step until you diagnose your issue:

  • 1. How to look for events from a specific DLC in QRadar Log Activity

    This task is not entirely straightforward. For Syslog payloads it's not obvious which events are coming through a DLC, unless you are using TLS Syslog where the log source name sometimes has the DLC UUID as a suffix. (A UUID looks similar to 4656a954-ac98-4997-a492-3897f8a0acf0 and is found on the DLC with ls -l /opt/ibm/si/services/dlc/keystore look for a name of a subdirectory.)
    If you registered the DLC in the Log Source Management (LSM) app, you can manually associate Syslog log sources with individual DLCs. After manually associating the log sources, you can use the DLC filter in the LSM app.
    image-20220809152203-1
    If the log sources are auto-detected, they don't have any obvious indication that the events are coming from a DLC. The best method is to do a packet capture and correlate the Source IP and payload information if possible. See the Are any events from a specific DLC being sent to QRadar step.

  • 2. The main configuration settings on the DLC

    Is destination type (protocol), IP, and destination port correctly set in config.json?

    [root@CentOS7-4938 ~]# head /opt/ibm/si/services/dlc/conf/config.json
{

        "Destination":{
               "destination.type":"UDP",
               "destination.ip":"10.10.218.220",
               "destination.port":"32500"
       },
       "TLS":{
               "tls.keystorefilepath":"/opt/ibm/si/services/dlc/keystore/",
[root@CentOS7-4938 ~]#
    Make any needed corrections by manually editing the file, save the file and exit, and restart the DLC service (systemctl restart dlc).
  • 3. Are events being sent to the DLC?

    To verify, you can use tcpdump from the DLC CLI:
    tcpdump -nnvvAs0 -i any port 514 || port 1514
    Example output:

    <13>Jul 22 16:54:38 WIN2019SRV AgentDevice=WindowsLog      AgentLogFile=Security   PluginVersion=WC.MSEVEN6.10.0.2.62      Source=Microsoft-Windows-Security-Auditing      Computer=WIN2019SRVR.DOMAIN.com    OriginatingComputer=10.55.218.223        User=   Domain= EventID=4634    EventIDCode=4634EventType=8     EventCategory=12545     RecordNumber=2026492    TimeGenerated=1658505234        TimeWritten=1658505234  Level=LogAlways Keywords=AuditSu
16:54:39.072887 IP (tos 0x0, ttl 128, id 28563, offset 0, flags [none], proto TCP (6), length 576)
    10.55.218.223.53313 > 10.55.218.229.514: Flags [.], cksum 0xd452 (correct), seq 9112:9648, ack 1, win 8212, length 536
E..@o.......    7..     7...A...*J..<..P. ..R..ccess    Task=SE_ADT_LOGON_LOGOFF        Opcode=Info     Message=An account was logged off.  Subject:  Security ID:  NT AUTHORITY\SYSTEM  Account Name:  WIN2019SRV$  Account Domain:  DOMAIN  Logon ID:  0x9DEB90  Logon Type:   3  This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
<13>Jul 22 16:54:38 WIN2019SRV AgentDevice=WindowsLog      AgentLogFile=Security   PluginVersion=WC.MSEVEN6.10.0.2.62
16:54:39.072902 IP (tos 0x0, ttl 128, id 28564, offset 0, flags [none], proto TCP (6), length 576)
    10.55.218.223.53313 > 10.55.218.229.514: Flags [.], cksum 0xd956 (correct), seq 9648:10184, ack 1, win 8212, length 536
E..@o.......    7..     7...A...*M..<..P. ..V..Source=Microsoft-Windows-Security-Auditing       Computer=WIN2019SRVR.DOMAIN.com    OriginatingComputer=10.55.218.223        User=   Domain= EventID=4672    EventIDCode=4672        EventType=8     EventCategory=12548     RecordNumber=2026493    TimeGenerated=1658505245        TimeWritten=1658505245  Level=LogAlways Keywords=AuditSuccess   Task=SE_ADT_LOGON_SPECIALLOGON  Opcode=Info     Message=Special privileges assigned to new logon.  Subject:  Security ID:  NT AUTHORITY\SYSTEM  Account Name:  WIN2019SRV$  Account Domain:  DOMAIN  Logon ID:  0x9DF9A7  Privileges:  SeSecurit
    If you can't see any incoming traffic, check:
    • Are your ports in the DLC accessible? If not, open them.
    • Is your DLC host accessible from the sending end? You need to check with your network admins whether there are missing routing entries or firewall rules.
  • 4. Are any events from a specific DLC being sent to QRadar?

    To verify, you can use tcpdump from the DLC CLI:
    tcpdump -nnvvAs0 -i any dst <QRadarEventCollectorIP>
    Example Output (IP 10.10.218.229 is the DLC, 10.10.218.220 is the QRadar host):

    12:43:00.429790 IP (tos 0x0, ttl 64, id 18962, offset 0, flags [DF], proto UDP (17), length 489)
    10.10.218.229.55590 > 10.10.218.220.32500: [bad udp cksum 0xca16 -> 0x8d24!] UDP, length 461
E...J.@.@.&.    7..     7...&~..........D....b...................dlc-CentOS7-4938......N.)....R.....2.....7...1<134>1 2022-07-15T12:43:00.326+01:00 ]... DLC 1354 - - [NOT:0....6000][10.10.218.229/- -] [-....LEEF:1.0|IBM|DLC|1.7.2
..Metrics|src=?..6      InstanceID=4656a954-ac98-4997-a492-3897f8a0acf0 ComponentType=queues....Name=EventProcessingFilterQ*..  ....ID=SpillFilesCount      Value=1
n..................0..
.7....ecs-dlc_...TCP_TO_QRADAR..)....5."...]sourc*..S... Monitor...B.CRate..?0.0...P-4938
    If the connection to QRadar is encrypted, you can't see readable payloads. If you want to see the payloads, make sure that in file: /opt/ibm/si/services/dlc/conf/config.json
    This value exists:  
    "destination.type":"UDP",
    For more information about these commands, see the article on Changing the QRadar server destination port.

    If you are unable to see any outgoing events, you need to examine /var/log/dlc/dlc.error for any messages that might give a clue to the cause. You can try:
    • Restarting the rsyslog or syslog-ng service on the DLC host.
    • Checking whether the network interface working.
    • Restarting the DLC service.
    • Rebooting the DLC host.

  • 5. How to establish whether any events from a specific DLC are being received in QRadar

    To verify, there are two alternative methods that you can use:

    1. Use tcpdump from the QRadar CLI.

      Command: tcpdump -nnvvAs0 -i any src <dlcIP>
      Example Output (IP 10.10.218.229 is the DLC, 10.10.218.220 is the QRadar host):

      13:52:01.811480 IP (tos 0x0, ttl 64, id 42880, offset 0, flags [DF], proto UDP (17), length 474)
    10.10.218.229.55590 > 10.10.218.220.32500: [udp sum ok] UDP, length 446
E.....@.@..b    7..     7...&~..........D....b...................dlc-CentOS7-4938.......))....R.....2.....7...1<134>1 2022-07-15T13:52:01.705+01:00 ]... DLC 1354 - - [NOT:0....6000][9.55.218.229/- -] [-....LEEF:1.0|IBM|DLC|1.7.2
..Metrics|src=?..6      InstanceID=4656a954-ac98-4997-a492-3897f8a0acf0 ComponentType=queues....Name=EventProcessingFilterQ*..  ....ID=SpillFilesCount      Value=1
n......5.0....ecs-dlc_...TCP_TO_QRADAR..)....5."...]sourc*..S... Monitor...B.CRate..?0.0...P-4938
    2. If the traffic is visible in a packet capture, investigate whether any events are visible in the Debug logging output. 
      To enable Debug logging on QRadar: 
      /opt/qradar/support/mod_log4j.pl -w yourName -al com.q1labs.semsources.sources.ibmqradardlc -duration 30m
      The Debug output is in /var/log/qradar.java.debug.
      To manually disable Debug logging: 
      /opt/qradar/support/mod_log4j.pl -w yourName -r
      Debug logging prints messages like this:
      Jul 19 15:09:00 ::ffff:10.10.218.220 [ecs-ec-ingress.ecs-ec-ingress] [IBMQRadarDLC Protocol Provider Thread: class com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider6] com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider: [DEBUG] PAYLOAD: <134>1 2022-07-19T15:09:00.464+01:00 CentOS7-4938 DLC 1357 - - [NOT:0000006000][10.10.218.229/- -] [-/- -]LEEF:1.0|IBM|DLC|1.7.2|DLCMetrics|src=10.10.218.229       InstanceID=4656a954-ac98-4997-a492-3897f8a0acf0 ComponentType=queues    ComponentName=ecs-dlc_dlc_TCP_TO_QRADAR MetricID=SpillFilesCount        Value=1
Jul 19 15:09:00 ::ffff:10.10.218.220 [ecs-ec-ingress.ecs-ec-ingress] [IBMQRadarDLC Protocol Provider Thread: class com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider6] com.q1labs.semsources.sources.ibmqradardlc.QRadarDLCProvider: [DEBUG] POSTED EVENT: Event: <?xml version="1.0" encoding="UTF-8"?><event xmlns:ev="http://www.eventgnosis.com/">
    <ev:host>127.0.0.1</ev:host>
    <ev:app>Syslog</ev:app>
    <ev:log>none</ev:log>
</event>
      These two lines tell us that one event was received, and the same event was ingested in the QRadar pipeline ("posted").

Extra troubleshooting

If you're still experiencing issues, try these suggestions.

  • Events are received in QRadar from the DLC, but no events are shown in Log Activity
    If events are received in QRadar, you need to examine the logs for any error messages that might give a clue about the cause.
    • DLC: /var/log/dlc/dlc.error
    Example error message in the dlc.error log:
    2022-06-29 10:57:20,647 [DLC Sec Event Forward Thread] com.ibm.si.frameworks.nio.network.TLSSocketConnector: [ERROR] [NOT:0000003000][127.0.0.1/- -] [-/- -][ERROR_COULD_NOT_CONNECT:58002] Connection Error. Cannot connect qradarconsole.<mydomain>:32500.
    java.net.SocketException: Network is unreachable (connect failed)
    This error message means that no route for the destination network could be found in the routing table on the reporting router. This message is commonly seen when a user tries to connect to a private address that is nonroutable across the Internet. 
    • QRadar: /var/log/qradar.error
    Example error messages in the QRadar log:
    :ffff:10.168.60.180 [ecs-ec-ingress.ecs-ec-ingress] [Thread-99] com.q1labs.semsources.sources.ibmqradardlc.listener.TLSObjectListener: [INFO] [NOT:0080004100][10.10.180.30/- -] [-/- -]IBMQRadarDLC  refused connection from [/10.10.102.146:54862] - Connection count: 
    :ffff:10.168.60.180 [ecs-ec-ingress.ecs-ec-ingress] [Thread-99] com.q1labs.semsources.sources.ibmqradardlc.listener.TLSObjectListener: [INFO] [NOT:0080004100][10.10.180.30/- -] [-/- -]IBMQRadarDLC  refused connection from [/10.10.102.146:54898] - Connection count:
    This error message implies that the number of sockets available for DLCs to connect to, are all used up. If you need to see the state of the port, whether there are established sessions or whether the port is listening, you can monitor port 32500 (default) on the QRadar host with netstat or ss: 
    [root@qr743 ~]# ss -nap | grep -w 32500
udp    UNCONN     0      0      [::]:32500              [::]:*                   users:(("java",pid=1867,fd=521))

[root@qr743 ~]# ss -unapt | grep 10.10.180.30
tcp    ESTAB      0      0      10.168.60.189:32500              10.10.180.30:51870               users:(("java",pid=62860,fd=590))
tcp    ESTAB      0      0      10.168.60.189:32500              10.10.180.30:59216               users:(("java",pid=62860,fd=687))
    If you want to see the number of established connections on the QRadar host, you can use:
    [root@qr743 ~]# ss -nap | grep -w 32500 | wc -l
51

    The example output implies that the max number of connections are used. Check whether:

    • More than 50 DLCs are trying to connect simultaneously to one port.
    • A firewall is interfering with the connection, causing the DLC to create new connections that use up all the sockets.
    • The connections are not terminating timely when idle. You can try to adjust the setting in Admin> System Settings> Advanced>  Timeout for Idle TCP Syslog Connections (seconds) to a smaller value than the default 900.
    • Note: Although it might resolve an issue temporarily, an ecs-ec-ingress service restart on the QRadar host might obscure the cause of an issue as the restart severs all connections. If you still opt to try a service restart, collect this information before you restart the service, for a possible Support case:
      1. List of JAR files that Ingress are loaded: lsof -F n -p $(systemctl status ecs-ec-ingress| grep "Main PID" | awk '{print $3}') | grep ^n/ | cut -c2- | sort -u | grep jar > /store/LOGS/ingress_restart_loaded_jars.log
      2. Status of Ingress: systemctl status ecs-ec-ingress -l > /store/LOGS/ecs-ec-ingress_status.log
      3. Thread dump of Ingress: /opt/qradar/support/threadTop.sh -p 7787 --full > /store/LOGS/ingress_threaddump.txt
  • How to replay events on a DLC

    If you want to replay events through a DLC, follow this process:
    Extract a few events from the DLC host into a text file: 

    
tail -n 200 /var/log/messages > /root/LinuxOSDLCTest.syslog
    Replay the file with the events with ncat (my DLC IP address is 192.168.3.115). Note the ingress port on a DLC is 1514: 
    ncat 192.168.3.115 1514 < /root/LinuxOSDLCTest.syslog

Related Information

Disconnected Log Collector

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt9AAA","label":"DLC"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
09 August 2022

UID

ibm16604017

Page Feedback