Troubleshooting
Problem
Different problems can occur after a Data Collector (DC) is installed and connected to the Log Insights server. This document covers some of the different problems that can occur along with how to fix those problems.
Diagnosing The Problem
A. Check the DC server to confirm whether the DC is running.
The status command is
systemctl status dlc
If the service is not running, start the service with command
systemctl start dlc
B. Check the configuration files:
1. Check the contents of /opt/ibm/si/services/dlc/conf/config.json
These are the items to look for in the config.json file:
1. Check the contents of /opt/ibm/si/services/dlc/conf/config.json
These are the items to look for in the config.json file:
{
"Destination": {
"destination.type": "KAFKA", ← The destination.type must be KAFKA. If this is set to UDP or TLS, then the DC is configured to send to QRadar. Run the startRegistration.sh script to fix the problem
"destination.ip": "localhost",
"destination.port": "32500"
},
"TLS": {
"tls.keystorefilepath": "/opt/ibm/si/services/dlc/keystore/",
"tls.keystorepassword": "",
"tls.keystoreexpirywindow": "14"
},
"EPS": 20000,
"DLCMetricsEventsEnabled": "false",
"TOPIC": "dlc-event-topic-pipeline-<pipelineid>", ← The TOPIC must be set to dlc-event-topic-<pipeline-pipelineid>. If this is missing, make sure you registered your DC and processed the connection bundle
"ManagementServiceConsumerTOPIC": "",
"ManagementServiceProducerTOPIC": ""
}
2. Check the contents of /opt/ibm/si/services/dlc/conf/cp4s_kafka_topics.yaml and look for these items:
---
topic:
- name: "dlc-event-topic-pipeline-<pipelineid>"
alias: null
producer:
client.id: "dlcClient1-dlc-,<UUID>"
bootstrap.servers: "datalake-kafka-cluster-kafka-bootstrap-<Log Insights server name>.ibm.com:443" ← bootstrap.server must include a value
key.serializer: "org.apache.kafka.common.serialization.LongSerializer"
value.serializer: "com.q1labs.sem.kafka.SIEventKafkaSerializer"
max.block.ms: "36000000"
delivery.timeout.ms: "36000000"
request.timeout.ms: "3000"
buffer.memory: "4194304"
max.request.size: "1000000"
acks: "all"
batch.size: "1000000"
linger.ms: "100"
compression.type: "gzip"
enable.idempotence: "true"
max.in.flight.requests.per.connection: "1"
security.protocol: "SSL"
ssl.trustmanager.algorithm: "PKIX"
ssl.truststore.location: "/opt/ibm/si/services/dlc/conf/registration/<UUID>/root.crt" ← truststore.location must point to a certificate (Re-run the GenerateKafkaFiles.sh script with connection bundle manually to fix the issue.)
ssl.truststore.type: "PEM"
ssl.key.password: "<Encrypted Password>" ← key.password must be set. This is done by the startRegistration.sh script
ssl.keystore.location: "/opt/ibm/si/services/dlc/conf/registration/<UUID>/dlc-CP4S-client.crt" ← (Re-run the GenerateKafkaFiles.sh script with connection bundle manually to fix the issue.)
ssl.keystore.type: "PEM"
ssl.enabled.protocols: "TLSv1.3,TLSv1.2"
C. Check the following MBean on the DC machine:
/opt/ibm/si/services/dlc/current/script/jmx.sh -p 7787 -b "com.q1labs.sem:application=dlc.dlc,type=destinations,name=CP4SForwardDestination"
The response includes:
com.q1labs.sem:application=dlc.dlc,type=destinations,name=CP4SForwardDestination
--------------------------------------------------------------------------------
KafkaTopic: dlc-event-topic-pipeline-<pipelineid>
KafkaCompressionType: gzip
Connected: true ← The Most important value to check if you are connected.
EPS: 0
TotalWrittenBytes: 0
DiscardEventCount: 0
EventsSeen: 99 ← The Number of events sent to Kafka
EPSThreshold: 20000
Compressed: true
RawEventCount: 0
D. For general connection troubleshooting, run the following script:
/opt/ibm/si/services/dlc/current/script/checkKafka.sh
If events are to Kafka on the Log Insights server, the response includes:
kafka.producer:type=producer-topic-metrics,client-id=dlcClient1-dlc-<UUID>,topic=dlc-event-topic-pipeline-e0881bd0
------------------------------------------------------------------------------------------------------------------------------------------------
record-retry-rate: 0.0
record-send-rate: 0.0
record-retry-total: 0.0
compression-rate: NaN
record-error-total: 0.0
byte-rate: 0.0
record-error-rate: 0.0
byte-total: 3914.0
record-send-total: 99.0 ← This value confirms DC has sent the number of records reported to Kafka
If nothing was sent to Kafka, this is the response seen:
No matching mbean
E. In the data returned from the command in step B, if connected is false and no Events seen, then do the following:
1. Check /var/log/dlc/dlc.error for message
1. Check /var/log/dlc/dlc.error for message
2. Check /var/log/dlc/kafka.error for message
a. A handshake failure typically means that the DC certificates are not correct. To fix the problem, redo the registration process manually.
b. If the following is in the /var/log/dlc/dlc.error:
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Failed to load PEM SSL keystore /opt/ibm/si/services/dlc/conf/registration/<UUID>/dlc-CP4S-client.crt
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configs
Caused by: java.security.spec.InvalidKeySpecException: Cannot retrieve the PKCS8EncodedKeySpec
This indicates the password stored in the cp4s_kafka_topics.yaml does not match the password used on the dlc-CP4S-client.key file. To fix this issue:
1. Edit the /opt/ibm/si/services/dlc/conf/cp4s_kafka_topics.yaml file and remove the encrypted password strings in ssl.key.password (all instances of this parameter).
2. Remove the file in /opt/ibm/si/services/dlc/conf/registration/<UUID>/
3. Redo the registration process manually
1. Edit the /opt/ibm/si/services/dlc/conf/cp4s_kafka_topics.yaml file and remove the encrypted password strings in ssl.key.password (all instances of this parameter).
2. Remove the file in /opt/ibm/si/services/dlc/conf/registration/<UUID>/
3. Redo the registration process manually
F. To check whether events are received by the DC, look at this section.
1. If collecting syslog data by UDP, check the following:
/opt/ibm/si/services/dlc/current/script/jmx.sh -p 7787 -b "com.q1labs.sem:application=dlc.dlc,type=sources,name=Syslog Source"
This returns data:
com.q1labs.sem:application=dlc.dlc,type=sources,name=Syslog Source
------------------------------------------------------------------
TotalDropped: 0
DroppedInLastInterval: 0
LargestQueueSizeSeen: 0
RunningTime: 247
RemainingCapacity: 100000
CurrentRate: 0.0
Posted: 2 ← events received by udp syslog
EventQueueSize: 0
EventsDroppedOnStartup: 0
2. If collecting syslog data by TCP, check the following:
/opt/ibm/si/services/dlc/current/script/jmx.sh -p 7787 -b "com.q1labs.sem:application=dlc.dlc,type=sources,name=TcpSyslogSource(0.0.0.0/1514) Source"
This returns data:
com.q1labs.sem:application=dlc.dlc,type=sources,name=TcpSyslogSource(0.0.0.0/1514) Source
-----------------------------------------------------------------------------------------
EventsRetrieved: 1
EventsPostedPerSecondSinceStartup: 1.235738E-4
NumberOfConnections: 1
TotalDropped: 0
DroppedInLastInterval: 0
CurrentRate: 0.0
Posted: 1 ← events received by tcp syslog
StartupTime: 18 Oct 2022 14:11:30
EventsDroppedOnStartup: 0
3. If no events are received, check the firewall settings on the DC by using the command:
firewall-cmd --list-all
Something like this will show (514 open and forwarded to 1514):
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: cockpit dhcpv6-client ssh
ports: 514/udp 514/tcp
protocols:
forward: no
masquerade: no
forward-ports:
port=514:proto=tcp:toport=1514:toaddr=
port=514:proto=udp:toport=1514:toaddr=
source-ports:
icmp-blocks:
rich rules:
G. To confirm if the events are on the Log Insights server:
- Log in to the Log Insights browser UI
- Go to Menu, Data Explorer, Search
- Depending upon the amount of data in the Data lake, the default AQL query might find the data from the DC. If it does not, modify the AQL query to find the specific DC data.
H. To enable debug logging on the DC
1. Edit the file /opt/ibm/si/services/dlc/conf/log4j2.xml
2. Locate:
<RollingFile name="InfoFileAppender" fileName="${APP_LOG_ROOT}/dlc.log" filePattern="${APP_LOG_ROOT}/archive/dlc-%d{MM-dd-yyyy}-%i.log.gz">
<Filters>
<ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
<RegexFilter regex=".* Health Agent .*" onMatch="DENY" onMismatch="ACCEPT"/>
<Filters>
<ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
<RegexFilter regex=".* Health Agent .*" onMatch="DENY" onMismatch="ACCEPT"/>
</Filters>
3. Change level=“INFO” to level=“DEBUG”
4. Also, locate:
<logger name="com.ibm.si" level="INFO" additivity="false">
<AppenderRef ref="InfoFileAppender" />
<AppenderRef ref="ErrorFileAppender" />
</logger>
5. Change level=“INFO” to level=“DEBUG”
6. Save the changes and restart the DC service
Note: Don't forget to turn off Debug once logs are collected.
7. To add or enable debug logging for all protocols defined in logSources.json, add the following in the logger section:
<logger name="com.q1labs.semsources.sources" level="DEBUG" additivity="false">
<AppenderRef ref="InfoFileAppender" />
<AppenderRef ref="ErrorFileAppender" />
</logger>
8. To enable debugging for specific protocols, add the specific class name in the name property.
For example:
For example:
<logger name="com.q1labs.semsources.sources.universalcloudrestapi" level="DEBUG" additivity="false">
<AppenderRef ref="InfoFileAppender" />
<AppenderRef ref="ErrorFileAppender" />
<AppenderRef ref="InfoFileAppender" />
<AppenderRef ref="ErrorFileAppender" />
Resolving The Problem
Resolving the problem depends upon the problem found. If the information in the "Diagnosing the problem" section does not resolve the problem, then contact support.
Document Location
Worldwide
[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSPE5X","label":"IBM Security QRadar Suite \u2013 Log Insights"},"ARM Category":[{"code":"a8m3p0000006xhZAAQ","label":"Ingestion->Data Collector"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
Was this topic helpful?
Document Information
Modified date:
20 December 2023
UID
ibm17095192