IBM Support

QRadar: Troubleshooting Log File Protocol

Troubleshooting


Problem

This is an overview on how to troubleshoot common issues with Log File Protocol.

Resolving The Problem

Unable to Pull Remote Files

Log File protocol is not pulling files from remote host. Logs are stored in the root directory on the remote server and new entries are added daily. When configuring the log source the "Ignore Previously Processed File(s)" box was checked.
image-20230818093414-1

Note: For this specific scenario, the file names are the same every day and there are no timestamp included in the file name.

There are two solutions to this problem:
  1. Uncheck the box for "Ignore Previously Processed File(s)". You will need to schedule the Log File Protocol pull to occur sometime after the DB2 file is generated. The risk is that if for some reason such as account lockout, or network issues, you might miss a pull. As a result you might loose an entire day worth of data.
  2. Preferred Procedure. Check "Ignore Previously Processed File(s)" and configure a script on the remote server to include a timestamp in the files name. This method ensures that duplicate records are not processed and allows for a missed poll and recovery of data.

Unable to Pull Correct Files

The log source is unable to point directly at the files the user wishes to pull. The directory that contains the events has 3 folders, one with data and two that are empty. When configuring the log source to recursively pull the files from these directories using FTP, you get incomplete download exceptions.
There are three possible issues to be identified here:

  1. The file path is invalid and the download has failed.
    example:
    Nov 1 15:23:36 ::ffff:10.10.10.10 [ecs-ec] [FTP Provider Protocol Provider Thread: class com.q1labs.semsources.sources.remote.transferprotocol.ftp.FTPProvider853] com.q1labs.semsources.sources.remote.transferprotocol.ftp.FTPProvider: [ERROR] [NOT:0000003000][10.10.10.10/- -] [-/- -]download failure for (/U/Qrdlogmf/Bartst//Work/Ssxmt.Pci.Ssxrp.Barts.Dly_20161101080019.Txt), reason: download incomplete
  2. The file path is to long, causing the download to fail.
    example:
    Oct 18 16:04:51 ::ffff:10.10.10.10 [ecs-ec] [FTP Provider Protocol Provider Thread: class com.q1labs.semsources.sources.remote.transferprotocol.ftp.FTPProvider308] com.q1labs.semsources.sources.remote.transferprotocol.AbstractMultiFileStreamHandler: [WARN] [NOT:0000004000][10.10.10.10/- -] [-/- -]File path has reached the maximum allowed length of 10000 characters.
     
  3. Attempting to use Log File Protocol to retrieve data from a folder, whose name dynamically changes, which results in incomplete download exceptions. This is expected behavior. Log File Protocol follows the following process:
  • Validate login credentials
  • List files from the configured remote directory
  • Download and process file(s) based on the previous step

Here are suggest solutions for this scenario:
  1. Configure Log File Protocol's polling interval so that it does not conflict with the folder renaming
    schedule.
  2. Create a folder with a static folder name and forward files from the dynamic folder to the static folder for processing.

Truncating Payloads

Log File Protocol is truncating payloads to 32000 bytes.
example:
Apr 28 11:08:28 ::ffff:10.10.10.10 [ecs-ec] [FTP Provider Protocol Provider Thread: class com.q1labs.semsources.sources.remote.transferprotocol.ftp.FTPProvider107] com.q1labs.semsources.sources.remote.transferprotocol.ftp.FTPProvider: [WARN] [NOT:0000004000][10.10.10.10/- -] [-/- -]Provider class com.q1labs.semsources.sources.remote.transferprotocol.ftp.FTPProvider107 returned a payload longer than 32000 bytes. Truncating to 32000 bytes.

Solution: Ariel, which is a proprietary event storage solution, does not support payloads greater than 32000 bytes.

Log retrieval using SFTP

Unable to retrieve logs via Log File protocol (SFTP) even though the command can be performed manually on the same event collector.

Possible solution: Authentication might be failing as per the example below:

Feb 5 12:34:52 ::ffff:10.10.10.10 [ecs-ec] [SFTP Provider Protocol Provider Thread: class com.q1labs.semsources.sources.remote.transferprotocol.sftp.SFTPProvider1145]
com.jcraft.jsch.JSchException: Auth fail


It's possible that password authentication is disabled on the remote site. Password Authentication is typically disabled on newer operating systems and the process of sending passwords from an application is different than manually typing the password. Your remote server Administrator will have to verify whether Password Authentication is enabled and enable it if it is not.

Algorithm Negotiation Failure

Connection failures with “Algorithm Negotiation Fail” error message. The protocol is attempting to connect to the remove server by using an unsupported or possibly disabled encryption method.

Solution: Ensure that the encryption method used is supported by both the client and server. If you want to use encryption greater than AES 128, you must install the Unrestricted JCE Policy files on the event collector being used.

For more information on installing encryption greater than AES 128 please refer to this IBM knowledge center article Installing the Java Cryptography Extension on QRadar.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwt0AAA","label":"Log Source"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"All Versions"}]

Document Information

Modified date:
18 August 2023

UID

swg22012805

Page Feedback