Log File protocol configuration options

To receive events from remote hosts, configure a log source to use the Log File protocol.

The Log File protocol is an active outbound protocol that is intended for systems that write daily event logs. It is not appropriate to use the Log File protocol for devices that append information to their event files.

Log files are retrieved one at a time by using SFTP, FTP, SCP, or FTPS. The Log File protocol can manage plain text, compressed files, or file archives. Archives must contain plain-text files that can be processed one line at a time. When the Log File protocol downloads an event file, the information that is received in the file updates the Log Activity tab. If more information is written to the file after the download is complete, the appended information is not processed.

The following table describes the protocol-specific parameters for the Log File protocol:

Table 1. Log File protocol parameters
Parameter	Description
Protocol Configuration	Log File
Log Source Identifier	Type a unique name for the log source. The Log Source Identifier can be any valid value and does not need to reference a specific server. It can also be the same value as the Log Source Name. If you have more than one configured Log File log source, make sure that you give each one a unique name.
Service Type	Select the protocol to use when retrieving log files from a remote server. SFTP - Secure File Transfer protocol (default) FTP - File transfer protocol FTPS - File transfer protocol secure SCP - Secure copy protocol The server that you specify in the Remote IP or Hostname field must enable the SFTP subsystem to retrieve log files with SCP or SFTP.
Remote IP or Hostname	Type the IP address or hostname of the device that contains the event log files.
Remote Port	If the remote host uses a nonstandard port number, you must adjust the port value to retrieve events.
Remote User	Type the username that you use to log in to the host that contains the event files.
Remote Password	Type the password that you use to log in to the host.
Enable Strict Host Key Checking	Enable this option to define a list of permitted public keys for the target host in the Host Key List parameter. Note: This option is only available when you select either SFTP (Secure File Transfer protocol) or SCP (Secure copy protocol) in the Service Type field.
Host Key List	Provide a list of Base64 encoded host keys to use when connecting to the target host. Separate multiple keys by using a newline and use blank lines for formatting. Supported host key types are: `ssh-dss`, `ssh-rsa`, `ecdsa-sha2-nistp256`, `ecdsa-sha2-nistp384`, and `ecdsa-sha2-nistp521`. You can obtain these keys by running the OpenSSH command `ssh-keyscan` on Linux, or `ssh-keyscan.exe` on Windows, or getting the public key from the target system directly from a location similar to `/root/.ssh/id_rsa.pub`. Use the Base64 hash only and not the hostname or algorithm. Note: This option is only available when you select either SFTP (Secure File Transfer protocol) or SCP (Secure copy protocol) in the Service Type field.
SSH Key File	If the system is configured to use key authentication, type the SSH key. When an SSH key file is used, the Remote Password field is ignored. The SSH key must be located in the /opt/qradar/conf/keys directory. Important: The SSH Key File field no longer accepts a file path. It can't contain "/" or "~". Type the file name for the SSH key. The keys for existing configurations are copied to the /opt/qradar/conf/keys directory. To ensure uniqueness, the keys must have "_`<Timestamp>`" appended to the file name.
Remote Directory	For FTP, if the log files are in the remote user’s home directory, you can leave the remote directory blank. A blank remote directory field supports systems where a change in the working directory (CWD) command is restricted.
Recursive	Enable this checkbox to allow FTP or SFTP connections to recursively search subfolders of the remote directory for event data. Data that is collected from subfolders depends on matches to the regular expression in the FTP File Pattern. The Recursive option is not available for SCP connections.
FTP File Pattern	The regular expression (regex) that is needed to identify the files to download from the remote host.
FTP Transfer Mode	For ASCII transfers over FTP, you must select `NONE` in the Processor field and `LINEBYLINE` in the Event Generator field.
FTPS TLS Version	The TLS versions that are compatible with FTPS connections. Select TLS 1.3 for the highest level of TLS security. When you select an option that supports multiple versions, the FTPS connection negotiates the latest version that is supported by both the client and the server. TLS 1.3 is compatible with QRadar 7.5.0 Update Package 5 and later. Important: TLS 1.0 and TLS 1.1 are no longer supported by QRadar 7.4.3 Fix Pack 3 and 7.5.0 Candidate Release versions, and subsequent releases will cease supporting them. Restriction: QRadar supports Explicit FTPS only If the FTP server supports session reuse, ensure that you disable it in the FTP server configuration file. This configuration option is applicable when FTPS is selected in the Service Type parameter.
SCP Remote File	For SCP file transfers, type the name of the file on the remote host. You can choose only a single file. This parameter does not support adding multiple files, including methods like file globbing or regular expressions.
Start Time	Select the time of day for the log source to start the file import. This parameter works with the Recurrence parameter to establish when and how often the remote directory is scanned for files.
Recurrence	The time interval to determine how frequently the remote directory is scanned for new event log files. The time interval can include values in hours (H), minutes (M), or days (D). For example, a recurrence of 2H scans the remote directory every 2 hours.
Run On Save	Starts the log file import immediately after you save the log source configuration. When selected, this checkbox clears the list of previously downloaded and processed files. After the first file import, the Log File protocol follows the start time and recurrence schedule that the administrator defines.
EPS Throttle	The maximum number of events per second that QRadar ingests. If your data source exceeds the EPS throttle, data collection is delayed. Data is still collected and then it is ingested when the data source stops exceeding the EPS throttle.
Processor	If the files on the remote host are stored in an archive format, select the processor that is used to decompress the event log. If the files are not stored in an archive format, select None. The default value is None.
Ignore Previously Processed File(s)	Select this checkbox to track files that the log source processes. This option prevents duplicate events from files that are processed a second time. This checkbox applies to FTP and SFTP file transfers.
Change Local Directory	Changes the local directory on the Target Event Collector to store event logs before they are processed.
Local Directory	The local directory on the Target Event Collector. The directory must exist before the Log File protocol attempts to retrieve events.
Event Generator	Choose one of the following file types to use as an event generator for the protocol. LineByLine Each line is processed as a single event. A 10 line file creates 10 separate events. HPTandem The file is processed as a HPTandem NonStop binary audit log. Each record in the log file (whether primary or secondary) is converted into text and processed as a single event. HPTandem audit logs use the following file name pattern: [aA]\d{7}. WebSphere Application Server Processes event logs for WebSphere Application Server. The remote directory must define the file path that is configured in the DSM. W3C Processes log files from sources that use the W3C format. The header of the log file identifies the order and data that is contained in each line of the file. Fair Warning Processes log files from Fair Warning devices that protect patient identity and medical information. The remote directory must define the file path to the event logs that the Fair Warning device generates. DPI Subscriber Data The file is processed as a DPI statistic log produced by a Juniper Networks MX router. The header of the file identifies the order and data that is contained in each line of the file. Each line in the file after the header is formatted to a tab-delimited `name=value` pair event. SAP Audit Logs Process files for SAP Audit Logs to keep a record of security-related events in SAP systems. Oracle BEA WebLogic Processes files for Oracle BEA WebLogic application log files. Juniper SBR Processes the event log files from Juniper Steel-belted RADIUS. ID-Linked Multiline Processes multiline event logs that contain a common value at the start of each line in a multiline event message. This option uses regular expressions to identify and reassemble the multiline event in to single event payload. Line Matcher Iterates through the lines until a line is found that matches the pattern, and discards any lines that do not match the pattern. Oracle OS XML Audit Processes the audit log produced by Oracle Database. Oracle OS Multiline Audit Processes multiline Oracle Audit logs that contains audit information like action, user, status, and so on. RegEx Based Multiline Iterates through the lines based on the start pattern, end pattern, and ignore pattern of the provided regular expressions, and discards any lines in the stream that do not match the patterns.
File Encoding	The character encoding that is used by the events in your log file.
Message ID Pattern	Type a regular expression (regex) that identifies a common value at the start of each line in a multiline event message.
Folder Separator	The character that is used to separate folders for your operating system. Most configurations can use the default value in the Folder Separator field. This field is intended for operating systems that use a different character to define separate folders. For example, periods that separate folders on mainframe systems.
Start Pattern RegEx	Type a regular expression (regex) that identifies the start pattern of each line.
End Pattern RegEx	Type a regular expression (regex) that identifies the end pattern of each line.
Ignore Pattern RegEx	Type a regular expressions (regex) to exclude a specific pattern in each line.
Date Time RegEx	Type a regular expression (regex) that identifies the date and time format of each line.
Date Time Format	Type a date and time format to identify the start of an event from each line.

Configure QRadar to use FTPS for the Log File protocol

To configure FTPS for the Log File protocol, you must place server SSL certificates on all QRadar Event Collectors that connect to your FTP server. If your SSL certificate is not RSA 2048, create a new SSL certificate.

The following command provides an example of creating a certificate on a LINUX system by using Open SSL:

openssl req -newkey rsa:2048 -nodes -keyout ftpserver.key -x509 -days 365 -out ftpserver.crt

Files on the FTP server that have a .crt file extension must be copied to the /opt/qradar/conf/trusted_certificates directory on each of your Event Collectors.