Log File protocol configuration options

To receive events from remote hosts, configure a log source to use the Log File protocol.

The Log File protocol is an active outbound protocol that is intended for systems that write daily event logs. It is not appropriate to use the Log File protocol for devices that append information to their event files.

Log files are retrieved one at a time by using SFTP, FTP, SCP, or FTPS. The Log File protocol can manage plain text, compressed files, or file archives. Archives must contain plain-text files that can be processed one line at a time. When the Log File protocol downloads an event file, the information that is received in the file updates the Log Activity tab. If more information is written to the file after the download is complete, the appended information is not processed.

The following table describes the protocol-specific parameters for the Log File protocol:

Table 1. Log File protocol parameters
Parameter	Description
Protocol Configuration	Log File
Log Source Identifier	Type a unique name for the log source. The Log Source Identifier can be any valid value and does not need to reference a specific server. It can also be the same value as the Log Source Name. If you have more than one configured Log File log source, ensure that you give each one a unique name.
Service Type	Select the protocol to use when retrieving log files from a remote server. SFTP - Secure File Transfer protocol (default) FTP - File transfer protocol FTPS - File transfer protocol secure SCP - Secure copy protocol The server that you specify in the Remote IP or Hostname field must enable the SFTP subsystem to retrieve log files with SCP or SFTP.
Remote IP or Hostname	Type the IP address or hostname of the device that contains the event log files.
Remote Port	If the remote host uses a nonstandard port number, you must adjust the port value to retrieve events.
Remote User	Type the username that you use to log in to the host that contains the event files.
Remote Password	Type the password that you use to log in to the host.
Enable Strict Host Key Checking	Enable this option to specify the public key of the target host from a predefined host key list.
Host Key List	Provide a list of accepted Base64 encoded host keys to connect to the target host. Specify multiple keys on separate lines, and use blank lines for formatting. Supported host key types are: `ssh-dss`, `ssh-rsa`, `ecdsa-sha2-nistp256`, `ecdsa-sha2-nistp384`, and `ecdsa-sha2-nistp521`. You can obtain the host keys by running the OpenSSH command `ssh-keyscan` on Linux or `ssh-keyscan.exe` on Windows, or get a public key from the target system directly from a location similar to /root/.ssh/id_rsa.pub.
SSH Key File	If the system is configured to use key authentication, type the SSH key. When an SSH key file is used, the Remote Password field is ignored. The SSH key must be located in the /opt/qradar/conf/keys directory. Important: The SSH Key File field no longer accepts a file path. It can't contain "/" or "~". Type the file name for the SSH key. The keys for existing configurations are copied to the /opt/qradar/conf/keys directory. To ensure uniqueness, the keys must have "_`<Timestamp>`" appended to the file name.
Remote Directory	For FTP, if the log files are in the remote user’s home directory, you can leave the remote directory blank. A blank remote directory field supports systems where a change in the working directory (CWD) command is restricted.
Recursive	Enable this checkbox to allow FTP or SFTP connections to recursively search subfolders of the remote directory for event data. Data that is collected from subfolders depends on matches to the regular expression in the FTP File Pattern. The Recursive option is not available for SCP connections.
FTP File Pattern	The regular expression (regex) that is needed to identify the files to download from the remote host.
FTP Transfer Mode	For ASCII transfers over FTP, you must select `NONE` in the Processor field and `LINEBYLINE` in the Event Generator field.
FTPS TLS Version	The versions of TLS that can be used with FTPS connections. To use the most secure version, select the TLSv1.2 option. When you select an option with multiple available versions, the FTPS connection negotiates the highest version available by both the client and server. This option can be configured only if you selected FTPS in the Service Type parameter.
SCP Remote File	For SCP file transfers, type the name of the file on the remote host. You can choose only a single file. This parameter does not support adding multiple files, including methods like file globbing or regular expressions.
Start Time	Select the time of day for the log source to start the file import. This parameter works with the Recurrence parameter to establish when and how often the remote directory is scanned for files.
Recurrence	The time interval to determine how frequently the remote directory is scanned for new event log files. The time interval can include values in hours (H), minutes (M), or days (D). For example, a recurrence of 2H scans the remote directory every 2 hours.
Run On Save	Starts the log file import immediately after you save the log source configuration. When selected, this checkbox clears the list of previously downloaded and processed files. After the first file import, the Log File protocol follows the start time and recurrence schedule that the administrator defines.
EPS Throttle	The number of Events Per Second (EPS) that the protocol cannot exceed.
Processor	If the files on the remote host are stored in an archive format, select the processor that is used to decompress the event log. If the files are not stored in an archive format, select None. The default value is None.
Ignore Previously Processed File(s)	Select this checkbox to track files that the log source processes. This option prevents duplicate events from files that are processed a second time. This checkbox applies to FTP and SFTP file transfers.
Change Local Directory?	Changes the local directory on the Target Event Collector to store event logs before they are processed.
Local Directory	The local directory on the Target Event Collector. The directory must exist before the Log File protocol attempts to retrieve events.
Event Generator	Choose one of the following file types to use as an event generator for the protocol. LineByLine Each line is processed as a single event. A 10 line file creates 10 separate events. HPTandem The file is processed as a HPTandem NonStop binary audit log. Each record in the log file (whether primary or secondary) is converted into text and processed as a single event. HPTandem audit logs use the following file name pattern: [aA]\d{7}. WebSphere Application Server Processes event logs for WebSphere Application Server. The remote directory must define the file path that is configured in the DSM. W3C Processes log files from sources that use the W3C format. The header of the log file identifies the order and data that is contained in each line of the file. Fair Warning Processes log files from Fair Warning devices that protect patient identity and medical information. The remote directory must define the file path to the event logs that the Fair Warning device generates. DPI Subscriber Data The file is processed as a DPI statistic log produced by a Juniper Networks MX router. The header of the file identifies the order and data that is contained in each line of the file. Each line in the file after the header is formatted to a tab-delimited `name=value` pair event. SAP Audit Logs Process files for SAP Audit Logs to keep a record of security-related events in SAP systems. Oracle BEA WebLogic Processes files for Oracle BEA WebLogic application log files. Juniper SBR Processes event log files from Juniper Steel-belted RADIUS. ID-Linked Multiline Processes multiline event logs that contain a common value at the start of each line in a multiline event message. This option uses regular expressions to identify and reassemble the multiline event in to single event payload.
File Encoding	The character encoding that is used by the events in your log file.
Message ID Pattern	Type a regular expression (regex) that identifies a common value at the start of each line in a multiline event message.
Folder Separator	The character that is used to separate folders for your operating system. Most configurations can use the default value in Folder Separator field. This field is intended for operating systems that use a different character to define separate folders. For example, periods that separate folders on mainframe systems.

Configure QRadar to use FTPS for the Log File protocol

To configure FTPS for the Log File protocol, you must place server SSL certificates on all QRadar® Event Collectors that connect to your FTP server. If your SSL certificate is not RSA 2048, create a new SSL certificate.

The following command provides an example of creating a certificate on a LINUX system by using Open SSL:

openssl req -newkey rsa:2048 -nodes -keyout ftpserver.key -x509 -days 365 -out ftpserver.crt

Files on the FTP server that have a .crt file extension must be copied to the /opt/qradar/conf/trusted_certificates directory on each of your Event Collectors.