Apache Kafka protocol configuration options

IBM QRadar uses the Apache Kafka protocol to read streams of event data from topics in a Kafka cluster that uses the Consumer API. A topic is a category or feed name in Kafka where messages are stored and published. The Apache Kafka protocol is an outbound or active protocol, and can be used as a gateway log source by using a custom log source type.

The Apache Kafka protocol supports topics of almost any scale. You can configure multiple QRadar collection hosts (EP/ECs) to collect from a single topic; for example, all firewalls. For more information, see the Kafka Documentation (http://kafka.apache.org/documentation/).

The following table describes the protocol-specific parameters for the Apache Kafka protocol:

Table 1. Apache Kafka protocol parameters
Parameter	Description
Log Source Identifier	Type a unique name for the log source. The Log Source Identifier can be any valid value and does not need to reference a specific server. It can also be the same value as the Log Source Name. If you have more than one configured Apache Kafka log source, ensure that you give each one a unique name.
Bootstrap Server List	The `<hostname/ip>:<port>` of the bootstrap server (or servers). Multiple servers can be specified in a comma-separated list, such as in this example: `hostname1:9092,1.1.1.1:9092`, `[fdff:ffff:ffff:ffff:ffff:ffff:ffff:ffff]:9092`.
Consumer Group	A unique string or label that identifies the consumer group that this log source belongs to. Each record that is published to a Kafka topic is delivered to one consumer instance within each subscribing consumer group. Kafka uses these labels to load balance the records over all consumer instances in a group.
Topic Subscription Method	The method that is used for subscribing to Kafka topics. Use the List Topics option to specify a specific list of topics. Use the Regex Pattern Matching option to specify a regular expression to match against available topics.
Topic List	A list of topic names to subscribe to. The list must be comma-separated; for example: `Topic1,Topic2,Topic3` This option is only displayed when List Topics is selected for the Topic Subscription Method option.
Topic Filter Pattern	A regular expression to match the topics to subscribe to. This option is only displayed when Regex Pattern Matching is selected for the Topic Subscription Method option.
Use SASL Authentication	This option displays SASL authentication configuration options. When used without client authentication, you must place a copy of the server certificate in the /opt/qradar/conf/trusted_certificates/ directory.
SASL Mechanism	Select the SASL mechanism that is compatible with your Kafka configuration: `PLAIN` `SCRAM-SHA-256` `SCRAM-SHA-512`
SASL Username	The username that is used for SASL authentication.
SASL Password	The password that is used for SASL authentication.
Use SSL	Select this option to enable SSL (TLS) encryption if your Kafka configuration supports or requires it.
Use Client Authentication	Displays the client authentication configuration options. You can enable this option only if you enable the Use SSL parameter and use SSL (TLS) for authentication and data transfer.
TLS Protocols	The versions of TLS that is allowed for this protocol. Clients must send request by using the same version that is selected for the server. TLSv1.3 is supported by QRadar version 7.5.0 UP5 and later.
Key Store/Trust Store Type	The archive file format for your keystore and truststore type. The following options are available for the archive file format: JKS PKCS12
Trust Store Filename	The name of the truststore file. The truststore must be placed in /opt/qradar/conf/trusted_certificates/kafka/. The file contains the username and password.
Key Store Filename	The name of the keystore file. The keystore must be placed in /opt/qradar/conf/trusted_certificates/kafka/. The file contains the username and password.
Use As A Gateway Log Source	This option enables collected events to go through the QRadar Traffic Analysis engine and to automatically detect the appropriate log sources.
Log Source Identifier Pattern	Defines a custom Log Source Identifier for events that are being processed, if the Use As A Gateway Log Source checkbox is selected. Key-value pairs are used to define the custom Log Source Identifier. The key is the Identifier Format String, which is the resulting source or origin value. The value is the associated regex pattern that is used to evaluate the current payload. This value also supports capture groups that can be used to further customize the key. Multiple key-value pairs are defined by typing each pattern on a new line. Multiple patterns are evaluated in the order that they are listed. When a match is found, a custom Log Source Identifier is displayed. The following examples show multiple key-value pair functions. Patterns `VPC=\sREJECT\sFAILURE` `$1=\s(REJECT)\sOK` `VPC-$1-$2=\s(ACCEPT)\s(OK)` Events `{LogStreamName: LogStreamTest,Timestamp: 0,Message: ACCEPT OK,IngestionTime: 0,EventId: 0}` Resulting custom log source identifier VPC-ACCEPT-OK
Show Advanced Options	Show optional advanced options for the Kafka configuration. The advanced option values are in effect whether they are shown or not.
Use Payload Extraction	Enable this parameter to extract the payload and send it to the event pipeline. This parameter identifies the specified payload if it is somewhere within the Kafka log records. Multiple regular expressions can be defined by entering each pattern on a new line. When multiple Payload Extraction patterns are used, they are evaluated in order until a match is found and an extracted payload can be returned. This payload extraction occurs before any character replacements.
Payload Extraction Regular Expression	A regular expression that identifies the specified payload within the Kafka log records so it can be sent to QRadar. This expression must include a capture group and uses the first capture group as the new payload.
Use Predictive Parsing	If you enable this parameter, an algorithm extracts log source identifier patterns and extracts payloads from events without running the regex for every event, which increases the parsing speed. In rare circumstances, the algorithm can make incorrect predictions. Enable predictive parsing only for log source types that you expect to receive high event rates and require faster parsing.
Character Sequence Replacement	Replaces specific literal character sequences that are in the event payload with actual characters. One or more of the following options are available: Newline(CR LF) Character (\r\n) Line Feed Character (\n) Carriage Return Character (\r) Tab Character (\t) Space Character (\s) Unescape JSON Data Tip: Select this option if you want the entire payload to be JSON unescaped after any payload extraction. When you unescape JSON messages, any character that prevents parsing is removed from the message. Enable this option when you want to extract JSON messages that are embedded in JSON objects.
Kafka Consumer Properties Override	A list of `key=value` pairs that can be used to provide specific configuration properties to the Kafka Consumer. The list uses one pair per line. For example, the `key=value` pair `session.timeout.ms=10000` configures the session timeout in milliseconds. For a list of available `key=value` pairs, see the Kafka Consumer Configuration documentation (https://ibm.biz/kafkaconsumerconfigs). Any parameters that are entered in this field override any previous ones set during the configuration phase of the log source. These parameters include, but are not limited to, the following examples: `fetch.max.bytes` `group.id` `ssl.enabled.protocols` You cannot enter any password-type properties with secret values in this field. These properties include, but are not limited to, the following examples: `ssl.key.password` `ssl.key.password` `ssl.keystore.password` `ssl.truststore.password` `sasl.jaas.config` `ssl.truststore.certificates` `ssl.keystore.certificate.chain` `ssl.keystore.key` Use the Private Key Password, Trust Store Password, Key Store Password, Private Key Password, or SASL Password fields to enter password-type Kafka consumer properties.
EPS Throttle	The maximum number of events per second that QRadar ingests. If your data source exceeds the EPS throttle, data collection is delayed. Data is still collected and then it is ingested when the data source stops exceeding the EPS throttle.