Apache Kafka protocol configuration options

IBM® QRadar® uses the Apache Kafka protocol to read streams of event data from topics in a Kafka cluster that uses the Consumer API. A topic is a category or feed name in Kafka where messages are stored and published. The Apache Kafka protocol is an outbound or active protocol, and can be used as a gateway log source by using a custom log source type.

The Apache Kafka protocol supports topics of almost any scale. You can configure multiple QRadar collection hosts (EP/ECs) to collect from a single topic; for example, all firewalls. For more information, see the Kafka Documentation (http://kafka.apache.org/documentation/).

The following table describes the protocol-specific parameters for the Apache Kafka protocol:
Table 1. Apache Kafka protocol parameters
Parameter Description
Log Source Identifier

Type a unique name for the log source.

The Log Source Identifier can be any valid value and does not need to reference a specific server. It can also be the same value as the Log Source Name. If you have more than one configured Apache Kafka log source, ensure that you give each one a unique name.
Bootstrap Server List The <hostname/ip>:<port> of the bootstrap server (or servers). Multiple servers can be specified in a comma-separated list, such as in this example: hostname1:9092,1.1.1.1:9092.
Consumer Group

A unique string or label that identifies the consumer group that this log source belongs to.

Each record that is published to a Kafka topic is delivered to one consumer instance within each subscribing consumer group. Kafka uses these labels to load balance the records over all consumer instances in a group.
Topic Subscription Method The method that is used for subscribing to Kafka topics. Use the List Topics option to specify a specific list of topics. Use the Regex Pattern Matching option to specify a regular expression to match against available topics.
Topic List

A list of topic names to subscribe to. The list must be comma-separated; for example: Topic1,Topic2,Topic3

This option is only displayed when List Topics is selected for the Topic Subscription Method option.
Topic Filter Pattern

A regular expression to match the topics to subscribe to.

This option is only displayed when Regex Pattern Matching is selected for the Topic Subscription Method option.
Use SASL Authentication

This option displays SASL authentication configuration options.

When used without client authentication, you must place a copy of the server certificate in the /opt/qradar/conf/trusted_certificates/ directory.
Use Client Authentication Displays the client authentication configuration options.
Key Store/Trust Store Type
The archive file format for your keystore and truststore type. The following options are available for the archive file format:
  • JKS
  • PKCS12
Trust Store Filename The name of the truststore file. The truststore must be placed in /opt/qradar/conf/trusted_certificates/kafka/.

The file contains the username and password.
Key Store Filename The name of the keystore file. The keystore must be placed in /opt/qradar/conf/trusted_certificates/kafka/.

The file contains the username and password.
Use As A Gateway Log Source This option enables collected events to go through the QRadar Traffic Analysis engine and to automatically detect the appropriate log sources.
Log Source Identifier Pattern

Defines a custom Log Source Identifier for events that are being processed, if the Use As A Gateway Log Source checkbox is selected.

Key-value pairs are used to define the custom Log Source Identifier. The key is the Identifier Format String, which is the resulting source or origin value. The value is the associated regex pattern that is used to evaluate the current payload. This value also supports capture groups that can be used to further customize the key.

Multiple key-value pairs are defined by typing each pattern on a new line. Multiple patterns are evaluated in the order that they are listed. When a match is found, a custom Log Source Identifier is displayed.

The following examples show multiple key-value pair functions.
Patterns
VPC=\sREJECT\sFAILURE
$1=\s(REJECT)\sOK
VPC-$1-$2=\s(ACCEPT)\s(OK)
Events
{LogStreamName: LogStreamTest,Timestamp: 0,Message: ACCEPT OK,IngestionTime: 0,EventId: 0}
Resulting custom log source identifier
VPC-ACCEPT-OK
Character Sequence Replacement Replaces specific literal character sequences in the event payload to actual characters. One or more of the following options are available:
  • Newline(CR LF) Character (\r\n)
  • Line Feed Character (\n)
  • Carriage Return Character (\r)
  • Tab Character (\t)
  • Space Character (\s)
EPS Throttle The maximum number of events per second (EPS). No throttling is applied if the field is empty.