Configuring the Kafka transport

The Kafka transport allows the probe to integrate with a Kafka server to consume events.

Kafka transport properties table

The following table describes the properties used to configure the kafkaTransport.properties file.

Table 1. Kafka transport properties
Property name	Description
KafkaClientMode	Use this property to set the transport as a Kafka client to run as a consumer or a producer. This property takes the following values: CONSUMER: A Kafka consumer reads data from topics. PRODUCER: A Kafka producer writes data to topics.
ConnectionPropertiesFile	Use this property to specify the JSON file holding the Kafka connection properties.

Kafka connection properties table

The following table describes the properties used to configure the kafkaConnectionProperties.json file.

Table 2. Kafka connection properties
Property name		Description
zookeeper_client	target	Use this property to specify the ZooKeeper endpoint. When this property is empty, the transport will not initiate connection to ZooKeeper. The default is empty.
	properties	Use this property to specify the path to a file holding ZooKeeper client properties in key-value format, for example: `key=value` The default is empty.
	java_sys_props	Use this property to specify the path to a file holding ZooKeeper client Java system properties required in a secure connection. The default is empty.
	topic_watch	Use this property to enable the ZooKeeper topic watch service. Valid values are: true: Enable the ZooKeeper topic watch service. false: Disable the ZooKeeper topic watch service. The default is true.
	broker_watch	Use this property to enable the ZooKeeper broker watch service. Valid values are: true: Enable the ZooKeeper broker watch service. false: Disable the ZooKeeper broker watch service. The default is true.
brokers		Use this property to specify broker endpoints in a comma-separated list. For example: `“localhost:9092, localhost:9093, localhost:9094”` The brokers must belong to the same cluster managed by a zookeeper. The default is empty.
topics		Use this property to specify topics in a comma-separated list. For example: `“topic1, topic2, topic3”` The default is empty.
Kafka_client	properties	Use this property to specify the path to a file holding Kafka client properties. The default is empty.
Kafka_client	java_sys_props	Use this property to specify the path to a file holding the Kafka client’s Java system properties required in a secure connection. The default is empty.

Kafka configuration for different connection protocols

Kafka supports following types of connection protocol:

SASL_PLAINTEXT
SASL_SSL

The following table describes the configuration required by each connection protocol.

Table 3. Connection protocol configuration
Connection protocol	Configuration required
`SASL_PLAINTEXT`	Kafka producer properties `security.protocol=SASL_PLAINTEXT` Note: Must combine with SASL-specific configurations.
`SASL_SSL`	Kafka producer properties `acks=all security.protocol=SASL_SSL sasl.mechanism=PLAIN ssl.protocol=TLSv1.2 ssl.enabled.protocols=TLSv1.2 ssl.truststore.location=<path>\<trust_store_file> ssl.truststore.password=<trust_store_password> ssl.truststore.type=JKS` Java system properties `java.security.auth.login.config=<path>/user_jaas.conf https.protocols=TLSv1.2` Note: Must combine with SASL-specific configurations.

Connection protocol

Configuration required

SASL_PLAINTEXT

Kafka producer properties

security.protocol=SASL_PLAINTEXT

Note: Must combine with SASL-specific configurations.

SASL_SSL

Kafka producer properties

acks=all   
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
ssl.protocol=TLSv1.2
ssl.enabled.protocols=TLSv1.2


ssl.truststore.location=<path>\<trust_store_file>
ssl.truststore.password=<trust_store_password>
ssl.truststore.type=JKS

Java system properties

java.security.auth.login.config=<path>/user_jaas.conf
https.protocols=TLSv1.2

Note: Must combine with SASL-specific configurations.

The following table describes SASL-specific configurations.

Table 4. SASL-specific configuration
SASL: Kafka user access control	SASL: Kerberos
Java system properties `java.security.auth.login.config=<path>/user_jaas.conf` Example user_jass.conf `KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="kafka" username="myUserName" password="myPasword"; };`	Java system properties `java.security.auth.login.config=<path>/user_jass.conf java.security.krb5.conf=<path>/krb5.conf` Example user_jass.conf When using IBM JDK `KafkaClient { com.ibm.security.auth.module.Krb5LoginModule required debug=true credsType=both useKeytab="<path>/kafka.keytab" principal="username/instance@realm"; };` When using Oracle JDK `KafkaClient { com.sun.security.auth.module.Krb5LoginModule required debug=true renewTicket=true serviceName="kafka" useKeyTab=true keyTab="<path>/kafka.keytab" principal="username/instance@realm"; };` Note: This is the generic format of principal: `username/instance@realm`. Some organizations might use `servicename` instead of `username` or without `username` `principal="servicename/instance@realm" principal="instance@realm` Consult your organization administrator for principal information.

SASL: Kafka user access control

SASL: Kerberos

Java system properties

java.security.auth.login.config=<path>/user_jaas.conf

Example user_jass.conf

KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  serviceName="kafka"
  username="myUserName"
  password="myPasword";
};

Java system properties

java.security.auth.login.config=<path>/user_jass.conf
java.security.krb5.conf=<path>/krb5.conf

Example user_jass.conf

When using IBM JDK

KafkaClient {
   com.ibm.security.auth.module.Krb5LoginModule required
   debug=true
   credsType=both
   useKeytab="<path>/kafka.keytab"
   principal="username/instance@realm";
};

When using Oracle JDK

KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   debug=true
   renewTicket=true
   serviceName="kafka"
   useKeyTab=true
   keyTab="<path>/kafka.keytab"
   principal="username/instance@realm";
};

Note: This is the generic format of principal: username/instance@realm. Some organizations might use servicename instead of username or without username

principal="servicename/instance@realm"
principal="instance@realm

Consult your organization administrator for principal information.

Note:

Kafka producer properties are configured in the file specified in the kafka_client.properties field.

Java system properties are configured in the file specified in the kafka_client.java_sys_props field.

In broker list configuration, a broker endpoint without a protocol prefix is assumed to be using the protocol configured in the security.protocol property. An unconfigured security.protocol denotes PLAINTEXT.