Securing Apache Kafka client communications

Set up secure, SSL-encrypted communication between Kafka clients and Kafka brokers (servers). Along with secured communication, you can also authenticate client applications with the Kafka brokers using either SASL or SSL.

Securing Kafka client communications using SSL
Securing Kafka client communications using SASL

Securing Kafka client communications using SSL

To secure Kafka client-server communications using SSL you must enable SSL for the broker and for each of the client applications.

Note: Before you begin, ensure that you have completed the steps documented in Creating SSL artifacts.

Enable SSL on the Kafka broker.
Kafka brokers use the server.properties file for security configuration.
Important: The server.properties configuration file is used by Apache Kafka components, so the passwords must be stored in clear text format. It is important to ensure that you protect the files using file system permissions.
1. Update the server.properties file to add the following security parameters:
```
listeners=SSL://<KAFKA_SERVER_HOSTNAME>:<KAFKA_SERVER_SSL_PORT>
ssl.keystore.location=<SSL_KEYSTORE_FILEPATH>
ssl.keystore.password=<KEYSTORE_PASSWORD>
ssl.key.password=<KEY_PASSWORD>
ssl.truststore.location=<SSL_TRUSTSTORE_FILEPATH>
ssl.truststore.password=<SSL_TRUSTSTORE_PASSWORD>
security.inter.broker.protocol=SSL
```
  Replace the placeholders with the corresponding values for your deployment.
2. If you want client authentication to be done using SSL, add the following additional parameter to the file:
```
ssl.client.auth=required
```
Enable SSL on the Kafka client applications.
The InfoSphere® MDM implementation of Apache Kafka uses multiple client applications such as the Database Connector, batch stream processor, and runtime stream processor. These all must be secured.
- Configuring security for the Database Connector
  The Database Connector uses the connect-distributed.properties file for security configuration.
  Important: The connect-distributed.properties configuration file is used by Apache Kafka components, so the passwords must be stored in clear text format. It is important to ensure that you protect the files using file system permissions.
  1. Update the connect-distributed.properties file to add the following security parameters:
```
# Worker authentication settings
bootstrap.servers= SSL://<KAFKA_SERVER_HOSTNAME>:<KAFKA_SERVER_SSL_PORT>
security.protocol=SSL
ssl.truststore.location=<CLIENT_SSL_TRUSTSTORE_FILEPATH>
ssl.truststore.password=<CLIENT_SSL_TRUSTSTORE_PASSWORD>

# Source Connector authentication settings
producer.security.protocol=SSL
producer.ssl.truststore.location=<CLIENT_SSL_TRUSTSTORE_FILEPATH>
producer.ssl.truststore.password=<CLIENT_SSL_TRUSTSTORE_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
  2. If client authentication using SSL is configured for the broker (ssl.client.auth=required), then configure the following additional properties in connect-distributed.properties:
```
producer.ssl.keystore.location=<CLIENT_SSL_KEYSTORE_FILEPATH>
producer.ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
producer.ssl.key.password=<CLIENT_KEY_PASSWORD>
ssl.keystore.location=<CLIENT_SSL_KEYSTORE_FILEPATH>
ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
ssl.key.password=<CLIENT_KEY_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
- Configuring security for Kafka in WebSphere® Application Server
  To configure SSL security for your Kafka client deployment in WebSphere Application Server, complete the steps documented in Enabling SSL on Kafka to secure MDM notifications.
- Configuring security for the batch stream processor
  The batch stream processor uses the kafkabatch.properties file for security configuration.
  Important: Encode the passwords for client stream applications using the PropFilePasswordEncoder utility in WebSphere Application Server.
  1. Update the kafkabatch.properties file to add the following security parameters:
```
enableSSLMessaging=true
security.protocol=SSL
kafkaconnect.ssl.truststore.location=<CLIENT_TRUST_STORE_FILE_PATH>
kafkaconnect.ssl.truststore.password=<CLIENT_TRUSTSTORE_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
  2. If client authentication using SSL is configured for the broker (ssl.client.auth=required), then configure the following additional properties in kafkabatch.properties:
```
kafkaconnect.ssl.keystore.location=<CLIENT_KEYSTORE_FILE_PATH>
kafkaconnect.ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
kafkaconnect.ssl.key.password=<CLIENT_KEY_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
- Configuring security for the runtime stream processor
  The runtime stream processor uses the kafkaruntime.properties file for security configuration.
  Important: Encode the passwords for client stream applications using the PropFilePasswordEncoder utility in WebSphere Application Server.
  1. Update the kafkabatch.properties file to add the following security parameters:
```
enableSSLMessaging=true
security.protocol=SSL
kafkaconnect.ssl.truststore.location=<CLIENT_TRUST_STORE_FILE_PATH>
kafkaconnect.ssl.truststore.password=<CLIENT_TRUSTSTORE_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
  2. If client authentication using SSL is configured for the broker (ssl.client.auth=required), then configure the following additional properties in kafkaruntime.properties:
```
kafkaconnect.ssl.keystore.location=<CLIENT_KEYSTORE_FILE_PATH>
kafkaconnect.ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
kafkaconnect.ssl.key.password=<CLIENT_KEY_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
- Configuring SSL for the MDM Statistics feature
  The MDM Statistics feature uses the statistics-streams.properties and statistics-connect-standalone.properties files for security configuration.
  1. Update the statistics-streams.properties file to add the following security parameters:
```
entity.statistics.ssl.truststore.location=<CLIENT_TRUST_STORE_FILE_PATH>
entity.statistics.ssl.truststore.password=<CLIENT_TRUSTSTORE_PASSWORD>
entity.statistics.security.protocol=SSL
transaction.statistics.ssl.truststore.location=<CLIENT_TRUST_STORE_FILE_PATH>
transaction.statistics.ssl.truststore.password=<CLIENT_TRUSTSTORE_PASSWORD>
transaction.statistics.security.protocol=SSL
```
    Replace the placeholders with the corresponding values for your deployment.
  2. If client authentication using SSL is configured for the broker (ssl.client.auth=required), then configure the following additional properties in statistics-streams.properties:
```
entity.statistics.ssl.keystore.location=<CLIENT_KEYSTORE_FILE_PATH>
entity.statistics.ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
entity.statistics.ssl.key.password=<CLIENT_KEY_PASSWORD>
transaction.statistics.ssl.keystore.location=<CLIENT_KEYSTORE_FILE_PATH>
transaction.statistics.ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
transaction.statistics.ssl.key.password=<CLIENT_KEY_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
  3. Update the statistics-connect-standalone.properties file to add the following security parameters:
```
consumer.ssl.truststore.location=<CLIENT_TRUST_STORE_FILE_PATH>
consumer.ssl.truststore.password=<CLIENT_TRUSTSTORE_PASSWORD>
consumer.security.protocol=SSL
ssl.truststore.location=<CLIENT_TRUST_STORE_FILE_PATH>
ssl.truststore.password=<CLIENT_TRUSTSTORE_PASSWORD>
security.protocol=SSL
```
    Replace the placeholders with the corresponding values for your deployment.
  4. If client authentication using SSL is configured for the broker (ssl.client.auth=required), then configure the following additional properties in statistics-connect-standalone.properties:
```
consumer.ssl.keystore.location=<CLIENT_KEYSTORE_FILE_PATH>
consumer.ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
consumer.ssl.key.password=<CLIENT_KEY_PASSWORD>
ssl.keystore.location=<CLIENT_KEYSTORE_FILE_PATH>
ssl.keystore.password=<CLIENT_KEYSTORE_PASSWORD>
ssl.key.password=<CLIENT_KEY_PASSWORD>
```
    Replace the placeholders with the corresponding values for your deployment.
- Configuring security for the MDM Publisher ongoing synchronization stream processor
  Complete the steps documented in Configuring InfoSphere MDM to enable ongoing synchronization.

Securing Kafka client communications using SASL

To secure Kafka client-server communications using SASL you must enable SASL for the broker and for each of the client applications.

Note: Before you begin, ensure that you have completed the steps documented in Creating SSL artifacts.

Enable SASL on the Kafka broker.
1. In each Kafka broker's config directory, create a new JAAS file named kafka_server_jaas.conf. The content should be similar to the following example:
```
KafkaServer {
            org.apache.kafka.common.security.plain.PlainLoginModule required
            username="admin"
            password="admin-secret"
            user_admin="admin-secret"
            user_mdmuser=" mdmuser-secret";
        };
```
  This example configuration defines two users: admin and mdmuser. The username and password properties in the KafkaServer section are used by the broker to initiate connections to other brokers. In this example, admin is the user for inter-broker communication. The set of properties user_userName defines the passwords for all users that connect to the broker. The broker validates all client connections, including those from other brokers using these properties. For more information, see the Apache Kafka documentation.
2. Pass the JAAS configuration file location to each Kafka broker as a JVM parameter:
```
Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf
```
3. Update the server.properties file to add the following security parameters and configure the SASL_SSL port and the SASL mechanism:
```
listeners=SASL_SSL://host.name:port
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
ssl.truststore.location=/kafkasecurity/server.truststore.jks
ssl.truststore.password=storep@ss
ssl.keystore.location=/kafkasecurity/server.keystore.jks
ssl.keystore.password=keyp@ss
ssl.key.password=keyp@ss
```
Enable SASL on the Kafka client applications.
The InfoSphere MDM implementation of Apache Kafka uses multiple client applications such as the Database Connector, batch stream processor, and runtime stream processor. These all must be secured.
- Configuring security for the Database Connector
  1. Create a JAAS configuration file named kafka_client_jaas.conf.
  2. Within kafka_client_jaas.conf, create a client login section named KafkaClient to configure a login module for the selected mechanism.
```
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="<USER_NAME>"
  password="<PASSWORD>";
};
```
    Replace the placeholders with the corresponding values for the user who will be used to authenticate with the Kafka broker.
  3. Pass the JAAS configuration location to each Kafka client as a JVM parameter.
```
Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
```
  4. Update the connect-distributed.properties file to add the following security parameters:
```
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
producer.protocol=SASL_SSL
producer.sasl.mechanism=PLAIN
```
- Configuring security for Kafka in WebSphere Application Server
  In WebSphere Application Server, the default JAAS file is named wsjaas.conf. This file is stored in the properties folder of the WebSphere Application Server profile.
  1. Edit wsjaas.conf and append the following KafkaClient section:
```
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="<USER_NAME>"
  password="<PASSWORD>";
};
```
    Replace the placeholders with the corresponding values for the user who will be used to authenticate with the Kafka broker.
  2. Add the same KafkaClient section to the default WebSphere Application Server wsjaas.conf configuration files found in the following locations:
    - <MDM_INSTALL_HOME>/KafkaProcessor/properties
    - <WAS_INSTALL_HOME>/AppServer/profiles/<PROFILE_NAME>/properties
  3. Complete the steps documented in Enabling SSL on Kafka to secure MDM notifications.
- Configuring security for the batch stream processor
  1. Create a JAAS configuration file named kafka_client_jaas.conf.
  2. Within kafka_client_jaas.conf, create a client login section named KafkaClient to configure a login module for the selected mechanism.
```
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="<USER_NAME>"
  password="<PASSWORD>";
};
```
    Replace the placeholders with the corresponding values for the user who will be used to authenticate with the Kafka broker.
  3. Pass the JAAS configuration location to each Kafka client as a JVM parameter.
```
Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
```
  4. Update the kafkabatch.properties file to add the following security parameters:
```
security.protocol=SASL_SSL
kafkaconnect.sasl.mechanism=PLAIN
```
    Replace the placeholders with the corresponding values for your deployment.
- Configuring security for the runtime stream processor
  1. Create a JAAS configuration file named kafka_client_jaas.conf.
  2. Within kafka_client_jaas.conf, create a client login section named KafkaClient to configure a login module for the selected mechanism.
```
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="<USER_NAME>"
  password="<PASSWORD>";
};
```
    Replace the placeholders with the corresponding values for the user who will be used to authenticate with the Kafka broker.
  3. Pass the JAAS configuration location to each Kafka client as a JVM parameter.
```
Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
```
  4. Update the kafkaruntime.properties file to add the following security parameters:
```
security.protocol=SASL_SSL
kafkaconnect.sasl.mechanism=PLAIN
```
    Replace the placeholders with the corresponding values for your deployment.
- Configuring SSL for the MDM Statistics feature
  1. Create a JAAS configuration file named kafka_client_jaas.conf.
  2. Within kafka_client_jaas.conf, create a client login section named KafkaClient to configure a login module for the selected mechanism.
```
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="<USER_NAME>"
  password="<PASSWORD>";
};
```
    Replace the placeholders with the corresponding values for the user who will be used to authenticate with the Kafka broker.
  3. Pass the JAAS configuration location to each Kafka client as a JVM parameter.
```
Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
```
  4. Update the statistics-streams.properties file to add the following security parameters:
```
entity.statistics.security.protocol=SASL_SSL
entity.statistics.sasl.mechanism=PLAIN
transaction.statistics.security.protocol=SASL_SSL
transaction.statistics.sasl.mechanism=PLAIN
```
  5. Update the statistics-connect-standalone.properties file to add the following security parameters:
```
consumer.protocol=SASL_SSL
consumer.sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
```
- Configuring security for the MDM Publisher stream processor
  1. Create a JAAS configuration file named kafka_client_jaas.conf.
  2. Within kafka_client_jaas.conf, create a client login section named KafkaClient to configure a login module for the selected mechanism.
```
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="<USER_NAME>"
  password="<PASSWORD>";
};
```
    Replace the placeholders with the corresponding values for the user who will be used to authenticate with the Kafka broker.
  3. Pass the JAAS configuration location to each Kafka client as a JVM parameter.
```
Djava.security.auth.login.config=/etc/kafka/kafka_client_jaas.conf
```
  4. To complete the configuration, follow the steps documented in Configuring InfoSphere MDM to enable ongoing synchronization.