IBM Support

Kafka Security Mechanism (SASL/PLAIN) - Hadoop Dev

Technical Blog Post


Kafka Security Mechanism (SASL/PLAIN) - Hadoop Dev


Starting from Kafka 0.10.x Kafka Broker supports username/password authentication. This Mechanism is called SASL/PLAIN. The steps below describe how to set up this mechanism on an IOP 4.2.5 Kafka Cluster.

1. Configure the Kafka brokers and Kafka Clients

Add a JAAS configuration file for each Kafka broker. Create a kafka_plain_jaas.conf file as specified below:

KafkaServer { required     username="kafka"     password="kafka-secret"     user_kafka="kafka-secret"     user_ibm="ibm-secret";  };  KafkaClient { required    username="ibm"    password="ibm-secret";  };

Let’s understand the content of kafka_plain_jaas.conf file and how Kafka Brokers and Kafka Clients use it.

KafkaServer Section:
The KafkaServer section defines two users: kafka and ibm. The properties username and password are used by the broker to initiate connections to other brokers. In this example, kafka is the user for inter-broker communication. The set of properties user_{userName} defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties.

KafkaClient Section:
The Kafka Client section describes how the clients, Producer and Consumer, can connect to Kafka Broker. The properties username and password in the Kafka Client section are used by clients to configure the user for client connections. In this example, clients connect to the broker as user “ibm”.

2. Pass the JAAS configuration location as a JVM Parameter to the Kafka broker

  Update the Advanced kafka-env section to add the properties below:  export KAFKA_PLAIN_PARAMS=""  export KAFKA_OPTS="$KAFKA_PLAIN_PARAMS $KAFKA_OPTS"    		  

3. Add the properties below to the custom Kafka broker configuration.

Kafka Broker Configuration

4. Update the listeners property in the Kafka Broker section to SASL_PLAINTEXT://localhost:6667

Kafka Broker Configuration

5. Restart Kafka.

Run the Kafka Client Console Producer / Console Consumer to authenticate to the Kafka broker with the specified username and password. In this example, the username used to connect to brokers is “ibm”. The JVM parameter for console producer and console consumer is set by the Step 2 above.
All the commands shown below are run from the path /usr/iop/

Create a topic:

[root@heel1 kafka]# bin/ --create --topic plain-topic --zookeeper --partitions 1 --replication-factor 1  [2017-04-10 12:06:22,239] WARN SASL configuration failed: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/iop/'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)  Created topic "plain-topic".

Run Kafka console producer

Before running the Kafka console Producer configure the file as shown:

[root@heel1 kafka]# cat   security.protocol=SASL_PLAINTEXT  sasl.mechanism=PLAIN
[root@heel1 kafka]# bin/ --broker-list --topic plain-topic --producer.config   Message 1  Message 2  Message 3  ^C

Run Kafka console consumer

Before running Kafka console consumer configure the file as shown:

[root@heel1 kafka]# cat   security.protocol=SASL_PLAINTEXT  sasl.mechanism=PLAIN
[root@heel1 kafka]# bin/ --bootstrap-server --topic plain-topic --from-beginning --consumer.config   Message 1  Message 2  Message 3  ^CProcessed a total of 3 messages

The username is used as the authenticated Principal, which is used in authorization (e.g. ACLs).
For usage of ACLS refer to the blog Kafka ACLs

For more information refer to the kafka documentation

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]