Configuring Apache Kafka to enable Client Authentication
This task discusses how to enable Client Authentication with Apache Kafka.
Before you begin
- Ensure that the ports that are used by the Kafka server are not blocked by a firewall.
- To enable client authentication between the Kafka consumers (QRadar) and a Kafka brokers, a key and certificate for each broker and client in the cluster must be generated. The certificates also need to be signed by a certificate authority (CA).
About this task
In the following steps, you generate a CA, sign the client and broker certificates with it, and add it to the client and broker truststores. You also generate the keys and certificates by using the Java keytool and OpenSSL. Alternatively, an external CA can be used along with multiple CAs, one for signing broker certificates and another for client certificates.
Procedure
-
Generate the truststore, keystore, private key, and CA certificate. Note: Replace PASSWORD, VALIDITY, SERVER_ALIAS and CLIENT_ALIAS in the following commands with appropriate values.
- Generate Server keystore. Note:
The common name (CN) of the broker certificates must match the fully qualified domain name (FQDN) of the server/host. The Kafka Consumer client that is used by QRadar compares the CN with the DNS domain name to ensure that it is connecting to the correct broker instead of a malicious one. Make sure to enter the FQDN for the CN/First and Last name value when you generate the Server keystore.
keytool -keystore kafka.server.keystore.jks -alias SERVER_ALIAS -validity VALIDITY -genkey
- Example
-
keytool -keystore kafka.server.keystore.jks -alias server.hostname -validity 365 -genkey
- Generate CA Certificate. Note:
This CA certificate can be used to sign all broker and client certificates.
openssl req -new -x509 -keyout ca-key -out ca-cert -days VALIDITY
- Example
-
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
- Create Server truststore and import CA Certificate.
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert
- Create Client truststore and import CA Certificate.
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert
- Generate a Server Certificate and sign it using the CA.
keytool -keystore kafka.server.keystore.jks -alias SERVER_ALIAS -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days VALIDITY -CAcreateserial
- Example
-
keytool -keystore kafka.server.keystore.jks -alias server.hostname -certreq -file cert-file
-
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial
- Import CA Certificate into the Server keystore.
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert
- Import Signed Server Certificate to the Server keystore.
keytool -keystore kafka.server.keystore.jks -alias SERVER_ALIAS -import -file cert-signed
- Example
-
keytool -keystore kafka.server.keystore.jks -alias server.hostname -import -file cert-signed
- Export the Server Certificate into the binary DER file. Note: The
keytool -exportcert
command uses the DER format by default. Place the certificate in the trusted_certificates/ directory of any EP that communicates with Kafka. You need the server certificate for every bootstrap server that you use in the configuration. Otherwise, QRadar rejects the TLS handshake with the server.keytool -exportcert -keystore kafka.server.keystore.jks -alias SERVER_ALIAS -file SEVER_ALIAS.der
- Example
-
keytool -exportcert -keystore kafka.server.keystore.jks -alias server.hostname -file server.hostname.der
- Generate a Client keystore.
keytool -keystore kafka.client.keystore.jks -alias CLIENT_ALIAS -validity VALIDITY -genkey
- Example
-
keytool -keystore kafka.client.keystore.jks -alias client.hostname -validity 365 -genkey
- Generate a Client Certificate and sign it using the CA.
keytool -keystore kafka.client.keystore.jks -alias CLIENT_ALIAS -certreq -file client-cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in client-cert-file -out client-cert-signed -days VALIDITY -CAcreateserial
- Example
-
keytool -keystore kafka.client.keystore.jks -alias client.hostname -certreq -file client-cert-file
-
openssl x509 -req -CA ca-cert -CAkey ca-key -in client-cert-file -out client-cert-signed -days 365 -CAcreateserial
- Import CA Certificate into the Client keystore.
keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert
- Import Signed Client Certificate to the Client keystore.
keytool -keystore kafka.client.keystore.jks -alias CLIENT_ALIAS -import -file client-cert-signed
- Example
-
keytool -keystore kafka.client.keystore.jks -alias client.hostname -import -file client-cert-signed
- Copy Client keystore and truststore and to QRadar.
- Copy the kafka.client.keystore.jks and kafka.client.truststore.jks to /opt/qradar/conf/trusted_certificates/kafka/ on each of the Event processors that the log source is configured for.
- Copy the server certificates <filename>.der that were generated for each broker to /opt/qradar/conf/trusted_certificates/.
- Generate Server keystore.
-
Configure Kafka brokers for Client Authentication.
- Find the Socket Server Settings section.
- Complete 1 of the following options:
- If you are not using SASL Authentication, change
listeners=PLAINTEXT://:<port>
tolisteners=SSL://:<PORT>
and addsecurity.inter.broker.protocol=SSL
. - If you are using SASL Authentication, change
listeners=PLAINTEXT://:<port>
tolisteners=SASL_SSL://:<PORT>
and addsecurity.inter.broker.protocol=SASL_SSL
.
- If you are not using SASL Authentication, change
- Add the following properties to force encrypted communication between brokers and
between the brokers and clients. Adjust the paths, file names, and passwords as you need them. These
properties are the truststore and keystore of the server:
security.inter.broker.protocol=SSL
ssl.client.auth=required
ssl.keystore.location=/somefolder/kafka.server.keystore.jks
ssl.keystore.password=test1234
ssl.key.password=test1234
ssl.truststore.location=/somefolder/kafka.server.truststore.jks
ssl.truststore.password=test1234Important: Since the passwords are stored in plain text in the server.properties, it is advised that access to the file is restricted by way of file system permissions. - Restart the Kafka brokers that had their server.properties modified.