Configuring Apache Kafka to enable Client Authentication

This task discusses how to enable Client Authentication with Apache Kafka.

Before you begin

  1. Ensure that the ports that are used by the Kafka server are not blocked by a firewall.
  2. To enable client authentication between the Kafka consumers (QRadar) and a Kafka brokers, a key and certificate for each broker and client in the cluster must be generated. The certificates also need to be signed by a certificate authority (CA).

About this task

In the following steps, you generate a CA, sign the client and broker certificates with it, and add it to the client and broker truststores. You also generate the keys and certificates by using the Java keytool and OpenSSL. Alternatively, an external CA can be used along with multiple CAs, one for signing broker certificates and another for client certificates.

Procedure

  1. Generate the truststore, keystore, private key, and CA certificate.
    Note: Replace PASSWORD, VALIDITY, SERVER_ALIAS and CLIENT_ALIAS in the following commands with appropriate values.
    1. Generate Server keystore.
      Note:

      The common name (CN) of the broker certificates must match the fully qualified domain name (FQDN) of the server/host. The Kafka Consumer client that is used by QRadar compares the CN with the DNS domain name to ensure that it is connecting to the correct broker instead of a malicious one. Make sure to enter the FQDN for the CN/First and Last name value when you generate the Server keystore.

      keytool -keystore kafka.server.keystore.jks -alias SERVER_ALIAS
       -validity VALIDITY -genkey
      Example
      keytool -keystore kafka.server.keystore.jks -alias server.hostname
       -validity 365 -genkey
    2. Generate CA Certificate.
      Note:

      This CA certificate can be used to sign all broker and client certificates.

      openssl req -new -x509 -keyout ca-key -out ca-cert -days VALIDITY
      Example
      openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
    3. Create Server truststore and import CA Certificate.
      keytool -keystore kafka.server.truststore.jks -alias CARoot
       -import -file ca-cert
    4. Create Client truststore and import CA Certificate.
      keytool -keystore kafka.client.truststore.jks -alias CARoot
       -import -file ca-cert
    5. Generate a Server Certificate and sign it using the CA.
      keytool -keystore kafka.server.keystore.jks -alias SERVER_ALIAS
       -certreq -file cert-file
      openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out
       cert-signed -days VALIDITY -CAcreateserial
      Example
      keytool -keystore kafka.server.keystore.jks -alias server.hostname 
      -certreq -file cert-file 
      openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out
       cert-signed -days 365 -CAcreateserial 
    6. Import CA Certificate into the Server keystore.
      keytool -keystore kafka.server.keystore.jks -alias CARoot
       -import -file ca-cert
    7. Import Signed Server Certificate to the Server keystore.
      keytool -keystore kafka.server.keystore.jks -alias SERVER_ALIAS -import
       -file cert-signed
      Example
      keytool -keystore kafka.server.keystore.jks -alias server.hostname
       -import -file cert-signed
    8. Export the Server Certificate into the binary DER file.
      Note: The keytool -exportcert command uses the DER format by default. Place the certificate in the trusted_certificates/ directory of any EP that communicates with Kafka. You need the server certificate for every bootstrap server that you use in the configuration. Otherwise, QRadar rejects the TLS handshake with the server.
      keytool -exportcert -keystore kafka.server.keystore.jks -alias
       SERVER_ALIAS -file SEVER_ALIAS.der
      Example
      keytool -exportcert -keystore kafka.server.keystore.jks -alias
       server.hostname -file server.hostname.der
    9. Generate a Client keystore.
      keytool -keystore kafka.client.keystore.jks -alias CLIENT_ALIAS
       -validity VALIDITY -genkey
      Example
      keytool -keystore kafka.client.keystore.jks -alias client.hostname
       -validity 365 -genkey
    10. Generate a Client Certificate and sign it using the CA.
      keytool -keystore kafka.client.keystore.jks -alias CLIENT_ALIAS
       -certreq -file client-cert-file
      openssl x509 -req -CA ca-cert -CAkey ca-key -in client-cert-file -out
       client-cert-signed -days VALIDITY -CAcreateserial 
      Example
      keytool -keystore kafka.client.keystore.jks -alias client.hostname
       -certreq -file client-cert-file 
      openssl x509 -req -CA ca-cert -CAkey ca-key -in client-cert-file
       -out client-cert-signed -days 365 -CAcreateserial
    11. Import CA Certificate into the Client keystore.
      keytool -keystore kafka.client.keystore.jks -alias CARoot
       -import -file ca-cert
    12. Import Signed Client Certificate to the Client keystore.
      keytool -keystore kafka.client.keystore.jks -alias CLIENT_ALIAS
       -import -file client-cert-signed
      Example
      keytool -keystore kafka.client.keystore.jks -alias client.hostname
       -import -file client-cert-signed
    13. Copy Client keystore and truststore and to QRadar.
      1. Copy the kafka.client.keystore.jks and kafka.client.truststore.jks to /opt/qradar/conf/trusted_certificates/kafka/ on each of the Event processors that the log source is configured for.
      2. Copy the server certificates <filename>.der that were generated for each broker to /opt/qradar/conf/trusted_certificates/.
  2. Configure Kafka brokers for Client Authentication.
    1. Find the Socket Server Settings section.
    2. Complete 1 of the following options:
      • If you are not using SASL Authentication, change listeners=PLAINTEXT://:<port> to listeners=SSL://:<PORT> and add security.inter.broker.protocol=SSL.
      • If you are using SASL Authentication, change listeners=PLAINTEXT://:<port> to listeners=SASL_SSL://:<PORT> and add security.inter.broker.protocol=SASL_SSL.
    3. Add the following properties to force encrypted communication between brokers and between the brokers and clients. Adjust the paths, file names, and passwords as you need them. These properties are the truststore and keystore of the server:

      security.inter.broker.protocol=SSL

      ssl.client.auth=required

      ssl.keystore.location=/somefolder/kafka.server.keystore.jks

      ssl.keystore.password=test1234

      ssl.key.password=test1234

      ssl.truststore.location=/somefolder/kafka.server.truststore.jks

      ssl.truststore.password=test1234
      Important: Since the passwords are stored in plain text in the server.properties, it is advised that access to the file is restricted by way of file system permissions.
    4. Restart the Kafka brokers that had their server.properties modified.