Kafka (Source and Target) operators

Kafka is a distributed streaming platform that provides a publish and a subscribe messaging transport protocol using topics.

In a streams flow, the Kafka operator can be of type Source or of type Target.

• Kafka as a Source operator is a subscriber to a topic in the Kafka cluster and processes the stream of records produced to that topic.

• Kafka as a Target operator is a publisher to a topic in the Kafka cluster and publishes a stream of records to that topic.

Required parameters per security protocol

When creating a connection to Kafka, there are many different security configurations that can be used and each have their own set of requirements. The following is a list of supported configurations:

SASL_PLAINTEXT

• Brokers
• SASL mechanism
• Username
• Password

SASL_SSL

• Brokers
• SASL mechanism
• Username
• Password
• Cluster CA certificate (optional)

SSL

• Brokers
• Client SSL certificate
• Client SSL private key
• Cluster CA certificate (optional)

SASL

SASL can be used with PLAINTEXT (SASL_ PLAINTEXT) or SSL (SASL_SSL) as the transport layer using the security protocol:

• SASL_PLAINTEXT:
  • Simple username/password authentication
  • No SSL encryption.
• SASL_SSL supports the following SASL mechanisms:

  • SASL/PLAIN:
    Simple username/password authentication mechanism that is used with TLS for encryption.

  • SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512:
    Stores SCRAM credentials in ZooKeeper and is suitable for production use where ZooKeeper is secure and on a private network.

Learn more

• Apache Kafka concepts