Connector: Apache Kafka

The Manta Event System - Apache Kafka connector provides users the option to route Manta events to their own Apache Kafka instance. Each event is sent as a single message to a configured Kafka topic. The Kafka message is defined as a key-value pair, where the value of the message is the Manta event in JSON format. The key of the message is user-configurable, as are most of the other message and Kafka connection properties.

Configuration

Users can configure multiple connections to Apache Kafka with different configurations.

  1. To create a new Kafka Event System connection, go to the Connections tab in Admin UI, click on Add Connection and select the Kafka Events Plugin option.

    A picture showing the optios for configuring new connection.

    note: You don’t need to restart the Admin UI service to apply the changes.

  2. The configuration is split into multiple sections. The first section, Required Configuration, contains properties that need to be provided in order to successfully initialize and use the Apache Kafka connector.

    1. First, the property Apache Kafka Connector Enabled (1) must be set to true for the connector to be initialized. If this property is *false*, all other configurations of this connector are ignored.

    2. Secondly, the connector requires a list of addresses of Kafka cluster bootstrap servers (2) in the format <HOST>:<PORT>.

    3. Finally, the connector requires the name of the topic (3) where Manta Events will be sent.

      A picture showing the filled Reguired configuration menu for Apache Kafka connector.

  3. In the section that follows Required Configuration, Basic Configuration, the user can configure the basic behavior of the Kafka connector. The section contains the following properties.

    1. Message key (1) is the key that will be used for Kafka messages with a Manta Event payload. The message key ensures the correct ordering of messages and determines which partition of the Kafka cluster the message is sent to. The default value is MANTA.

    2. Security protocol (2) is the property that defines which protocol should be used to secure the communication. The possible values are mapped to the possible values of the security.protocol property of the Kafka producer and they share their behavior; for example, if the protocol is configured to PLAINTEXT, any SSL configurations will be ignored.

    3. Client ID (3) is a string that the Manta Event message producer uses to identify itself to the Kafka cluster when making requests. The purpose of this ID is to be able to identify the source of requests in the logs of a Kafka server or when performing other monitoring activities. The default client ID of the producer is MANTA.

      No alt text provided

  4. If a more advanced configuration of the producer is required, head over to the Advanced tab of the configuration and change the properties according to your requirements. The advanced configuration properties are mapped to the properties as defined by Apache Kafka. Their behavior is exactly as documented in the full documentation for https://kafka.apache.org/documentation.html#producerconfigs .

  5. Restart the Manta Service Utility instance to make the changes valid (the connector is initialized while Manta Service Utility is starting).