Interaction between clustered watch folder and the external Kafka sink

Clustered watch folder supports sending watch events to an external Kafka sink.

To send watch events to an external Kafka sink, a minimum of two attributes must be present when you enable a clustered watch:
  • A list of accessible broker addresses with ports to access the external Kafka queue.
  • The topic name on the external Kafka queue where the clustered watch publishes events.
In addition to the two required attributes, authentication or authorization can also be specified. If authentication or authorization is not given when you enable a clustered watch, it is assumed that it is not needed. The following types of authentication or authorization are supported:
  • NONE: This is the default. It can also be specified by excluding any type of authentication configuration.
  • PLAINTEXT: Use Kafka plain text authentication. You must provide a PRODUCER_USERNAME and PRODUCER_PASSWORD with the authentication information for the producer to write to the external Kafka sink.
  • SASL: Use SASL based authentication between the IBM Storage Scale cluster hosting the clustered watch and the external Kafka sink. You must provide a PRODUCER_USERNAME and PRODUCER_PASSWORD with the authentication information for the producer to write to the external Kafka sink and you must provide the specific mechanism to use:
    • SCRAM256
    • SCRAM512
  • CERT: Use certificate-based authentication and encryption of data in flight between the IBM Storage Scale cluster hosting the clustered watch and the external Kafka sink. You must provide an extra three (optional fourth) parameters when specifying this type of authentication for the producer to write to the external Kafka sink:
    • CA_CERT_LOCATION: Full path (including the actual file) to the location of the ca-cert file. This field is required.
    • CLIENT_PEM_CERT_LOCATION: Full path (including the actual file) to the location of the client certificate (.pem format) file. This field is required.
    • CLIENT_KEY_FILE_LOCATION: Full path to the location of the client key (client.key) file. This field is required.
    • CLIENT_KEY_FILE_PASSWORD: Key file password. This field is optional.
For more information about specifying these parameters, see mmwatch command.
Note: In order for the producer to write to the external Kafka queue, the firewall ports must be open between the source IBM Storage Scale cluster and the external Kafka queue.
Note: All of these parameters are used in the --sink-auth-config flag of the mmwatch command. This parameter is optional. When it is used, you must pass a configuration file with specific parameters. For more information, see the following examples. The first example is of a clustered watch folder setup with SCRAM512 to the external Kafka sink.

SINK_AUTH_TYPE:SASL
SINK_AUTH_MECHANISM:SCRAM512 
PRODUCER_USERNAME:<will be found in external kafka config> 
PRODUCER_PASSWORD:<will be found in external kafka config>
The second example is of a CERT-based authentication setup between the IBM Storage Scale cluster and the external Kafka sink that the clustered watch folder uses.

SINK_AUTH_TYPE:CERT
CA_CERT_LOCATION:<path to certs>
CLIENT_PEM_CERT_LOCATION:<path to pem cert>
CLIENT_KEY_FILE_LOCATION:<path to key>
CLIENT_KEY_FILE_PASSWORD:<password from certificate setup>