Interaction between the clustered watch folder and the external Kafka sink

Clustered watch folder supports sending watch events to an external Kafka sink.

Before the events are sent to the external Kafka sink, they are compressed by using the LZ4 algorithm. Make sure that the lz4 library is installed on the system from which the events are meant to be consumed. Each consumer library is different and may bundle LZ4 directly or have a preferred method of installation; therefore, to get the appropriate LZ4 installation instructions, consult the consumer library documentation.

To send watch events to an external Kafka sink, a minimum of two attributes must be present when you enable a clustered watch:
  • A list of accessible broker addresses with ports to access the external Kafka queue.
  • The topic name on the external Kafka queue where the clustered watch publishes events.
In addition to the two required attributes, authentication or authorization can also be specified. If authentication or authorization is not given when you enable a clustered watch, it is assumed that it is not needed. The following types of authentication or authorization are supported:
  • NONE: This is the default. It can also be specified by excluding any type of authentication configuration.
  • PLAINTEXT: Use Kafka plain text authentication. You must provide a PRODUCER_USERNAME and PRODUCER_PASSWORD with the authentication information for the producer to write to the external Kafka sink.
  • SASL: Use SASL-based authentication between the IBM Storage Scale cluster that hosts the clustered watch and the external Kafka sink. You must provide a PRODUCER_USERNAME and PRODUCER_PASSWORD with the authentication information for the producer to write to the external Kafka sink; and you must provide the specific mechanism to use:
    • SCRAM256
    • SCRAM512
  • SASL_SSL: Use SASL_SSL-based authentication between the IBM Storage Scale cluster that hosts the clustered watch and the external Kafka sink. You must provide a PRODUCER_USERNAME and PRODUCER_PASSWORD with the authentication information for the producer to write to the external Kafka sink; and you must provide the specific mechanism to use:
    • SCRAM256
    • SCRAM512
    For the producer to write to the external Kafka sink, you must also provide three extra parameters (the fourth one is optional) when you specify this type of authentication:
    • CA_CERT_LOCATION: Full path (including the actual file) to the location of the ca-cert file. This field is required.
    • CLIENT_PEM_CERT_LOCATION: Full path (including the actual file) to the location of the client certificate (.pem format) file. This field is required.
    • CLIENT_KEY_FILE_LOCATION: Full path to the location of the client key (client.key) file. This field is required.
    • CLIENT_KEY_FILE_PASSWORD: Key file password. This field is optional.
  • CERT: Use certificate-based authentication and encryption of data in flight between the IBM Storage Scale cluster hosting the clustered watch and the external Kafka sink. For the producer to write to the external Kafka sink, you must also provide three extra parameters (the fourth one is optional) when you specify this type of authentication:
    • CA_CERT_LOCATION: Full path (including the actual file) to the location of the ca-cert file. This field is required.
    • CLIENT_PEM_CERT_LOCATION: Full path (including the actual file) to the location of the client certificate (.pem format) file. This field is required.
    • CLIENT_KEY_FILE_LOCATION: Full path to the location of the client key (client.key) file. This field is required.
    • CLIENT_KEY_FILE_PASSWORD: Key file password. This field is optional.
For more information about specifying these parameters, see mmwatch command.
Note: For the producer to write to the external Kafka queue, the firewall ports must be open between the source IBM Storage Scale cluster and the external Kafka queue.
Note: All of these parameters are used in the --sink-auth-config flag of the mmwatch command. This parameter is optional. When it is used, you must pass a configuration file with specific parameters. For more information, see the following examples. The first example is of a clustered watch folder setup with SCRAM512 to the external Kafka sink.

SINK_AUTH_TYPE:SASL
SINK_AUTH_MECHANISM:SCRAM512 
PRODUCER_USERNAME:<will be found in external kafka config> 
PRODUCER_PASSWORD:<will be found in external kafka config>
The second example is of a CERT-based authentication setup between the IBM Storage Scale cluster and the external Kafka sink that the clustered watch folder uses.

SINK_AUTH_TYPE:CERT
CA_CERT_LOCATION:<path to certs>
CLIENT_PEM_CERT_LOCATION:<path to pem cert>
CLIENT_KEY_FILE_LOCATION:<path to key>
CLIENT_KEY_FILE_PASSWORD:<password from certificate setup>