Specifying a custom classpath for Kafka client libraries

You can specify a custom classpath for Kafka client libraries if you need to use a different version of Kafka client or add more libraries to be executed by your Kafka client, such as OAuth and Interceptor handlers.

1. Create the kafka-client-classloader.cp file in the CDC_Kafka_instance_directory/conf directory.
2. Open the file and provide a custom classpath. The format must match your platform. For example, on Linux you need to provide a colon-separated list:

   library1.jar:library2.jar:...:libraryX.jar

   Restart your CDC Replication Engine for Kafka instance.

   Example:

   Enabling OAuth libraries for IBM event stream

   1. Build the necessary Kafka client libraries and OAuth handler by using CDC's Java version or earlier. For more details, see IBM event stream sample with Java 8.
   2. Modify kafkaconsumer.properties and kafkaproducer.properties in the instance conf directory of CDC to include OAuth connection options for the Kafka cluster. Ensure that these parameters are tested by using Apache Kafka consumer or producer.
      Example:

      cat <CDC instance dir>/conf/kafkaconsumer.properties 
      
      bootstrap.servers=<bootstrap servers>
      sasl.mechanism=OAUTHBEARER
      sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required grant_type="urn:ibm:params:oauth:grant-type:apikey" apikey="<api key>";
      sasl.login.callback.handler.class=com.ibm.eventstreams.oauth.client.IAMOAuthBearerLoginCallbackHandler
      sasl.oauthbearer.token.endpoint.url=https://iam.cloud.ibm.com/identity/token
      sasl.oauthbearer.jwks.endpoint.url=https://iam.cloud.ibm.com/identity/keys
      security.protocol=SASL_SSL
      ssl.protocol=TLSv1.2
      ssl.enabled.protocols=TLSv1.2
      ssl.endpoint.identification.algorithm=HTTPS

      Use the same connection properties for kafkaproducer.properties as well.
   3. In the instance conf directory of CDC, create a file that is named kafka-client-classloader.cp. Include all OAuth handler and kafka-client libraries/dependencies in it. Full path names are separated by semicolons.
      Example:

      cat <CDC instance dir>/conf/kafka-client-classloader.cp
      <path>/jackson-annotations-2.14.2.jar:<path>/jackson-databind-2.14.2.jar:<path>kafka-clients-3.3.2.jar:<path>/oauth-client-0.1.2.jar:<path>/slf4j-simple-1.7.36.jar:<path>/zstd-jni-1.5.2-1.jar:<path>/jackson-core-2.14.2.jar:<path>/jose4j-0.9.3.jar:<path>/lz4-java-1.8.0.jar:<path>/slf4j-api-1.7.36.jar:<path>/snappy-java-1.1.8.4.jar

   4. If you enabled a KCOP to write audit records in Avro format (see, Enabling a KCOP to write audit records in Avro format), update the kcop.properties file to include schema.registry.url and all serializer.property connection arguments that start with schema.registry.property. If you include schema registry and Kafka client libraries in the instance conf directory user-classloader.cp file, then also include Kafka client libraries and additional libraries in user-classloader.cp rather than kafka-client-classloader.cp.
      Example:

      cat <path>/kcop.properties
      audit.jcfs=ENTTYP,CCID
      before.image.prefix=B_
      before.update.record.mode=ALWAYS
      schema.registry.url=<schema registry url>
      serializer.property.basic.auth.credentials.source=USER_INFO
      serializer.property.schema.registry.basic.auth.user.info=<user>:<password>

      Note:

      To pass serializer properties, use serializer.property.propertyName. For more details, see Using IBM Event Streams Version 10.

      To add OAuth libraries to connect to the schema registry for Avro KCOP, see Specifying a custom classpath for KCOPs.

3. Restart your CDC Replication Engine for Kafka instance.