Integrating the gateway with Kafka

• G_XML.props
• kafkaTransport.properties
• kafkaConnectionProperties.json
• kafkaClient.properties

1. Install/update the Message Bus Gateway. See Installing the gateway.
2. Edit the gateway configuration in the following gateway properties file:

   $OMNIHOME/gates/<arch>/G_XML.props

   Where <arch> is the architecture directory, for example linux2x86

3. Update the following property values with the appropriate path:

   Gate.XMLGateway.TransportFile		: '$OMNIHOME/java/conf/kafkaTransport.properties'
   Gate.XMLGateway.TransportType		: 'KAFKA'
   
   

4. Configure the Kafka transport properties.
   1. Edit the Kafka transport configuration in the following transport properties file:

      $OMNIHOME/java/conf/kafkaTransport.properties

   2. Update the following property value with the appropriate path:

      kafkaClientMode=PRODUCER
      connectionPropertiesFile=$OMNIHOME/java/conf/kafkaConnectionProperties.json
      

5. For descriptions of the Kafka transport properties, see the Kafka transport properties table.
6. Configure the Kafka connection properties.

   Kafka connection properties are defined in the kafkaConnectionProperties.json file. This file contains the following properties:

   {
       "zookeeper_client" : 
           {
               "target" : "",
               "properties" : "",
               "java_sys_props" : "",
               "topic_watch": true,
               "broker_watch": true
           },
       "brokers" : "",
       "topics": "",
       "kafka_client" : 
           {
               "properties" : "",
               "java_sys_props" : ""
           }
    }

   1. Within the sample configuration file supplied with the gateway, update the path to the ZooKeeper client properties file.

      "zookeeper_client" : 
        {
            "target" : "localhost:2181",
            "properties" : "<Path & File of zookeeper client properties file>",
            "java_sys_props" : "",
            "topic_watch": true,
            "broker_watch": true
        },

   2. Update the path to the Kafka client properties file:

        "brokers" : "PLAINTEXT://localhost:9092",
        "topics": "topicABC,topicXYZ",
        "kafka_client" : 
          {
            "properties" : "<full_omnihome_path>/java/conf/kafkaClient.properties",
            "java_sys_props" : ""	
          }

7. For descriptions of the Kafka connection properties, see the Kafka connection properties table.
8. Configure the Kafka client properties.
   1. Edit the Kafka client configuration in the following properties file:

      $OMNIHOME/java/conf/kafkaClient.properties

   2. Uncomment the common Kafka client properties and the producer specific properties in the kafkaClient.properties file.
   3. For details of the configuration required by the connection protocols supported by the Kafka producer, see Kafka configuration for different connection protocols.
   Note: The default setting for the Kafka producer’s key.serializer and value.serializer is org.apache.kafka.common.serialization.StringSerializer. The current Kafka Transport does not support any Kafka producer using a serializer class other than org.apache.kafka.common.serialization.StringSerializer. Specifying any other serializer class causes issues in the producer’s send operation.
9. Run the gateway using the following command:

   $OMNIHOME/bin/nco_g_xml -propsfile $OMNIHOME/etc/G_XML.props

   .
   Note: The gateway property file must be copied over from $OMNIHOME/gates/xml to $OMNIHOME/etc if it is going to be run from that directory.

1. Obtain the lz4 and snappy library jar file from Kafka installation directory: <kafka_installation_location>/libs

   The lz4 jar file is lz4-<version>.jar.

   The snappy jar file is snappy-java-<version>.jar.

2. Copy the jar files to: $OMNIHOME/java/jars

3. Add the full path of the jars file to the $CLASSPATH environment variable.

   For example:

   export CLASSPATH=$CLASSPATH:$OMNIHOME/java/jars/lz4-<version>.jar:$OMNIHOME/java/jars/snappy-java-<version>.jar

   Note: Gate.Java.ClassPath is assumed to be containing the $CLASSPATH token.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Log4j2 Configuration for Kafka Transport -->
<Configuration status="debug">
  <Appenders>
    <File name="LOGFILE" fileName="<log_filename.log>">
      <PatternLayout pattern="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>
    </File>
  </Appenders>
  <Loggers>
    <Root level="debug" >
      <!-- Increase the logging level if necessary  -->
      <AppenderRef ref="LOGFILE"/>
    </Root>
  </Loggers>
</Configuration>

• SASL_PLAINTEXT
• SASL_SSL