Configuring Kafka processors and scripts

This procedure configures the Kafka processors and scripts to enable synchronization between Elasticsearch and the MDM database.

Procedure

  1. Browse to the folder MDM_INSTALL_HOME\KafkaProcessor\properties.
  2. Open kafkaruntimeForES.properties and set values for the following properties:
    • kafkaconnect.bootstrap.servers
    • elasticsearch.hostname
    • elasticsearch.port
    • elasticsearch.user
    • elasticsearch.password
      Note: The elasticsearch.user and elasticsearch.password credentials are created with X-Pack plugin's setup-passwords script. For more information, see Setting up the X-Pack plugin to secure Elasticsearch.
    • elasticsearch.ssl_enabled
    • mdm.hostname
    • mdm.port
    • mdm.user
    • mdm.password
  3. Open kafkabatchForES.properties and set values for the following properties:
    • kafkaconnect.bootstrap.servers
    • elasticsearch.hostname
    • elasticsearch.port
    • elasticsearch.user
    • elasticsearch.password
      Note: The elasticsearch.user and elasticsearch.password credentials are created with X-Pack plugin's setup-passwords script. For more information, see Setting up the X-Pack plugin to secure Elasticsearch.
    • elasticsearch.ssl_enabled
    • mdm.hostname
    • mdm.port
    • mdm.user
    • mdm.password
  4. Open kafkadbconnect.properties and set values for the MDM database connection parameters. If SSL is enabled on the MDM database, set values for the SSL related properties.
  5. Open DBConnectConfig.properties and set values for the following properties:
    • kafkaconnect.hostname
    • kafkaconnect.portnum
  6. Browse to the folder MDM_INSTALL_HOME\KafkaProcessor\bin.
  7. Open createTopics_Sample.bat or createTopics_Sample.sh and set values for the following properties:
    • KAFKA_HOME
    • hostname
    • port
  8. Open startConnectorDistributed.bat or startConnectorDistributed.sh and set values for the following properties:
    • KAFKA_HOME
    • DB_DRIVER_JARPATH
  9. Open runDBConnectorDistributed.bat or runDBConnectorDistributed.sh and set a value for JAVA_HOME.
  10. Open runKafkaStreamsForES.bat or runKafkaStreamsForES.sh and set a value for JAVA_HOME.
  11. Open runKafkaBatchStreamsForES.bat or runKafkaBatchStreamsForES.sh and set a value for JAVA_HOME.
  12. Start Apache Zookeeper and the Kafka server:
    1. Browse to MDM_INSTALL_HOME\kafka_2.11.0.10.2.1\bin\windows.
    2. Open a command prompt and run 
      zookeeper-server-start.bat ../../config/zookeeper.properties
    3. Open a different command prompt and run 
      kakfa-server-start.bat ../../config/server.properties
  13. Browse to MDM_INSTALL_HOME\KafkaProcessor\bin.
  14. Run createTopics_Sample.bat on Windows or createTopics_Sample.sh on Linux or UNIX.
    This script creates the Kafka topics and is a one-time activity.