Configuring InfoSphere MDM to enable ongoing synchronization

To enable ongoing synchronization between IBM® MDM Publisher and the MDM database, you must set up Apache Kafka and Eclipse Jetty Server within your InfoSphere® MDM installation.

1. Install InfoSphere MDM 11.6.0.11 or above. Apache Kafka, the Kafka Processor, and Jetty Server are all installed as part of the InfoSphere MDM installation.

   After installation, you can enable InfoSphere MDM 11.6.0.11 to support more recently released versions of Kafka than the one included in the base InfoSphere MDM installation. To enable support for Kafka 2.6, 2.7, or 2.8, complete some important configuration changes. For more information, see Configuring InfoSphere MDM to support Apache Kafka 2.6, 2.7, or 2.8.

   Important: If you deploy InfoSphere MDM Version 11.6.0.11 or higher on Docker, OpenShift or Kubernetes containers, then it is already configured to enable ongoing synchronization and you do not need to complete the remaining steps.

   If you install InfoSphere MDM using traditional installation methods, then additional configuration is required. Proceed with the next step.

   • For information about installing InfoSphere MDM using containers, see Installing InfoSphere MDM (accelerated deployment).
   • For information about installing InfoSphere MDM using traditional methods, see Installing InfoSphere MDM (standard deployment)
2. Run the madconfig target configure_mdm_publisher:

   cd <MDM_INSTALL_PATH>/mds/scripts
   ./madconfig.sh configure_mdm_publisher

   The script prompts you to provide the details of your InfoSphere MDM, MDM database, and Kafka server.
   When the madconfig script completes, it restarts the InfoSphere MDM application server to apply the changes it made.
3. On the machine where Kafka is installed, go to <KAFKA_INSTALL_HOME>/bin and run the following command:

   ps -ef | grep zookeeper.properties ps -ef | grep server.properties

4. Start Apache Zookeeper and Apache Kafka Server.
   1. Start Zookeeper:

      ./zookeeper-server-start.sh ../config/zookeeper.properties

   2. Set the KAFKA_OPTS environment variable to point to the path of kakfa_server_jaas.conf.

       export KAFKA_PLAIN_PARAMS="-Djava.security.auth.login.config=/kafkasecurity/kafka_server_jaas.conf"
       export KAFKA_OPTS="$KAFKA_PLAIN_PARAMS $KAFKA_OPTS"

   3. Start Kafka:

      ./kafka-server-start.sh ../config/server.properties

5. Create Kafka Topics using the default script:

   ./<MDM_INSTALL_HOME>/KafkaProcessor/bin/createTopics_Sample.sh

6. Start the Kafka Stream Processor:

   ./<MDM_INSTALL_HOME>/KafkaProcessor/bin/runKafkaStreams.sh ../properties/MdmRealTimeSync_v1.2.xml

7. Optional: By default, Apache Kafka is set up as a single broker. Optionally, you can complete the following additional steps to configure Kafka brokers in a cluster.
   1. In each Kafka broker node, edit the following properties in the ./<MDM_INSTALL_HOME>/kafka<version>/config/server.properties file.
      • broker.id=<ID> (The ID must be unique to each node.)
      • listeners=PLAINTEXT://localhost:9092
      • zookeeper.connect=:<zookeeper_port> (All brokers should point to the same Zookeeper port.)
      • log.dirs=/tmp/kafka-logs
   2. Start Zookeeper and all the broker nodes:

      ./<MDM_INSTALL_HOME>/kafka<version>/bin/zookeeper-server-start.sh config/zookeeper.properties
      ./<MDM_INSTALL_HOME>/kafka<version>/bin/kafka-server-start.sh config/server.properties

   3. In a cluster deployment, recreate the Kafka Topics with multiple replication factor and partitions, according to your deployment needs.
      For example:

      bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic <TOPIC_NAME>

   4. On the Jetty Server instance, open the properties file <MDM_INSTALL_HOME>/KafkaProcessor/properties/ and edit the property kafkaconnect.bootstrap.servers to include all Kafka broker nodes:

      kafkaconnect.bootstrap.servers=<KAFKA_HOSTNAME_01>:<KAFKA_BOOTSTRAP_PORT_01>,<KAFKA_HOSTNAME_02>:<KAFKA_BOOTSTRAP_PORT_02>

   5. In the IBM WebSphere® Application Server Integrated Solutions console (admin console), edit the bootstrap.servers property to enable InfoSphere MDM to publish messages in the Kafka cluster:
      1. Go to Resource > Resource environment > Resource environment entries.
      2. Select the custom properties entry.
      3. Select bootstrap.servers and update the value to include all Kafka broker nodes:

         <KAFKA_HOSTNAME_01>:<KAFKA_BOOTSTRAP_PORT_01>,<KAFKA_HOSTNAME_02>:<KAFKA_BOOTSTRAP_PORT_02>

      4. Restart InfoSphere MDM and Jetty Server.
8. If your instance of InfoSphere MDM is running on Docker containers or Kubernetes pods, complete the following additional steps:
   1. In the db2_container, turn off the bpmnotificationtype option:

      update bpmnotificationtype set expiry_dt=current_timestamp, last_update_dt=current_timestamp

   2. Restart the mdm_container:

      ./stopServer.sh <server_name>
      ./startServer.sh <server_name>