To enable ongoing synchronization between IBM® MDM Publisher
and the MDM database, you must set up Apache Kafka and Eclipse Jetty Server within your InfoSphere®
MDM
installation.
Procedure
-
Install InfoSphere
MDM
11.6.0.11 or above. Apache Kafka, the Kafka Processor, and Jetty Server are all installed as part of
the InfoSphere
MDM
installation.
After installation, you can enable InfoSphere MDM 11.6.0.11 to
support more recently released versions of Kafka than the one included in the base InfoSphere MDM installation.
To enable support for Kafka 2.6, 2.7, or 2.8, complete some important configuration changes. For
more information, see Configuring InfoSphere MDM to support Apache Kafka 2.6, 2.7, or 2.8.
Important: If you deploy
InfoSphere
MDM
Version 11.6.0.11 or higher on Docker, OpenShift or Kubernetes containers, then it is already
configured to enable ongoing synchronization and you do not need to complete the remaining steps.
If you install InfoSphere
MDM
using traditional installation methods, then additional configuration is required. Proceed with the
next step.
- Run the madconfig target
configure_mdm_publisher
:
cd <MDM_INSTALL_PATH>/mds/scripts
./madconfig.sh configure_mdm_publisher
The
script prompts you to provide the details of your
InfoSphere
MDM,
MDM database, and Kafka server.
When the madconfig script completes, it restarts
the InfoSphere
MDM
application server to apply the changes it made.
- On the machine where Kafka is installed, go to
<KAFKA_INSTALL_HOME>/bin and run the following
command:
ps -ef | grep zookeeper.properties ps -ef | grep server.properties
- Start Apache Zookeeper and Apache Kafka Server.
- Start Zookeeper:
./zookeeper-server-start.sh ../config/zookeeper.properties
- Set the
KAFKA_OPTS
environment variable to point to the path of
kakfa_server_jaas.conf.
export KAFKA_PLAIN_PARAMS="-Djava.security.auth.login.config=/kafkasecurity/kafka_server_jaas.conf"
export KAFKA_OPTS="$KAFKA_PLAIN_PARAMS $KAFKA_OPTS"
- Start Kafka:
./kafka-server-start.sh ../config/server.properties
- Create Kafka Topics using the default script:
./<MDM_INSTALL_HOME>/KafkaProcessor/bin/createTopics_Sample.sh
- Start the Kafka Stream Processor:
./<MDM_INSTALL_HOME>/KafkaProcessor/bin/runKafkaStreams.sh ../properties/MdmRealTimeSync_v1.2.xml
- Optional: By default, Apache Kafka is set up as a single broker. Optionally,
you can complete the following additional steps to configure Kafka brokers in a cluster.
- In each Kafka broker node, edit the following properties in the
./<MDM_INSTALL_HOME>/kafka<version>/config/server.properties
file.
broker.id=<ID>
(The ID must be unique to each node.)
listeners=PLAINTEXT://localhost:9092
zookeeper.connect=:<zookeeper_port>
(All brokers should
point to the same Zookeeper port.)
log.dirs=/tmp/kafka-logs
- Start Zookeeper and all the broker nodes:
./<MDM_INSTALL_HOME>/kafka<version>/bin/zookeeper-server-start.sh config/zookeeper.properties
./<MDM_INSTALL_HOME>/kafka<version>/bin/kafka-server-start.sh config/server.properties
- In a cluster deployment, recreate the Kafka Topics with multiple replication factor
and partitions, according to your deployment needs.
For
example:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic <TOPIC_NAME>
- On the Jetty Server instance, open the properties file
<MDM_INSTALL_HOME>/KafkaProcessor/properties/ and edit
the property
kafkaconnect.bootstrap.servers
to include all Kafka broker
nodes:
kafkaconnect.bootstrap.servers=<KAFKA_HOSTNAME_01>:<KAFKA_BOOTSTRAP_PORT_01>,<KAFKA_HOSTNAME_02>:<KAFKA_BOOTSTRAP_PORT_02>
- In the IBM WebSphere® Application Server Integrated Solutions console (admin console), edit the
bootstrap.servers
property to enable InfoSphere
MDM
to publish messages in the Kafka cluster:
- Go to .
- Select the custom properties entry.
- Select bootstrap.servers and update the value to include all Kafka broker
nodes:
<KAFKA_HOSTNAME_01>:<KAFKA_BOOTSTRAP_PORT_01>,<KAFKA_HOSTNAME_02>:<KAFKA_BOOTSTRAP_PORT_02>
- Restart InfoSphere
MDM
and Jetty Server.
- If your instance of InfoSphere
MDM
is running on Docker containers or Kubernetes pods, complete the following additional steps:
- In the
db2_container
, turn off the
bpmnotificationtype
option:
update bpmnotificationtype set expiry_dt=current_timestamp, last_update_dt=current_timestamp
- Restart the
mdm_container
:
./stopServer.sh <server_name>
./startServer.sh <server_name>
Results
When you have completed these steps, you can set up a source endpoint in the IBM MDM Publisher user
interface,
with the configuration of Jetty Server Host
pointing to the current host and
Jetty Server Port
pointing to port 4070
by default.
What to do next
To complete the setup, ensure that the Apache Kafka configuration steps documented in the
following topics are done: