Overview of the Kafka batch processor

Use the Kafka batch stream processor to synchronize MDM data in large (or small) batches. Once you connect to the MDM database, you can synchronize MDM data in batch mode.

The batch stream processor works by following a two stage process:
  1. The Kafka database connector reads the primary keys for each entity matching specified search criteria.
  2. The batch processor collects the entity IDs and processes the entity for further transformation and persistence to one or more downstream systems.
Figure 1. Kafka batch process flow
Kafka batch flow