Configuring event streaming in IBM Match 360

Configure and tune the master data event streaming capabilities of IBM Match 360.

About this task

By using the event streaming capability, you can stream record and entity data changes from IBM Match 360 through an Apache Kafka connection to ensure that your users and systems always have the most up-to-date master data. For more information about using the event streaming capability, see Streaming record and entity data changes.

Roles and permissions
Cluster administrator To complete this task, you must have the cluster administrator role.

All IBM Match 360 event streaming configuration is defined in a configmap named mdm-data-streaming-configmap. The mdm-data-streaming-configmap has usable default values. Ideally, you can use the default values in your deployment without any problems. However, depending on your specific deployment requirements, you might want to fine tune properties such as the streaming message size or configure Apache Kafka properties such as timeout values, retry counts, buffer sizes, or others. To achieve this level of customization, a cluster administrator must modify the configmap.

The configmap includes different sections of properties:
  • connection_properties - Connection properties represent additional Kafka Producer properties. You can include any standard Kafka Producer properties in this section of the configmap. The following producer properties are already defined: value.serializer, key.serializer, enable.idempotence, acks, max.in.flight.requests.per.connection, delivery.timeout.ms, and retries.

    Ignore the transformations properties. These properties are used internally to transform the properties from the Kafka Connector to the Apache Kafka connection property.

  • standard_properties - These properties enable you to customize streaming behavior by defining message size and controlling reconciliation behavior.
  • recon_rmq_properties - These properties enable you to configure the properties of the internal RabbitMQ staging queue. This section has two subsections: rmq_general_properties and rmq_queue_properties
    • In rmq_general_properties, you can define the properties of the RabbitMQ queue. Some standard properties such as x-max-length, x-overflow, x-consumer-timeout, and others are already predefined here. It is not recommended to update these settings unless you are certain of the change.
    • In rmq_general_properties, you can customize the properties of the message reconciliation queue.

For more information and guidance about event streaming message reconciliation, see IBM Match 360 event streaming message reconciliation.

To control message sizes, wait times, and the message reconciliation queue, you can edit the following properties.
Table 1. Editable ConfigMap properties to control message sizes, wait times, and the reconciliation queue
Property Default value Description
standard_properties: waitTime 300 The time interval in seconds before reconciliation attempts are restarted.
standard_properties: reconEnabled true Boolean. If true, reconciliation is enabled for the event streaming feature.
standard_properties: writeFirstHandleFailureLater true Boolean. Determines the reconciliation mode:
  • true - Messages are staged first and then published. In this pattern the message is always sent to internal RabbitMQ staging before being publishing to the external Kafka.
  • false - Messages are processed in the same order that they were produced.

Even if this property is set to true, the message order is almost in the same order as it is produced, but is not guaranteed to be exactly the same. For example, if a failure causes the message to be requeued, the order of those messages can change during reprocessing. Also, the order is not guaranteed if there are multiple consumers (consumerCount >1).
standard_properties: includeRecordComposition 3 Defines the message structure for entity events. The value of this property determines whether "before" or "current" information is included in the message.
Valid values:
  • 0 - remove both current and before
  • 1 - remove only current
  • 2 - remove only before
  • 3 - include both current and before
standard_properties: maxRecordSizePerEntity 100 When the value of includeRecordComposition is set to any value other than 0, the entity count in "before" or "current" is limited by the value of this property.
standard_properties: serializerType JSON The message serializer type. Supported values areJSON or AVRO.

Using the AVRO binary compressible format can help to reduce message size.
rmq_general_properties: reconRetryCount 5 The maximum number of times the reconciliation process will attempt to republish a message.
rmq_general_properties: reconRetryInterval 120000 The interval of time in seconds between reconciliation attempts.
rmq_general_properties: consumerCount 2 The total number of consumers for the RabbitMQ queue.

Procedure

To configure the master data event streaming capabilities of IBM Match 360, update mdm-data-streaming-configmap:

  1. Open the OpenShift® Console and click Workloads > ConfigMaps.
  2. Search for and open mdm-data-streaming-configmap. Open the YAML tab to edit the configmap.
  3. Modify the configuration properties as required.
    See table 1 for details of the editable properties.
  4. Save the edited configmap YAML and exit the console.
  5. Restart the IBM Match 360 data-ms and model-ms pods to enable the new configuration to take effect.
  6. If you edited any RabbitMQ properties, delete and restart the message queue to see your changes reflected.