Streaming record and entity data changes (IBM Match 360)

Types of master data events that can be streamed

Downstream systems often need to synchronize their data with the most current master data provided by IBM Match 360. By using the event streaming capability, you can subscribe to data change events at the entity or record level.

IBM Match 360 is a live system. Any time a record is added, updated or deleted, a record change event is created. When the matching engine runs, record changes are included in the matchmaking process and can affect entities. Any time matching gets run, IBM Match 360 creates entity change events to capture newly created entities, changes to the membership of existing entities, or updates to an entity's composite attribute values.

Changes to entity data that trigger streaming events

The member records of a master data entity can change when:

• Matching gets run after a change in record data (add, update, or delete).
• Matching gets run after a data engineer updates the matching algorithm configuration.
• A data steward manually links or unlinks records.

The attribute values of a master data entity can change when:

• A data steward manually updates an entity attribute value.
• There are changes to the entity's member records' attributes that cause different values to be selected by attribute composition rules. For more information about attribute composition rules, see Defining attribute composition rules.

An entity can be deleted when it no longer has any member records.

Changes to record data that trigger streaming events

Record data changes when:

• Records are added to IBM Match 360.
• Records are updated.
• Records are deleted.

If you create a streaming subscription for entity change events, the underlying record change events are also included in the stream. However, there are some scenarios where you might want to stream only record data, such as:

• If your custom data model includes record types that aren't associated with any entity types.
• If your entity streaming subscription includes a source level filter to include or exclude certain entity types.
• If you want to process record change events separately from entity change events by sending them to a different Kafka topic.

Configuring master data event streaming

Configure a new streaming subscription to start synchronizing your master data entities and records with downstream systems.

The master data event streaming capability supports the following connection types and security types.

To enable the IBM Match 360 master data event streaming capability:

1. Create a Cloud Pak for Data Apache Kafka connection asset in a project or catalog:

   a. Create a project or catalog to handle all of your connection assets. For information about creating a project, see Creating a project. For information about creating a catalog, see Creating a catalog.

   b. In your project or catalog, go to the Manage tab. In the General section, copy the project ID a project or catalog ID for a catalog. You'll need this ID to use as the container ID when creating your streaming subscription.

   c. Go to the Assets tab, and click New asset > Connection.

   d. From the list of connectors, click Apache Kafka, then click Select.

   e. Enter the Kafka connection information such as a name, description, and target Kafka server host name. Ensure that the following configuration items are set correctly:

   • Set the Credentials option to Shared.
   • Disable the Mask sensitive credentials retrieved through API calls option.
   • Select a supported connection type and security type for your connection.

2. Get the Apache Kafka connection asset ID and your project or catalog ID (container ID). You need these IDs to use as input when creating a master data event streaming subscription. There are two ways you can get these IDs:

   • From the asset URL: Open the Apache Kafka connection asset that you created your project or catalog. Copy the asset ID and container ID from the URL bar in your browser. Refer to the following example: https://cpd-namespace.apps.samplecp4d.cp.example.com/connections/<ASSET ID>?project_id=<CONTAINER ID>&context=icp4data

   • By using the connection API: Run the following API curl command:

     curl --location --request GET 'https://api.dataplatform.cloud.ibm.com/v2/connections?limit=100&entity.datasource_type=f13bc9b7-4a46-48f4-99c3-01d943334ba7&project_id=xxx&userfs=false' --header 'Accept: application/json' --header 'Authorization: Bearer xxx’
     

3. Use the IBM Match 360 API to create your master data streaming subscription. Use the following API methods in the model-ms microservice to create, update, or delete a subscription:

• Create method: GET /mdm/v1/event_subscription
• Update method: PUT /mdm/v1/event_subscription
• Delete method: DELETE /mdm/v1/event_subscription

For information about using the IBM Match 360 API, see IBM Match 360 API reference documentation.

Example event_subscription payload:

   {
  "filter": ["person_entity"],
  "event_type": "entity_change",
  "created_user": "user123",
  "last_update_user": "user123",
  "stream_connection": {
    "stream_type": "Kafka",
    "asset_scope": "Project",
    "topic": "PersonEntityTopic",
    "asset_id": "<ASSET-ID>",
    "container_id": "<CONTAINER-ID>"
  },
  "subscription_description": "Create PersonEntityRecordSub event  subscription SASL SSL EventStream",
  "subscription_name": "PersonEntityRecordSubSASLSSLEventStream",
  "active": true,
  "created_date": "1680297619428",
  "last_update_date": "1680297619428"
}

Enabling logging for master data event streaming

To enable WebSphere Liberty Profile logging for the IBM Match 360 event streaming capability:

1. From the OpenShift console, go to Administration > CustomResourceDefinition. Search for and select MasterDataManagement (CRD).

2. Select the Instance tab, then select mdm-cr and open the YAML tab.

3. To enable logging parameters, add the following lines to the mdm-cr YAML:

   wlp:
    logging:
      trace:
        specification: "com.ibm.mdmx.common.events.*=all:com.ibm.entity.matching.operational.core.streaming.*=all"
   

4. Click Save, then reload the page to ensure that the parameters were added correctly.

   After enabling logging, it takes some time for the mdm-cr to get resolved. Wait for the Cloud Pak for Data instance to show as enabled.

5. Review the logs to check whether IBM Match 360 successfully sends event messages to your Kafka endpoint.

Learn more

• Configuring event streaming in IBM Match 360
• IBM Match 360 API reference documentation
• Apache Kafka connection
• Adding data from a connection to a project

Parent topic: Configuring master data