IBM Infosphere Master Data Management Hybrid Solution: Technical Overview

Content

Enabling Hybrid:

In v-11 after a regular MDM install, the installer creates a foldernamed Hybrid in the install directory and it is this directory which is used to enable Hybrid MDM. The precise steps on how to install it are included in ReadMe.txt file but in general we configure the script by modifying the setupEnvVariables.sh file and then run it using hybridPostInstall.sh script. The installer does the following:

1. Prepare the physical side for hybrid by updating the configElement table to support hybrid (like setting /IBM/Hybrid/enabled to true), disabling the Standardization feature, disabling the Suspect Duplicate Processing feature, and disabling the Derived Data Synchronization feature, etc.

2. Prepare WebSphere for hybrid by creating objects like the default JMS queue (or MQ) to provide a secure platform for virtual and physical systems to exchange XML messages.

3. Deploy the default virtual MDM hybrid party model. It creates the default data models (attributes) for virtual and physical and maps them for persistence.
The default party model also prepares the virtual side for hybrid by setting up the event notification framework with hybrid defaults like when, what, and where to send XML notifications required for persistence.

In v-11, we can perform the third step of creating and deploying the virtual MDM model though the workbench by opening the 'MDM Configuration' perspective, clicking on the Navigator, 'New' > 'Configuration Project' > Give it a Name > select Hybrid Party > chose the server and click Finish. This will create the virtual hybrid party model with default configuration for event notification and you may deploy this to the server.

In future versions, all three of these work items may be moved to workbench and the hybridPostInstall script may be deprecated. Please refer to the information center for details on the higher version.


Configuration and Customization:

For persistence to work, the hybrid solution must have three components: attributes in virtual MDM, attributes in physical MDM, and a mapping file which tells the system how they map to each other. This way, when a virtual attribute is populated, the mapping file will indicate which corresponding physical attribute to update, and what formatting and transformation ought to take place before storing the value.

The Hybrid MDM project adds the following to the workspace:
1. Configuration Project (including the .imm file)
This project contains all the virtual components, configurations, algorithm, as well as specifics of event notification framework.

2. MDMSharedResources
Among other assets, this project contains the physical MDM schema (PhysicalMDM.xsd) .

3. MDMHybridMapping
This project contains the .map files that specifies the associations between virtual MDM attributes and physical MDM attributes along with any formatting that should occur before persisting the values. The .map mapping files are within the transform folder. You may either move the values as is or perform transform the data (via custom XSLT function) before saving the value in physical side.

In the virtual configuration imm file, we can also specify the event types which should trigger an XML notification, what information should be included in them, and which destination should these notifications be sent to. To add a destination to send XML notifications, open the .imm file, go to Events section 'Event Notification' tab, and click on 'Add'.
As part of the default party model, the WebSphere default jms queue is already configured and the corresponding events to listen for are specified at the bottom section; these include "Entity Create", "Entity Update", or "Entity Delete". The default handler name is MDMLink and to customize the event notification handler to conditionally persist entities based on your own logic by modifying an external rule (MDMLinkRule.java, registered as rule #210).

The information to send in the XML messages is controlled by the HybridPerson composite view and we can extend or edit the HybridCompositeView.java file in the MDM_INSTALL_HOME/Samples directory to alter any survivorship rules. There is a hybrid.properties file where you specify the entities to persist, corresponding composite views, and the database schema.
Additionally, we have the ability to create external rules to modify persistence behavior as per business need. There are two types of rules:
1. The hybrid MDM maintenance rules are used by the persistEntity transaction to perform operations on the data values before persisting them.
2. The hybrid MDM event handling rule is used by the hybrid MDM event handler component to perform persistence.

We also have a com.ibm.mdm.mds.event.manager.cfg file to configure parameters like whether to enable group processing (enableGroupProcessing), whether to suspend all event notifications if an unrecoverable error occurs (enableAutoSuspension), and the user account for running the event handlers (workManagerUserName and workManagerPassword).

You may use just the default data model or create your own attributes in virtual and physical side and then map them for persistence. Similarly, you may either use the default configuration and persistence behavior or modify any one or more elements. We highly recommend starting with the default hybrid party model as base for efficiency since a lot of the work is already done. Refer to the information center for details on all the above features .


Deploying the artifacts:

To deploy the configuration (including the .imm file) from within workbench, go to 'Master Data Management' > 'Deploy Configuration Project' and on the resultant pop-up, select the project and the corresponding WebSphere server. This will connect to the operational engine, temporarily suspend it, update the configuration, and resume the operational engine (no restart necessary).

In your workspace, you will have a <HybridProjectName>.cba file which is a ready-to-deploy version of the hybrid configuration project that optionally may also contains your custom code and the mapping (this is configured while creating the hybrid model with the following check box 'Add hybrid mapping project to composite').
You may upload one or more composite bundles (cba) to the enterprise bundle (eba) from either workbench or Websphere as shown below:

1. Go to the Server view within workbench, select the corresponding WebSphere server where you want to deploy, right click, and select 'Add and Remove', and add the desired cba to the server. Once completed, right click on the cba (being uploaded) in the server view and select 'Manage Extensions.' This will allow you to add the cba to the eba.

2. Right-click on cba project within workbench and select "Export" > "OSGI Composite Bundle (CBA)" > select a desired location and name for your target CBA file and click "Finish". Then open WebSphere Administration Console and navigate to Environment > 'OSGI bundle repositories' > 'Internal bundle repository' and click "New" > "Browse" > navigate to where you exported your CBA, select the CBA file name and Save. Once it’s finished, your CBA will show up under the Internal bundle repository page.
The cba is uploaded to WebSphere and now we need to add it to the eba. Navigate to Applications -> Application Types -> Business-level applications and click on 'MDM-operational-server-EBA-E001' > 'com.ibm.mdm.hub.server.app-E001_001.eba' > Click on "Extensions for this composition unit" under Additional Properties of MDM eba > click Add, check the cba from the list and click "Add" and save.

In either of the above cases, we need to update the runtime of the operational server to include this newer version of the cba. So whilst on the eba view discussed in pt#2, click on "Update latest deployment…" button and in the following page, you’ll be able to see your CBA in the list. Follow the prompts to deploy it to the eba.



WebSphere:

WebSphere facilitates a peer-to-peer messaging network where virtual MDM sends XML messages and physical MDM consumes it. There is no direct communication between the two and each client connects to WebSphere messaging service that provides a system for creating, sending, receiving, and reading messages. The sender does not need to know anything about the receiver, nor does the receiver need to know anything about the sender(they don't even have to available at the same time). By default we use a Java Message Service (JMS) queue for this communication. If you install v-11 and go through the steps for enabling hybrid, all required artifacts will be created automatically and no further information will be needed.

You may review this set-up by logging into WebSphere Admin Console > Resources > JMS. It is these destinations that we configure in workbench for event notification framework.
Using WebSphere tools, you may monitor this messaging in real time or troubleshoot any/all problems which may arise.

For comprehensive details, please refer to the WebSphere Application Server Information Center.


Data Flow:

In a hybrid solution whenever an interaction occurs, it is evaluated to check whether it meets the criteria specified in the workbench at the Events screen > Event Notification tab > 'Event Lists' section. If it matches, say an entity was created and that is one of the specified event types, then a row is added to the mpi_entoque_XY table where XY is the prefix for the entity.

The database is monitored by the event manager to detect new entoque entries to send messages for. After that an 'Event Work Manager' creates external event messages based on those events and passes the message to the event handler for publishing. The event work manager is configured in Workbench through Events > "Event Notification" screen and it will create messages based on the composite view which has been specified. Once the event has been detected and a message created, the event handler publishes the event message to the JMS Queue created within WebSphere.

The physical side of MDM then monitors the WebSphere queue to look for new messages. If there are new messages, then it will consume them and invoke the persistEntity API transaction. This interaction will use the .map mapping file to determine the physical counterparts for virtual attributes and whether certain data formatting or execution of external rules are required before saving them into the physical database. The change from the virtual side is thus manifested into the physical database.

Developers or administrators may also invoke the persistEntity API manually without the event notification workflow (information center has details on usage). When invoking this API, we have PersistenceMode as an optional parameter (acceptable values are 1 and 2) to specify level of persistence. Value of 1 corresponds to entity registration mode which persist only the cross-reference keys of the virtual entity (that is, the virtual entity ID and source member IDs associated with the entity). Value of 2 corresponds to full persistence mode which persist the single, configured composite view of the virtual entity, as well as the cross-reference keys.


Troubleshooting:

To diagnose issues, you may use any of the regular virtual MDM, physical MDM, or WebSphere troubleshooting methodology since the solution leverages all of these components. You may even trace or monitor the queue from WebSphere; you can see the messages in JMS queue in WAS Admin Console as they are sent before being consumed (removed after consumption). But an audit trail will not be saved by default, and you will need to install additional supporting WebSphere applications for that. Contact your WAS Admin for details.

Specific to hybrid MDM, we have a new logger named com.ibm.mdm.hybrid.component. This special logging may be enabled from WebSphere Application Server Administration Console. Log into the WAS Admin Console > Troubleshooting > Logs and trace and select the server where the operational server is deployed. Then under 'Diagnostic Trace' you will see two options, Runtime tab and Configuration tab. Changes made in Runtime take effect right away but may be lost after a server restart whereas changes made in Configuration tab take effect only after a restart but are not lost. So chose the appropriate option (or set up the logging in both), then go to 'Additional Properties' > Click the Change log detail levels, and add the log string to the text box ": com.ibm.mdm.hybrid.component.*=all". Review the SystemOut.log file for this enhanced logging.