The MDM Developers Community is open for any developers using IBM's Master Data Management software to connect with each other, stay posted on the latest MDM technical resources, learn how to get the most out of MDM tools, ask questions, and share tips and answers to technical problems.
Product: Infosphere Master Data Management. Component: IBM Stewardship Center. Version: 11.5.0
Other prerequisite software: IBM Business Process Manager 8.5.6 , IBM Process Designer 8.5.6, IBM Stewardship Center 11.5.0 installed and configured.
IBM Stewardship Center provides the ability to perform reactive data stewardship activities around Physical MDM. When a suspect gets created on MDM , a notification is being sent to BPM and a Suspected Duplicate task gets created on BPM Process portal for a data steward to act upon. ISC default implementation creates all SDP tasks for a single group called DataStewardGroup.
This document explains how to delete all those suspect tasks for a list of parties. This could be because those tasks are no longer valid that could have happened because of other collapses in MDM.
1. Open Physical MDM Suspected Duplicates process application in editable mode in IBM Process Designer or you can create a new process application and follow rest of steps.
2. In Process App create new epv “PartyIdListEpv” .To do so select Data - > Exposed Process Value
3.Now add Exposed Process Value Variable “partyIdList”
4. Create a new “General System Service” from Implementation option in the left side pane and name it “Delete Process Instance”.
5. Now go to variable tab click on “Link Epv” and select “PartyIdListEpv”
6. Now go to Diagram tab inside the service.
7. Create a new Server Script by dragging Server Script box from right side pane and name it as “Get all active SDP tasks and terminate which are mentioned in list”.
8. Inside the implementation section in properties tab add below code
var processName = "Resolve Suspected Duplicate process";
// Set Search condition.
var partyIdStore = tw.epv.PartyIdListEPV.partyIdList + "";
search.conditions = new Array(conProcessType, conStatus);
// Execute search and returns all active SDP process instances.
var processInstances = search.executeForProcessInstances();
for (var j = 0; j < partyIdArray.length; j++) {
for (var i = 0; i < processInstances.length; i++) {
var instanceId = processInstances[i].businessData.get("partyId");
if (instanceId === partyIdArray[j]) {
//terminate process instance whose party id matches
processInstances[i].abort();
break;
}
}
9. Create a “Heritage Human Service” from User Interface option in left side pane.
10. Name it as “Delete Admin Service”.
11. Now go to variable tab click on “Link Epv” and select PartyIdListEpv”
12. Inside the overview tab, we need to set Exposed to as “All Users” and Exposed As will be “Administration Service”. Attach a nested service here to "Delete Process Instance" and connect to start and end points. Refer to below screenshots.
13. Save all.
14. If using Process Server version then a new snapshot is to be created from the Process designer and installed on Process Server for the Process Admin console to have new administrative service available for users. If using Process Center version , all the changes done are available at Tip version of snapshot.
15. To set list of party Ids , go to Process Admin console and expand “Admin Tools” option on the left side pane , click on “Manage EPV”. From snapshot dropdown select Physical MDM Suspect Duplicate and from name dropdown select “PartyListEpv”. Set a comma separated string to value for “partyIdList”. These are the party ids for which suspect tasks are checked and removed if present.
16. In the same Process Admin console , expand "Physical MDM Suspected Duplicates" option on left hand side pane and click on "Delete Admin Service" option. This step may take sometime based on number of tasks in the system. Once done it gives a message saying "The service has finished".
Note : This list of steps work for any version of IBM Stewardship Center post of 11.5.0.
I have spent the last 3 years playing with MDM CE and trying to figure out how to make this powerful product easy to use and understand. And I've finally gotten around to how can one set up an easy to use security framework that can actually simplify the application for users. And after much talk with Anup Gandhi, our Chief Engineer, and Bryce Crapse who is part of IBM's MDM Services organization, we have come up with an approach that frankly makes this fairly easy..... or at least you will let me know if you find it so.
Create a role called "ACG Creation" which is only used for creating these ACG's. Do this from Admin UI which is what we are calling the UI that is used in version 11.6 to maintain the system and is the only UI in earlier versions by
Data Model
Security
Role
The starting point is to set up an ACG that will be related to each Object and name them clearly. I recommend naming the ACG's with the pattern [Object Name] [Object Type]. So the catalog with the name of "Products" would have an associated ACG called "Product Catalog ACG".
Data Model
Security
Attribute Control Groups
Attribute Control Group Console
Assign each ACG to the ACG Creation Role, and select this and nothing else:
Catalog List if of type Catalog
Hierarchy List if of type Hierarchy
Collaboration List if of type Collaboration Area
Assign the related object to the ACG
Data Model
Security
Attribute Control Groups
Object to Attribute Control Group Mapping
Create each role and select the ACG's related to the object that role should have access to, then select the additional access privileges if the related object is a Catalog or Hierarchy. The related privileges for Collaboration Areas are defined by the workflow and workflow steps. In addition, in order to the person to be able to select from associated lookup tables, settings in the default ACG will also need to be set as follows:
Select Catalog List & View
Select List View
As part of Version 11.6, we introduced a Business User Experience that leverages these same ACG's, Roles, and Users to control object access and introduced a JSON to simplify access to various buttons on the screen. I'll create a separate blog on this, but suffice it to say that the JSON links a Role in CE with the screen layout in the Business UX.
Create your Users and assign them the appropriate roles, including one or more of the roles releated to the Business UX and all should be well with the world.
Note that this format has the embedded concept of not including the object type in the object name. Since object names are currently what gets displayed to users, I am not currently aware of why a business user needs to know the difference between a catalog, hierarchy, item or category... but this is for another blog....
Let me know what you think and please add your other ideas on how to simplify. And please use the forums here to ask me questions. I will do my best to get you answers.
For Master Data Management v11.6.0.x and Reference Data Management v11.6.0.x on Ubuntu machine, the below exception may be noticed while running madconfig.sh utility.
Buildfile: build.xml
java.io.IOException: Cannot run program "env": error=2, No such file or directory
Reason:
This exception is occurred because ‘env’ program is not installed on Ubuntu machine.
This can be confirmed by executing the command env from the shell
<MDM-INSTALL-DIR>/mds/scripts$env
The program 'env' is currently not installed. You can install it by typing:
sudo apt-get install coreutils.
Solution:
To Install the "env" program on Ubuntu box, follow the below steps :
Product: Infosphere Master Data Management. Component: Data Stewardship and Governance. Version: 11.5.0
Other prerequisite software: IBM Business Process Manager 8.5.6 , IBM Process Designer 8.5.6, IBM Stewardship Center 11.5.0 installed and configured.
IBM Stewardship Center provides the ability to perform reactive data stewardship activities around Physical MDM. When a suspect gets created on MDM , a notification is being sent to BPM and a Suspected Duplicate task gets created on BPM Process portal for a Data steward to act upon. ISC default implementation creates all SDP tasks for a single group called DataStewardGroup.
This document explains how to delete all those Suspect tasks that get created.
1. Open Physical MDM Suspect Duplicate Process from BPM Process Designer.
2 .Create a new “General System Service” from Implementation option in the left side pane.
3. Name the service as “Delete Process Instance”
4. Now go to Diagram tab inside the service.
5. Create a new Server Script by dragging Server Script box from right side pane and name it as “Get all active SDP tasks and terminate them”.
6.Inside the implementation section in properties tab add below code
var processName = "Resolve Suspected Duplicate process";
// Set Search condition.
var conProcessType = new TWSearchCondition();
conProcessType.column = new TWSearchColumn();
conProcessType.column.name = TWSearchColumn.ProcessColumns.Name;
conProcessType.column.type = TWSearchColumn.Types.Process;
conProcessType.operator = TWSearchCondition.Operations.Equals;
conProcessType.value = processName;
var conStatus = new TWSearchCondition();
conStatus.column = new TWSearchColumn();
conStatus.column.name = TWSearchColumn.ProcessInstanceColumns.Status;
conStatus.column.type = TWSearchColumn.Types.ProcessInstance;
conStatus.operator = TWSearchCondition.Operations.Equals;
conStatus.value = "Active";
var search = new TWSearch();
search.conditions = new Array(conProcessType, conStatus);
// Execute search and returns all active SDP process instances.
var processInstances = search.executeForProcessInstances();
for (var i=0; i<processInstances.length; i++) {
var instanceId = processInstances[i].id
processInstances[i].abort(); // This will terminate each active process instance.
}
7. Create a “Heritage Human Service” from User Interface option in left side pane.
8. Name it as “Delete Admin Service”.
9.Inside the overview tab, we need to set Exposed to as “All Users” and Exposed As will be “Administration Service”. Refer below screenshot.
10. Now in the diagram add a new Nested Service and name it as “Delete Process Instance”.
11. Inside implementation attached a nested service which is created in step 3.
12. Now go to Process Admin Console and expand “Physical MDM Suspected Duplicates” option on the left side pane
Note : If using Process Server version then a new snapshot is to be created from the Process designer and installed on Process Server for the Process Admin console to have new administrative service available for users.
13. Now click on the “Delete Admin Service” option. This step may take sometime based on the number of tasks in the setup.
14. We will get below screen which means all the sdp task are terminated.
15. Now go to any was profile path something similar to this “C:\IBM\WebSphere\profiles\AppSrv01\bin”
Target modify_default_queues
Modifies Queue names in the Java Messaging Services configuration of the application server in which MDM is installed.
When to use:
When MDM is configured to use WebSphere Messaging Queue (WMQ) certain default queue names are used. To modify this configuration to use custom queue names the post configuration target modify_default_queues can be used.
Inputs obtained:
The target reads the below properties from file <MDM_INSTALL_DIR>/properties/post_config.properties
# The MQ queue name for AsynchronousWorkQueue, the default name is MDM.ASYNCHRONOUS.WORK. mqAsynchronousWorkQueue =
# The MQ queue name for Customer Completed Work, the default name is CUSTOMER.COMPLETED.WORK mqCustomerCompletedWork =
# The MQ queue name for Customer Integration, the default name is CUSTOMER.INTEGRATION mqCustomerIntegration =
# The MQ queue name for Customer Scheduled Work, the default name is CUSTOMER.SCHEDULED.WORK mqCustomerScheduledWork =
# The MQ queue name for Customer Tail, the default name is CUSTOMER.TAIL mqCustomerTail =
# The MQ queue name for EMQUEUE, the default name is EMQUEUE mqEMQUEUE =
# The MQ queue name for MDM Change Broadcast Queue, the default name is MDM.BROADCAST mqMDMChangeBroadcastQueue =
# The MQ queue name for MDM Messaging Backout Queue, the default name is MDM.MESSAGING.BACKOUT mqMDMMessagingBackoutQueue =
# The MQ queue name for MDM Messaging Failed Response Queue, the default name is MDM.MESSAGING.FAILED_RESPONSE mqMDMMessagingFailedResponseQueue =
# The MQ queue name for MDM Messaging Request Queue, the default name is MDM.MESSAGING.REQUEST mqMDMMessagingRequestQueue =
# The MQ queue name for MDM Messaging Successful Response Queue, the default name is MDM.MESSAGING.SUCCESSFUL_RESPONSE mqMDMMessagingSuccessfulResponseQueue =
# The MQ queue name for MDS Queue, the default name is MDS.QUEUE mqMDSQueue =
# The MQ queue name for Flex Queue, the default name is FLEX.QUEUE mqFlexQueue =
Please fill in the above file before invoking the target.
Properties pertaining to MDM application server are read from <MDM_INSTALL_DIR>/properties/mdm_install.properties file.
Invocation:
When the operating system is Windows:
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke madconfig modify_default_queues
In other operating systems
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke ./madconfig.sh modify_default_queues
Tasks performed:
Modifies the queue names in the Java Messaging Services (JMS) configuration in the application server based on inputs provided in <MDM_INSTALL_DIR>/properties/post_config.properties
Target switch_to_mq
Uninstalls configuration related to WebSphere Embedded Messaging (WEM) and configures the application server to make use of WebSphere Messaging Queues.
When to use:
When MDM has be installed using WebSphere Embedded Messaging (WEM) and has to be configured to use WebSphere Messaging Queue (WMQ) the target switch_to_mq can be used.
Inputs obtained:
The target reads the below properties from file <MDM_INSTALL_DIR>/properties/post_config.properties
# MQ server host name
messagingHost=<MQ-host>
# MQ server listener port
messagingPort=1414
# MDM user name to access to MQ server
messagingUser=<MQ-user>
# MDM user password to access to MQ server
messagingPassword=<MQ-user-password>
# The MQ Queue Manager name for MDM server
messagingQueueManager=<MDM-QMGR>
# The MQ server connection channel name for MDM server
messagingChannel=<MDM-SVR-CHANNEL>
# The MQ queue transport type for MDM server, CLIENT or BINDING
messagingTransport=CLIENT
# The MQ server home path
messagingHomePath=/usr/mqm
Please fill in the above file before invoking the target.
Properties pertaining to MDM application server are read from <MDM_INSTALL_DIR>/properties/mdm_install.properties file.
Please note that the default queue names are used during configuration. The default queue names can be found in file <MDM_INSTALL_DIR>/properties/post_config.properties
Invocation:
When the operating system is Windows:
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke madconfig switch_to_mq
In other operating systems
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke ./madconfig.sh switch_to_mq
Tasks performed:
Removes WEM related configuration from the application server on which MDM is installed.
Configures the server to use WMQ by obtaining input from <MDM_INSTALL_DIR>/properties/post_config.properties
For MDM v11.6 onwards, Modify option for MDM and RDM from Installation manager is not supported.
To support modify scenario, new post configuration targets Modify_MDM and Modify_RDM were introduced for MDM and RDM respectively. These targets help users to add or remove UI’s from current installation.
During the installation, on all of the UI panels an option to include/skip UI installation is provided as shown below.
Business Administration panel:
Inspector panel:
In above installation scenario, only Inspector UI is selected and remaining UI’s are skipped.
After extraction phase completes in Installation manager, target Configure_MasterDataManagement or Configure_ReferenceDataManagement invoked for MDM or RDM respectively.
Now, to add or remove UI’s user has to invoke Modify_MDM target from <MDM_HOME>/mds/scripts directory. This target provides user an interactive way for adding or removing of UI’s.
Adding UI to MDM installation:
If user input for Add feature, it lists out already installed UI’s and UI’s which are available for install.
Once user enters particular UI selection, it will ask user to input values for installation.Also, users given an option to use default values that were used for MDM operational server install (excepting passwords). To use default values user just has to press enter key at prompt.
After taking in input values, the target will perform WebSphere Application Server status check against user input. When the entries are valid deployment of UI starts and completes successfully.All the input values are updated in property file <MDM_INSTALL_DIR>/properties/mdm_install.properties and will be used for post configuration processes. Passwords are not saved.
Removing UI from MDM installation:
On the operation type selection, if user enters for remove UI option, it will show UI’s that are part of MDM installation as below.
Once particular UI is selected for removal, relevant passwords are expected to be entered at prompt.
A WebSphere Application Server status check is performed for and the target uninstalls the UI if the requirement is met.
Note: For RDM modify scenario, the target to be used is Modify_RDM. The available UI is Business Administration UI.
For Master Data Management v11.5.0 and Master Data Management v11.6.0 installation on Windows, installation of Microsoft Visual C++ is a pre-requisite.
Issue
The below Exception will be noticed in <MDM_INSTALL_DIR>/logs/madconfig/Configure_MasterDataManagement_<TIMESTAMP>.log when Microsoft Visual C++ is not installed.
test_datasource:
Running ODBC SQL statement [select 1 from sysibm.sysdummy1;]...
Executing C:\Program Files\IBM\MDM\mds\bin\madsql.exe Result: -1073741515
Return Code: -1073741515, Time elapsed: 0.053 sec
The error occurs because certain libraries that come with Microsoft Visual C++ are missing.
Inserts MDM gold data corresponding to the locales provided as input.
When to use:
During MDM installation, the list of locales corresponding to which MDM gold data to be inserted into the database is obtained (Code Languages) as input. When MDM gold data has to be inserted for one or more locales after MDM installation and configuration, this target can be used.
Invocation:
When the operating system is Windows:
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke madconfig install_new_db_lang_code
In other operating systems
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke ./madconfig.sh install_new_db_lang_code
Inputs obtained:
The database password
Comma separated list of one or more locales
Other details on database configuration are obtained from mdm_install.properties found in <MDM_INSTALL_DIR>/properties folder. (In MDM v11.5, other details are obtained from <MDM_INSTALL_DIR>/properties/db.properties)
Tasks performed:
Inserts rows in tables that have locale specific gold data, corresponding to the locales provided by the user
Uninstalls and installs a specified user interface application.
Invocation:
When the operating system is Windows:
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke madconfig redeploy_user_interface
In other operating systems
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke ./madconfig.sh redeploy_user_interface
Inputs obtained:
The user interface that has to be redeployed (amongst the deployed user interfaces)
The WebSphere Application Server administrator user password corresponding to the server in which the UI is installed
The WebSphere Application Server administrator user password corresponding to the server in which MDM is installed
Other details on application configuration are obtained from mdm_install.properties found in <MDM_INSTALL_DIR>/properties folder. (In MDM v11.4 FP3 and MDM v11.5, other details are obtained from the properties file corresponding to the UI in directory <MDM_INSTALL_DIR>/properties)
Tasks performed:
Uninstalls the web application (UI) specified by the user
Starts the application server on which the web application has to be installed.
Configures the web application based on details from the properties file and user input
Installs the web application on the server
Restarts the server on which the web application has been deployed
Logs:
Logs can be found at
<MDM_INSTALL_DIR>/logs/madconfig
<MDM_INSTALL_DIR>/logs/madconfig/java
Available in:
MDM v11.4 FP3
MDM v11.5
MDM v11.6
Note:
The properties file(s) that contains configuration details for the web application can be found at:
Business Administration UI: <WAS_PROFILE_HOME>/installedApps/<CELL>/ba-App-<INSTANCE_ID>.ear/propertiesUI.jar
Web Reports: <WAS_PROFILE_HOME>/installedApps/<CELL>/webreports-App-<INSTANCE_ID>.ear/webreports.war/WEB-INF/classes/webreports.properties
Uninstalls and installs the native engine which is the core virtual MDM component
When to use:
In cases where native engine deployment failed during MDM configuration.
Invocation:
When the operating system is Windows:
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke madconfig redeploy_native_ear
In other operating systems
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke ./madconfig.sh redeploy_native_ear
Inputs obtained:
The WebSphere Application Server administrator user password
The database password
Other details on application configuration are obtained from mdm_install.properties found in <MDM_INSTALL_DIR>/properties folder. (In MDM v11.4 FP3 and MDM v11.5, other details are obtained from <MDM_INSTALL_DIR>/properties/app.properties)
Tasks performed: The native engine is the core virtual MDM component and has certain configuration files including ODBC data source configuration file.
This target:
Provides details on the WebSphere Application Server environment
Obtains user consent to proceed with native ear redeployment
Checks the server status
Uninstalls the native engine
Installs the native engine and generates the configuration files
Logs:
Logs can be found at
<MDM_INSTALL_DIR>/logs/madconfig
<MDM_INSTALL_DIR>/logs/madconfig/java
Available in:
MDM v11.4 FP3
MDM v11.5
MDM v11.6
Note:
The configuration files can be found at <WAS_PROFILE_DIR>/installedApps/<CELL>/MDM-native-<INSTANCE_ID>.ear/native.war/conf folder.
Please restart the application server after invoking redeploy_native_ear.
Uninstalls and installs the Enterprise Business Application (EBA) and performs security configuration for the EBA.
Invocation:
When the operating system is Windows:
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke madconfig redeploy_EBA
In other operating systems
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke ./madconfig.sh redeploy_EBA
Inputs obtained:
The WebSphere Application Server administrator user password
MDM Administrator user password
Other details on application configuration are obtained from mdm_install.properties found in <MDM_INSTALL_DIR>/properties folder. (In MDM v11.4 FP3 and MDM v11.5, other details are obtained from <MDM_INSTALL_DIR>/properties/app.properties)
Tasks performed: The MDM Operational Server is a Business Level Application that comprises of the EBA and a jar file that contains crucial property files that determine the MDM application's functionality.
This target:
Provides details on the WebSphere Application Server environment
Obtains user consent to proceed with OSGI bundles redeployment
Checks the server status
Uninstalls the jar file that holds properties files
Uninstalls the enterprise business application
Installs the enterprise business application
Install the jar file that contains property files
Performs security configuration for the enterprise business application
Helps you obtain a vanilla instance of the MDM database with the gold data.
When to use:
In test instances, where test data has to be removed and the instance can be re-used.
In cases where database load failed during MDM configuration, due to issues like missing tablespace(s)
Please Note:
All tables in the schema in which MDM is configured will be dropped. A vanilla instance of the MDM database will be created containing database objects and gold data that come with the MDM installer. Customizations and rows inserted on a functional MDM instance will be lost. Hence, It is adviced to use this target in Development and Test instances only.
Invocation:
When the operating system is Windows:
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke madconfig recreate_database
In other operating systems
Go to <MDM_INSTALL_DIR>/mds/scripts
Invoke ./madconfig.sh recreate_database
Inputs obtained:
The database password
The WebSphere Application Server administrator user password
Other details on database configuration are obtained from mdm_install.properties found in <MDM_INSTALL_DIR>/properties folder. (In MDM v11.4 FP3 and MDM v11.5, other details are obtained from <MDM_INSTALL_DIR>/properties/db.properties)
Tasks performed:
Obtains user consent to proceed with database recreation
Stops the server
Drop all the triggers, foreign keys, indexes, sequences and tables in the given database schema.
Removes and creates the ODBC Data Source
Creates Virtual MDM database objects and populates Virtual MDM data
Create Physical MDM database objects and populates gold data
Updates the Configuration tables with the instance details
Starts the server
Please Note: This target does not drop or create tablespaces
Configuring MDM Standard Edition for external LDAP
Configurations to be done WAS Admin console of MDM.
Configurations to be done on imm file.
Configurations to be done on WAS Admin console:
When we are using external LDAP , we have to create federated repositories on MDM WAS. The reason for this being there are default groups like mdm_admin , mdm_all_cvws etc that are created during MDM installation.
Following are the steps to configure MDM to external LDAP using WebSphere Application Server Administrative Console.
Open up the WAS administrative console of MDM.
Navigate to Security -> Global security
Select “Configure” for Federated repositories and the screen would be like this.
Click on “Manage repositories” link under section “Related Items”.
Add a new LDAP repository. Details of the LDAP repository to be given are in the screenshot below. Need to specify the Primary host name and port number details for the LDAP which you want to use. Click on Apply once the details are given and this new LDAP repository gets added to the list of repositories for this WAS.
Now go back to Global security -> Federated repositories. Here Click on “Add repositories(LDAP, custom,etc)…”
It will list down the new LDAP repository we have created. Select that repository. We also need to give unique distinguished name of base entry in this repository. For the sample LDIF file we have this would be the string “cn=localhost”. All users and groups from this LDIF file are created from this base entry. It could be different based on your LDAP.
Click on Apply and save configuration.
Now when we go back to Federated repositories it would look like this.
Save all the configurations.
We will now define how to identify groups uniquely in our LDIF file. A snapshot of the way groups defined in our LDIF file here :
We will have to now define this in our LDAP configuration in WAS. Go to the following link:
We have to make sure Object classes property is set to an appropriate value based on your LDIF or LDAP configuration. In this example it had to be “groupOfUniqueNames”.
We will now define how to get the relationship between users and groups uniquely in our LDIF file. Go to the following link Global security > Federated repositories > Manage repositories > LDAP1 > Group attribute definition > Member attributes If not present already create a new member attribute with “Name of member attribute” set as “uniquemember” and Object class as “groupOfUnqiueNames”. This is specifically for the LDIF configuration shown here where a user is associated with group by uniquemember/groupOfIniqueNames combination.
Save all the configurations and restart MDM server.
Configurations to be done on IMM in MDM Workbench
Open the imm file and in the Groups tab create a new group with the same name as the one in external LDAP. In our example it is “Group1”.
In all the corresponding sub sections i.e., Attributes/Sources , Relationship Attributes , Composite Views , Interactions , Operations you need to give the appropriate access for this group.
I will just take an example of giving users of this group ability to do read write access to legal name and do a memput interaction. Screenshots for the same
Save the imm file and do deploy hub configuration.
Now when we use SOAP UI to do a memput for PerLegalName attribute with these users present in group the txn is successful. Any other user from the LDAP doesn’t have access and MDM fails with the same error.
Tip : To make users from an LDAP group ability to login to inspector the list of Interactions that are to be given access are APPGETINFO , DICGET , USRGETINFO. Once done this we need to save it and redeploy hub configuration.
Note : If you create a user and group on WAS instead of external LDAP , in addition to doing imm config changes you would need to add the user entry to mpi_usrhead table.
Inquiry transactions for certain business objects in MDM use Optimized Transparent SQLs (OTS) to improve performance. These transactions include getParty, getPerson, getOrganization, getContract and getProductInstance. The OTS feature can be leveraged by setting value for property optimized.sql to true in TCRM.properties in com.ibm.mdm.server.resources.properties.jar. By default, this value is set to true.
The SQLs to retrieve the business object and child business objects can be found in table INQLVLQUERY.
When extensions are added to the business objects, the SQLs to obtain the business objects have to be regenerated so that the new attributes that are part of the extension are also retrieved. The SQLs in the INQLVLQUERY table can be regenerated using the updateInqLevel transaction.
When this transaction is executed with the GenerateQuery value set to YES (Y), the query corresponding to the given inquiry level and business transaction gets regenerated in the INQLVLQUERY table, such that it can return Attributes added using Extensions.
The list of Inquiry Levels(InquiryLevelId) and Business Transaction Type Code (BusinessTxType) for which OTS has to be regenerated can be found by executing the below SQL:
select distinct inqlvlquery.inqlvl_id, business_tx_tp_cd from inqlvl, inqlvlquery where inqlvl.inqlvl_id = inqlvlquery.inqlvl_id and description like '%<Business Object>%'
For example, for Person object, the below query can be executed:
select distinct inqlvlquery.inqlvl_id, business_tx_tp_cd from inqlvl, inqlvlquery where inqlvl.inqlvl_id = inqlvlquery.inqlvl_id and description like '%Person%'
Oracle database automatically creates a schema when a user is created. When a user logs in the default schema used is the one with the same name as the user.
In order for InfoSphere Master Data Management v11.6 to use a schema that is not the same the user name, the Logon Trigger given below has to be created and certain privileges have to be granted to the schema name (user).
Logon Trigger
The Oracle native driver does not provide a property to specify the schema name when the schema name is not the same as the User name. Hence a trigger has to be executed, when the schema that has to be used is different from the user name.
CREATE OR REPLACE TRIGGER LOGON_TRIGGER
AFTER LOGON ON DATABASE
BEGIN
IF (USER IN ('<USER>')) THEN
EXECUTE IMMEDIATE 'ALTER SESSION SET CURRENT_SCHEMA = <SCHEMA>';
END IF;
EXCEPTION
WHEN OTHERS
THEN NULL;
END LOGON_TRIGGER;
/
In the above trigger, the placeholders <USER> and <SCHEMA> have to be replaced with the appropriate values.
Privileges to be granted
Create schema script that comes with MDM provides the below privileges to the user:
GRANT CREATE SESSION TO <SCHEMA>;
GRANT UNLIMITED TABLESPACE TO <SCHEMA>;
GRANT CREATE SEQUENCE TO <SCHEMA>;
GRANT CREATE ANY SYNONYM TO <SCHEMA>;
GRANT CREATE TABLE TO <SCHEMA>;
GRANT CREATE TRIGGER TO <SCHEMA>;
GRANT CREATE TYPE TO <SCHEMA>;
GRANT CREATE VIEW TO <SCHEMA>;
GRANT SELECT ANY TABLE TO <SCHEMA>;
GRANT IMP_FULL_DATABASE TO <SCHEMA>;
GRANT SELECT ANY DICTIONARY TO <SCHEMA>;
GRANT RESOURCE TO <SCHEMA>;
GRANT CONNECT TO <SCHEMA>;
GRANT CREATE SNAPSHOT TO <SCHEMA>;
In addition to the above privileges and access to tablespaces, the below privileges have to be granted:
GRANT SELECT ANY SEQUENCE TO <SCHEMA>;
GRANT ANALYZE ANY TO <SCHEMA>;
GRANT LOCK ANY TABLE TO <SCHEMA>;
The purpose for each Privilege is explained below:
GRANT SELECT ANY SEQUENCE TO <SCHEMA>;
This privilege is required to access sequences that are created during RDM installation.
GRANT ANALYZE ANY TO <SCHEMA>;
This privilege is required to execute SQLs which are found in Insensitive_search_enabled.sql which executes statements similar to the below:
Login to IBM Process designer with appropriate credentials.
Open up MDM stewardship toolkit and open up the Resource bundle with name “CoachLabels”. It already has the list of strings used in the UI and translated text for the default languages supported by BPM and MDM.
You can add a new Indian locale that is supported by this editor like Marathi, Punjabi, Gujarati, Tamil, Kannada and Telugu. For example lets add the locale Telugu(India) among the list of locales.
Once the locale is added , you need to put the translated text for each string in that locale.
Save once all the translated text for all the strings are given to the new locale added.
Take a snapshot of MDM Stewardship toolkit and upgrade the dependency in MDM Party Maintenance.
Create a new user using Process Admin console of BPM(for example user name is DSUserTelugu).
For the new user created mention the Locale in Process Admin Console as te-IN and update it. Screenshot for the same is
Now login with user “DSUserTelugu” in Process Portal and you should be able to see the localized strings in Telugu language for this user for Search dashboard.
Note : The Localization Resource editor should show a full list of locales existing in JRE. But some of the languages are missing definitely. For example Hindi is not present in the list to select and add translated text. So the workaround in such cases are these steps to be followed instead of steps 3,4 and 5 in the above list.
Localization Resource editor not only allows adding locales and keys one by one but also allows exporting/importing of a zip with multiple *.properties files.
Export the Localization resource “CoachLabels” from MDM stewardship toolkit as a zip file into your local filesystem.
Lets say you want to add Hindi(Locale code : hi_IN) support for this localization resource.
Add a new file called CoachLabels_hi_IN.properties file into this zip file and copy the contents from CoachLabels_en.properties.
After copying the contents translate the values to Hindi language by keeping keys same in this new property file.
Import this zip file back into MDM Stewardship toolkit and you see new locale Hindi coming up with translated text for Hindi.
Proceed with rest of steps 6 to 9 as mentioned above to use Hindi as locale for a user. Locale for the user to be mentioned in Process Admin Console in this case would be “hi_IN”.
The error ", already exists and INSERT_ONLY" may seem a bit cryptic. If you have ever encountered it on the add a record page in Inspector the good news is that it is very simple to fix. While it may look like there is a clash with the , the problem is in fact due to clashing memrecnos.
It may be that data has been bulk loaded, but the source sequence identifier settings have not taken this into account.
Firstly work out which memrecnos are already in use using the search by Source:ID option. Increase the ID in blocks until there is no data being returned.
In the workbench, go to the Source Sequence Identifiers window, connect the server and edit the entry for the source you are trying to add data too. Set the Next Identifier value to the next free value, and that will resolve the issue.
Product: Infosphere Master Data Management. Component: Data Stewardship and Governance. Version: 11.5.0
Other prerequisite software: IBM Business Process Manager 8.5.6 , IBM Process Designer 8.5.6, IBM Stewardship Center 11.5.0 installed and all process applications imported on BPM and EPVs for MDM Connection details set for all of them.
This blog provides detailed steps of connecting two MDM instances to a single Process center environment for IBM Stewardship Center(ISC) component. It provides the manual steps and doesn't do run any scripts. It also doesn’t explain all the steps required for installing and configuring ISC but only those additional steps when two MDM instances are involved.
This document assumes that you haven’t done any configurations on MDM or BPM WAS consoles for configuring ISC. If you have already done please delete those additional configurations.
Lets say you have two MDM instances MDM1 , MDM2 and single Process Center instance BPM1.
Steps to be done on WebSphere Administrative Console of MDM1.
1. Create an Alias Destination on MDM Server's SIBus as below:
a) Navigate to Service Integration -> Buses -> <BUS Name>
b) Click the <Bus name> link to open the configurations of the Bus.
c) Click on the Destination link to open the list of destinations.
d) Click on New and Select Alias and click Next
Provide the Destination attribute values:
Identifier: MDMBPMAlias
Bus: Select the SIbus of MDM environment. Note: The Bus name in typical MDM installation will be MDM.SIB.server1
Target identifier: Select Other, please specify from the drop down list. Specify the identifier of the BPM environment target. Note: Target identifier name in a typical Process center installation will be eventqueueDestination.SingleCluster
Target bus: Select Other, please specify from the drop down list. Specify the bus name of the BPM environment. Note: Target bus name In a typical Process center installation will be BPM.ProcessCenter.Bus
After providing the detail click Next
Click Finish.
Save the configurations
2. Create JMS Queue on MDM Environment as below:
a) Navigate to Resources -> JMS -> Queues
b) Select the scope. If it is a Standalone MDM server set it to Node, Server (Example, Node=mdmNode, Server=server1). If MDM installed in cluster, set the scope to Cluster
c) Click New
d) Select Default messaging provider and click OK
e) Set the values of the attribute as below:
Name: MDMBPMQueue
JNDI name: notification/MDMBPMQueue
Bus name: Select the SIbus of MDM environment. Note: The Bus name in typical MDM installation will be MDM.SIB.server1
Queue name: MDMBPMAlias
f) After setting the attribute values, Click OK and Save the configurations.
3. Create JMS Queue Connection Factory on MDM Environment as below:
a) Navigate to Resources -> JMS -> Queue connection factories
b) Select the scope. If it is a Standalone MDM server set it to Node, Server (Example, Node=mdmNode, Server=server1). If MDM installed in cluster, set the scope to Cluster
c) Click New
d) Select Default messaging provider and click OK
e) Set the values of the attribute as below:
Name: MDMBPMQueueConnectionFactory
JNDI name: notification/MDMBPMQueueConnectionFactory
Bus name: Select the SIbus of MDM environment. Note: The Bus name in typical MDM installation will be MDM.SIB.server1
Provider endpoints: It is generallyof the format “host:port:Inbound Transport Chain”. You can look at the existing Queue connection factories of MDM and get this value
f) After setting the attribute values, Click OK and Save the configurations.
4. Creation of Foreign Bus Connection on MDM Environment as below:
a) Navigate to Service Integration -> Buses -> <BUS Name>. Note: The Bus name in typical MDM installation will be MDM.SIB.server1
b) Click the <Bus name> link to open the configurations of the Bus.
c) Click on the Foreign Bus Connection link to open the list of destinations.
d) Click New…
e) Select Direct Connection and click Next
f) Select Service integration bus and click Next
g) Select the messaging engine on MDM environment from the list and click Next
h) Set the values of the attribute as below:
Name of service Integration bus to connect to (the foreign bus): Set the value to the name of the SIBus in BPM environment. Note: The Bus name in typical BPM Process center installation will be BPM.ProcessCenter.Bus
Gateway messaging engine in the foreign bus: Set the value to the name of the messaging engine of BPM environment. Note: The messaging name in typical BPM Process center installation will be SingleCluster.000-BPM.ProcessCenter.Bus
Service integration bus link name: MDM_BPM_LINK_ONE
Bootstrap service integration bus provider endpoints: Set the value as <bpmNodeHostName>:<port>:<protocol> (For example mdmdemowin:7278:BootstrapBasicMessaging) Note: If the BPM cluster has multiple nodes provide the endpoints for all the nodes as comma separated list.
i) After setting the values click Next
j) Review the summary and click Finish and Save the configurations
Steps to be done on WebSphere Administrative Console of MDM2
1.Create a new Service Integration Bus on MDM2. The reason for this to be done is because both MDM environments are identical and have same SIBus names connecting back from BPM will be an issue if BPM has to connect to two SIBuses with same name.
a) Navigate to Service Integration -> Buses -> <BUS Name>
b) Click New..
c) Give a name to the new SIBus to be created. For example MDM.SIB.NewBPMSer
2. For the new SIBus created , add a Bus member. This is required to create a messaging engine for the new bus created.
a) Navigate to Service Integration -> Buses -> <BUS Name>
b) Click the <Bus name> link to open the configurations of the Bus. Here click the SIBus created during step 1.
c) Click Bus members and Click on Add.
d) Choose the server or cluster to add to the new bus that got created. If MDM is installed on a server then server option has to be selected and the corresponding server. If cluster then the cluster option to be selected. Click Next.
e) If server option is selected , then you select option “Data store” in the type of message store. Click Next
f) If server option is selected, then select the option “Create default data source with generated JNDI name”in properties of data store. Click Next
g) Leave performance parameters as defaults and click Next.
h) Review the details and finish the configuration.
3. Follow steps 1 to 4 in the previous section. Wherever the SIBus name is required or to be selected , select the new Bus that’s created in step 1 in this section. Please note that while creating foreign bus connection provide the link name as “MDM_BPM_LINK_TWO” instead of “MDM_BPM_LINK_ONE” mentioned in the previous section.
Steps to be done on Websphere Administrative console of BPM
1. Creation of Foreign Bus Connection on BPM Environment as below:
a) Navigate to Service Integration -> Buses -> <BUS Name>
b) Click the <Bus name> link to open the configurations of the Bus.
c) Click on the Foreign Bus Connection link to open the list of foreign bus connections.
d) Click New…
e) Select Direct Connection and click Next
f) Select Service integration bus and click Next
g) Select the messaging engine on BPM environment from the list. Provide Inbound user id as admin user id of BPM and click Next
h) Set the values of the attribute as below:
Name of service Integration bus to connect to (the foreign bus): Set the value to the name of the SIBus in MDM1 instance.
Gateway messaging engine in the foreign bus: Set the value to the name of the messaging engine of MDM1 instance
Service integration bus link name: MDM_BPM_LINK_ONE
Bootstrap service integration bus provider endpoints: Set the value as <mdmInstance1HostName>:<port>:<protocol> (For example mdmdemowin:7276:BootstrapBasicMessaging)
Note: If the MDM cluster has multiple nodes provide the endpoints for all the nodes as comma separated list.
i) After setting the values click Next
j) Review the summary and click Finish and Save the configurations
Repeat the same steps but this time giving details of MDM2 instance. The values to be given in this case are
Name of service Integration bus to connect to (the foreign bus): Set the value to the name of the new SIBus created during step 1 in previous section in MDM2 instance.
Gateway messaging engine in the foreign bus: Set the value to the name of the messaging engine associated with the new SIBus created during step 1 in previous section in MDM2 instance.
Service integration bus link name: MDM_BPM_LINK_TWO
Bootstrap service integration bus provider endpoints: Set the value as <mdmInstance2HostName>:<port>:<protocol>.
If the MDM cluster has multiple nodes provide the endpoints for all the nodes as comma separated list.
So by the end of these steps you would have two foreign bus connections created on BPM. MDM_BPM_LINK_ONE pointing to MDM1 instance and MDM_BPM_LINK_TWO pointing to the new SIBus created on MDM2 instance.
Steps to be done on BPM Process Center console
1. Login to BPM Process Center Console with admin access
2. All the process applications for ISC are available at the installed location “<ISC_INSTALLED_FOLDER>/mdmg/processes”.
3. If not already imported you need to import all of them using Process Center console.
4. Create two snapshots “TESTSNAPSHOTONE” and “TESTSNAPSHOTTWO” for both MDM Party Maintenance and Physical MDM Suspected Duplicates process applications. Our focus is mainly on testing only these two.
5. Make all these 4 new snapshots Active.
Steps to be done on BPM Process Admin Console
1. Login to BPM Process Admin Console with admin access.
2. Go to Admin Tools > Manage EPVs
3. Select MDM Party Maintenance(TESTSNAPSHOTONE) and select EPV MDM_Connection_Details. Set the details of MDM1 instance over here. Provide non secure port for example 9080 for simplicity purposes, usessl option to be not checked i.e., set to false.
4. Select MDM Party Maintenance(TESTSNAPSHOTTWO) and select EPV MDM_Connection_Details. Set the details of MDM2 instance over here. Provide non secure port for example 9080 for simplicity purposes, usessl option to be not checked i.e., set to false.
5. Select Physical MDM Suspected Duplicates(TESTSNAPSHOTONE) and select EPV MDM_Connection_Details. Set the details of MDM1 instance over here. Provide non secure port for example 9080 for simplicity purposes, usessl option to be not checked i.e., set to false.
6. Select Physical MDM Suspected Duplicates(TESTSNAPSHOTTWO) and select EPV MDM_Connection_Details. Set the details of MDM2 instance over here. Provide non secure port for example 9080 for simplicity purposes, usessl option to be not checked i.e., set to false.
SQLs to be run on MDM databases
1. Run these SQLs on the database connected to MDM1 instance
UPDATE CONFIGELEMENT SET VALUE = 'true', LAST_UPDATE_DT = CURRENT_TIMESTAMP WHERE NAME = '/IBM/DWLCommonServices/Notifications/enabled';
UPDATE CDEVENTDEFTP SET ENABLE_NOTIFY='Y', LAST_UPDATE_DT=CURRENT_TIMESTAMP WHERE (LANG_TP_CD=100 AND EVENTDEF_TP_CD=8);
update configelement set value='true', last_update_dt=current_timestamp where name='/IBM/Party/SuspectProcessing/enabled';
If you give snapshot names in the Process Center Console different than these make sure to change these SQLs as well. Make sure all the SQLs are run successfully.
If any duplicate errors come up while inserting data to BPMNOTIFICATIONTYPE table delete the existing entries causing the issue and run the sqls again.
Restart all the instances MDM1, MDM2 and BPM. You need to restart the Application servers, nodeagents , deployment managers.
When using InfoSphere Master Data Management (MDM) customers will see coredumps being created when an application server is being started or stopped in an environment with multiple application servers.
Symptom
There are several use cases that can lead to this behavior, all involve a WebSphere Application Server topology of more than one JVM in a clustered or non-clustered environment.
Some examples below:
- JVMs recycle by themselves no apparent cause
- JVM1 is up and running, JVM2 is started and causes JVM1 to crash
- JVM1 and JVM2 are up and running, runbatch.sh process is started and causes JVM1 to crash
- JVM1 and JVM2 are up and running, if a user attempts to stop JVM1 or JVM2, the JVM that is being stopped crashes
The javacore and heapdumps will have two things in common:
1. The crash occurring in the ld-linux-x86-64.so.2 library (Red Hat Linux) which is called from dlsym() call in the JVM
2. A process tries to access the libMAD.so library (MDM) from an existing JVM causing it to crash
Contact IBM Support if you suspect you are experiencing this issue. Ensure the MustGather: Crash on Linux is followed to collect all required information and that the core has been processed through jextract.
Resolving the problem
In order to have immediate relief to this issue a workaround is available.
Follow steps below:
1. Stop any JVM that has the core MDM application installed
2. Locate the libMAD.so library and set attribute to IMMUTABLE
Ensure to locate the library in expandedBundles directory.
3. As root, run the command below:
chattr +i /some/path/libMAD.so
Note:
chattr (Change Attribute) is a command line Linux utility that is used to set/unset certain attributes to a file in Linux system to secure accidental deletion or modification of important files and folders.
The defect seems to be opening/closing the libMAD.so, setting this property will prevent such activity, which will prevent the crash.
Product: Infosphere Master Data Management. Component: Data Stewardship and Governance. Version: 11.5.0
Other prerequisite software: IBM Business Process Manager 8.5.6 , IBM Process Designer 8.5.6, IBM Stewardship Center 11.5.0 installed and configured.
IBM Stewardship Center provides the ability to perform proactive data stewardship activities around Physical MDM. When a suspect gets created on MDM , a notification is being sent to BPM and a Suspected Duplicate task gets created on BPM Process portal for a Data steward to act upon. This can be disabled if the Customer has huge number of suspects especially during intial load or the tasks creation rate is slowing down the process or Customer already has suspects and want to create tasks for them in BPM.
This document explains how to create SDP tasks on BPM Process Portal by fetching suspect data from MDM database.
Steps to be done on BPM Websphere Administrative Console.
1. Create a new Data Source to point to MDM DB.
Please make sure to give the data source name as “jdbc/MDMDB”. Something else can be given too but then you would need to change in the process application for that.
Please make sure to give correct database name, server name and port number related to MDM database.
Create a new entry with Alias as MDM_DB_Alias and User ID and password as credentials to the MDM database.
Click Apply and save the configuration.
Now apply this alias as Authentication alias for XA recovery and Component-managed authentication alias for the new data source created for MDMDB.
Click apply and save the configuration.
Do a Test connection for the newly created data source and if required synchronize the changes to all nodes. The test connection part for this new data source created should be successful.
Restart BPM Server after all these changes done and saved.
Steps to be done on IBM Process designer and Process Admin console
1. Import the process application into BPM using IBM Process designer or Process Center Console.
2. Provide correct values for "PHYSICAL_SDP_PROCESS_APP_SNAPSHOT" and "MDM_DATA_SOURCE_JNDI_NAME" in Process Admin console under Manage EPVs for the snapshot drop down selected as "SDP Create tasks". Screenshot for the same.
3. Run the service "Create SDP tasks" from the same Process Admin console. It is available as last item on the left hand side menu as shown in previous screenshot
This service creates SDP tasks for all the suspects fetched from MDM. It first checks if a task is existing for a Party Id. If not present it creates a new Suspected Duplicate task.This needs to be customized if a Customer wants to get only some subset of supect list and so on. The SDP task thus created is available on BPM Process Portal for a Datasteward to act upon.
Product: Infosphere Master Data Management. Component: Data Stewardship and Governance. Version: 11.5.0
Other prerequisite software: IBM Business Process Manager 8.5.6 , IBM Process Designer 8.5.6, IBM Stewardship Center 11.5.0 installed and configured.
Communication from MDM to BPM for IBM Stewardship Center is always via messaging. In case of some specific events happening on MDM, the events are to be notified to BPM to create a new task. BPM can only listen to messages that are put in its event queue. To make this possible we create a link between MDM and BPM. MDM can be installed with MQ as messaging provider. BPM suggests to use only WAS default messaging provider. To be able to send messages a Customer needs to do these steps
Configuration steps on MQ associated with MDM installation.
Configuration steps on WAS Admin console associated with MDM installation.
Configuration steps on WAS Admin console associated with BPM installation.
Configuration steps on MQ using MQ explorer
1. Create a new Sender channel under the Queue manager that is default created for MDM. installation.
2.Create a Receiver channel under the Queue manager that is default created for MDM.
Channel name : Could be anything. In our setup we named it BPMReceiver.
Transmission protocol : TCP
3. Create a local queue with usage type as Transmission under the Queue manager that is default created for MDM.
Queue name : Service Integration Bus name of BPM. In our setup the SIBus name of BPM server is BPM.ProcessCenter.Bus hence the same given.
Scope : Queue manager
Usage : Transmission
4. Create a remote queue under the Queue manager that is default created for MDM.
Queue name : Could be anything. In our setup we named it EVENTRQ.
Remote queue manager : Service Integration Bus name of BPM. In our setup the SIBus name of BPM server is BPM.ProcessCenter.Bus hence the same given.
Transmission queue : Same name as given for the queue name in previous step. It is same as Service integration bus name of BPM. In our setup it is “BPM.ProcessCenter.Bus”.
Remote queue : Destination name for which messages are to be sent on BPM Service Integration bus. Check for destination name that starts with eventqueueDestination in the list of destinations under SIBus for BPM Server.
5. Start channels created on step 1 and step 2. Sender channel created in step 1 should be in running state.
This completes list of steps to be done on Websphere MQ Explorer. Now we will see the list of steps to be done on MDM and BPM installations.
Configuration steps on Administrative Console of WAS where MDM is installed.
1. Create a new Queue Connection Factory with following details.
Navigate to Resources -> JMS - >Queue Connection Factories. Select New to create a new one.
Create it at the same scope as existing Queue Connection Factories.
Queue name : It should be same as the remote queue name created in previous section in step 4. In our setup it is EVENTRQ.
Queue manager or Queue sharing group name : Name of the Queue Manager that gets created from MDM installation. In our setup it is QM.E001.
Save all and restart WAS for these configurations to take effect.
Configuration steps on Administrative Console of WAS where BPM is installed
1.Create a new foreign bus connection to the existing Service Integration Bus of BPM.
Navigate to Service integration > Buses > BUS_NAME > Foreign bus connections -> New
Bus connection type : Direct connection
Foreign bus type : WebSphere MQ
Messaging engine to host the connection : Leave the default one listed.
Virtual queue manager name: Name of the SIBus of BPM. In our setup it is BPM.ProcessCenter.Bus
Foreign bus name: Name of the Queue Manager that gets created from MDM installation. In our setup it is QM.E001
MQ link name : Could be anything. In our setup we gave it as MDM_BPM_LINK
Enable Service integration bus to WebSphere MQ message flow : Select this checkbox if not already done.
WebSphere MQ receiver channel name : Name of the receiver channel created on step 2 of section 1 in this document. In our setup it is BPMRECEIVER
Hostname: Hostname of the MQ machine where the MQ associated with MDM is running and Queue managers are created by MDM installation. In our setup it is localhost.
Port: Port number where the Queue Manager of MQ created by MDM installation is listening. In our setup it is 1414.
Enable WebSphere MQ to Service integration bus message flow : Select this checkbox if not already done.
WebSphere MQ sender channel name: Name of the sender channel created on step 1 of section 1 in this document. In our setup it is BPMSENDER
SIB inbound user ID : An admin user id who have the ability to put messages to BPM Queue. This could be a default admin user that gets created with BPM installation. In our setup it is “admin”.
Save all and restart WAS for these configurations to take effect.
Product: Infosphere Master Data Management. Component: Data Stewardship and Governance. Version: 11.5.0
Other prerequisite software: IBM Business Process Manager 8.5.6 , IBM Process Designer 8.5.6, IBM Stewardship Center 11.5.0 installed and configured.
IBM Stewardship Center provides the ability to perform proactive data stewardship activities around Physical MDM. When a suspect gets created on MDM , a notification is being sent to BPM and a Suspected Duplicate task gets created on BPM Process portal for a Data steward to act upon. ISC default implementation creates all SDP tasks for a single group called DataStewardGroup.
This document explains how to create tasks to different data steward groups based on certain data on the notification message.
1. Open Physical MDM Suspected Duplicates process application in editable mode using IBM Process Designer.
2. Add one input variable called "partySourceId" in process "Resolve Suspected Duplicate process".
3. Create two groups called DSSourceOneGroup and DSSourceTwoGroup . Each of them have two users. DSSourceOneGroup has two users called DSSourceOneUser1 and DSSourceOneUser2. DSSourceTwoGroup has two users called DSSourceTwoUser1 and DSSourceTwoUser2. Basically users and groups creation is done using Process admin console
4. Create two teams with names "PSDP Source One DS Team" and "PSDP Source Two DS Team" and associate each team with one group created in previous step. PSDP Source One DS Team has group “DSSourceOneGroup” and "PSDP Source Two DS Team" has group “DSSourceTwoGroup” associated.
This step is done in Phsyical MDM Suspected Duplicates process application using IBM Process Designer.
5. Change the assignments part for the step called "Suspected Duplicate" in process "Resolve Suspected Duplicate process".
So basically if partySourceId is 1 then we are assigning to the team called "PSDP Source One DS Team" which internally assigns to first group and two users associated with that. Similary if partySourceId is 2 then the task is assigned to second data steward team and group.
6. Add the following script to step called "Create Task" on the service "Process Suspected Duplicate Event Message"
So when we run with making partySourceId as 1 the task gets created for users of group with name “DSSourceOneGroup” and with 2 it gets assigned for users of group with name “DSSourceTwoGroup”. This can be checked by logging into Process portal with different users from two groups that we created.
This is just to show the concept of dynamically routing tasks. In any customer scenario we need to identify the attributes based on which the routing has to be done. This attribute information also need to be available from the message or notification sent from MDM to BPM. This would also need customizing notification classes on MDM to have the required information sent in the message.
Product: Infosphere Master Data Management. Component: Data Stewardship and Governance. Version: 11.5.0
Other prerequisite software: IBM Business Process Manager 8.5.6 , IBM Process Designer 8.5.6, IBM Stewardship Center 11.5.0 installed and configured.
IBM Stewardship Center provides the ability to perform proactive data stewardship activities around Physical MDM with a dashboard called “SEARCH”. It allows you to Search for an Entity , select one among various search results and View/Update various details about that Entity like Addresses, Names and so on.
It also provides a graphical view of the relationships of the party with other parties in a tab namely “Explore” in Edit Party screen.
This document concentrates on getting the required labels on this graphical view for custom added relationships on MDM.
Problem: I added a new relationship type with type code “300000” in cdreltp of MDM and I have two parties with relationship created among them. The screenshot in Explore tab now is:
1. Login to IBM Process Designer with required credentials.
2. Provide edit access to this user to MDM Application Toolkit in the list of toolkits.
Make sure “Allow users to update toolkit” checkbox is checked.
3. Open MDM Application toolkit in designer.
4. Go to service “MDM Hierarchy Localization Service” under User Interface category.
5. Modify the script “Load Label Map”.
6. Add the following line of code at the end of the script.
tw.local.labels.put("PARTYREL_300000",”CEO”);
Note : If you need to do localization for this label , you can add it in the Localization Resource “MDM_Hierarchy_Resource” and use it accordingly in the script as
tw.local.labels.put("PARTYREL_300000", tw.resource.MDM_Hierarchy_Resource.CEO_Relationship_Label);
7. Once done these changes, save all the modifications.
8. Take a new snapshot of MDM Application toolkit
9. Upgrade the dependency of MDM Application toolkit in MDM Stewardship Toolkit and create a new snapshot of MDM Stewardship Toolkit.
10. Upgrade the dependencies of MDM Stewardship Toolkit and MDM Application Toolkit in MDM Party Maintenance process application.
Please make sure not to miss step 9.
11. Reload Search dashboard from Process Portal and search for the same record and go to edit page. The explore tab now looks like
IBM InfoSphere MDM provides a set of out-of-the-box entity processing rules, like 'partyMatch' or 'collapseParties'. These rules are extendible and this blog entry will walk through the process of extending one of them – 'collapsePartiesWithRules'.
Assuming that a development project has already been created in the workspace, code for the project has been generated and setup SQL scripts have been ran – create a new package in the project's 'src' folder (For example: 'com.ibm.mdm.customRules'). Within that package create a new java class (For example: 'CustomCollapse.java'). As we are looking at extending 'collapsePartiesWithRules' the Java class should extend the 'CollapsePartiesWithRules' class and its 'execute' method should be overridden.
Within the new class the default 'collapse' can be used by calling the 'collapseObjectsSurvivingRules()' method on the TCRM component and supplying the parties that need collapsing inside a 'Vector' object. Additional behaviour can be defined as well.
The created package and Java class should then be added to the 'blueprint.xml' file present in the development project as follows:
Export the 'CBA' using the wizard and deploy it to the server (instructions for one of the approaches to deploying a CBA can be found here).
To make the system use the modified rule an update to the database needs to be run. Search the 'JAVAIMPL' table for an entry with the 'JAVA_CLASSNAME' of 'com.dwl.tcrm.externalrule.CollapsePartiesWithRules' and note the rule id ('1038' in this case). Run the following sql to update the class name to the updated rule:
UPDATE SCHEMA.JAVAIMPL SET JAVA_CLASSNAME = 'com.ibm.mdm.customRules.CustomCollapse', LAST_UPDATE_DT = CURRENT_TIMESTAMP WHERE EXT_RULE_IMPL_ID = 1038;
Restart the server.
The next step would be to re-configure the Optimized Transparent SQL (OTS) queries. OTS queries allow the 'SELECT' statements used to retrieve data from the database to be customized and optimized for a given deployment.
The 'INQLVL' table defines the set of OTS capable entities and their associated inquiry levels. Depending on the types of the objects that will be collapsed – the appropriate 'GROUP_NAME' should be looked up. In this example we will be working with Organizations.
The values to note are 'INQLVL_ID' and 'INQLVL'. It's worth noting that the 'SELECT' statements themselves are not stored in this table, but rather are contained within 'INQLVLQUERY' table.
Each combination of Entity-Inquiry Level contains one or more 'SELECT' statements within the table. These statements are also associated with a 'BUSINESS_TX_TP_CD' value (32) which is derived from the 'CDBUSINESSTXTP' table and defines the associated query transaction.
The 'CDINQLVLQUERYTP' table defines the possible type code values for entries in the 'INQLVLQUERY' table.
'CDBUSINESSTXTP' table contains a list of available transactions and associated 'BUSINESS_TX_TP_CD' values.
The OTS queries need to be re-generated and updated to include the extended fields after the customized CBA has been deployed, otherwise those fields will be omitted from the response. The 'add' and 'update' transactions are not affected as they are not query transactions. There exists an 'updateInqLevel' transaction to re-build these queries.
For each Inquiry Level Id resolved above a single 'updateInqLevel' transaction needs to be ran against the server. The following fields need to be filled out: InquiryLevelId (from the 'INQLVL' table), InquiryLevelLastUpdateDate (from the 'INQLVL' table), GenerateQuery (set to 'Y' to rebuild the queries) and BusinessTxType (from the 'INQLVLQUERY' table). So for Organization (inquiry level 3) the following transaction will be used:
<ErrorMessage>The data submitted already exists on the database; no update applied.</ErrorMessage>
<ErrorType>DRECERR</ErrorType>
<LanguageCode>100</LanguageCode>
<ReasonCode>603</ReasonCode>
<Severity>5</Severity>
<SeverityValue>Warning</SeverityValue>
</DWLError>
</DWLStatus>
</DWLInqLevelBObj>
</ResponseObject>
</TxResponse>
</DWLAdminService>
Because the transaction does not make any changes to the 'INQLVL' table itself – the last update date does not get changed and the transaction reports an error stating that the data has not been updated. However, the contents of the 'INQLVLQUERY' table shows that the changes have been applied and both the queries and the last update date have changed.
Custom rules should now work and transactions implementing these rules should return the contents of the default and extended fields.
New in MDM v11.4 is the ability to submit REST requests to the service controller.
No more complex SOAP client code, or worst EJB remote calls!
Previously, interactions with MDM Operational server were possible with EJB/RMI, JMS, JAX-WS and JAX-RPC(deprecated). We have now added JAX-RS to the mix.
Possible payloads that are accepted are application/xml and application/json. JSON support was added in v11.4 FP1.
The same xml request/response payload used for EJB/RMI is used for REST interactions.
ASI (Adaptive Service Interface) can be used in conjunction with the RESTful service. This is particularly useful if a simplified XML/JSON structure is required for interactions or if the MDM solution must adapt to a certain XML industry standard such as IFW, ACCORD, and NIEM.
For the full list of capabilities and supported request headers consult the following documentation link:
The default MDM JSON model is actually based on the core XML schema model (MDMCommon.xsd and MDMDomains.xsd). Internally, MDM will validate the JSON using these schemas.
We use a “Mapped notation” api to build the JSON. A couple things to note about this implementation:
XML elements that contain nested elements will be mapped to JSON objects (ie.<TCRMPersonBObj> will be "TCRMPersonBObj": { … })
XML element values will be mapped to a JSON name/value pair (ie.<GivenNameOne>Jane</GivenNameOne> will be "GivenNameOne": "Jane")
XML element attributes become JSON name/value pairs within the JSON object. (ie. <tcrmParam name="PartyId">1</tcrmParam> will be "tcrmParam": { "@name": "PartyId", "$": "1" })Note the attribute is prefix with “@”. The xml element value will be mapped to the name/value pair using the name “$”. The “@” and “$” characters are used to distinguish the attribute value from the element’s value.
XML array elements, an XML document which contains multiple nested elements of the same name will automatically be mapped to a JSON array. (ie. <TCRMPersonNameBObj>
<PersonNameIdPK>822844191261056191</PersonNameIdPK>
<NameUsageType>1</NameUsageType>
...
</TCRMPersonNameBObj>
<TCRMPersonNameBObj>
<PersonNameIdPK>821544198626606646</PersonNameIdPK>
<NameUsageType>2</NameUsageType>
...
</TCRMPersonNameBObj> will be
There are instances where we need to force the return of a JSON array. This is the case for list type elements in MDM or results which include multiple elements. To control this behavior, you can set the elements which need to be forced as JSON array using the Configuration Management element “/IBM/DWLCommonServices/Restful/JsonResponse/ArrayObjects” (comma separated value list).
Don’t want to write any code to test your MDM services?
Using Firefox/Chrome POSTMAN REST client extension, you can submit your requests directly to the backend server securely.
Choose “PUT” as the HTTP method
Set headers: Content-Type and Accept to either application/xmlor application/json
Set authorization by supplying credentials using Basic Auth tab.
Another great tool is SOAPUI. This tool can be used to fully automate your functional and regression testing by submitting REST requests. For more info on this tool visit http://www.soapui.org/.
Another option is using cURL (http://curl.haxx.se/)
cURL is a cmd line tool for transferring data using a URL syntax.
From version 11.4 FixPack 3 the MDM Application Toolkit has a new hierarchy widget, which replaces the now deprecated MDM Tree coach view.
This new widget, the MDM Hierarchy coach view uses the latest in web visualisation technology to render hierarchies in BPM coaches. As well as using this new technology the MDM Hierarchy coach view also has a new method of interacting with the MDM operational server.
To highlight some of the new features, this post presents a step-by-step guide of how to get up and running with the new MDM Hierarchy coach view. I will assume a degree of familiarity with IBM BPM, in particular Process Designer.
Set-up:
Before opening BPM ensure the MDM operational server has a hierarchy, containing at least one hierarchy node.
Create a new Process App, I have called mine Hierarchy Demo.
Open the new Process App in IBM Process Designer
Import the MDM Application Toolkit
Log in to Process Admin in a browser, expand Admin Tools, click on Manage EPV’s. In the snapshot drop down select the Process App you have just created. In the name drop down select MDM_Connection_Details. Configure the EPV values in the variable list with your MDM servers’ details.
Create a Client-Side Human Service, I’ve called mine Demo Hierarchy
Open up the default coach, created in the Client-Side Human Service
Step 1: Drag and drop the MDM Hierarchy coach view from the palette onto the canvas, it is listed under the MDMAT grouping.
Switch to the configuration tab. You will notice that most of the fields have default values. In the rootNodeId field enter the values for the hierarchy and a node in the hierarchy in the format <hierarchyID/hierarchyNodeID>
Step 2: Press the “Run” button in BPM. This will launch a browser, showing the coach you have just created. The hierarchy will be visible, and should render data if it has been set up correctly.
That is all that is required to get the MDM Hierarchy coach view up and running.
The coach view has a set of other configuration options; please see the documentation for more details on the configuration options.
The MDM Hierarchy coach view can be augmented by connecting it to a set of other coach views, which provide pop-up dialogs with additional behaviour that complements the hierarchy. These are the MDM Hierarchy Dialog Add, the MDM Hierarchy Dialog Details, the MDM Hierarchy Dialog Error and MDM Hierarchy Dialog MultiParent.
While each of these coach views can be added independently, the instructions below will guide you through adding them all.
Step 1: Adding the other coach views.
Drag and drop the MDM Hierarchy Dialog Add, the MDM Hierarchy Dialog Details, the MDM Hierarchy Dialog Error and MDM Hierarchy Dialog MultiParent on to the canvas that contains the MDM Hierarchy coach view.
Step 2: Create a new MDM_Hierarchy_Event_Framework for all of the widgets to use.
Switch to the variables tab. Create a new Private variable, call it “events”. Change the Variable Type to MDM_Hierarchy_Event_Framework. This variable is used by the MDM Hierarchy coach view to communicate with the other coach views, it provides a hot swappable mechanism so that the out of the box dialogs can be replaced with custom dialogs making use of the event framework.
Step 3: Configure all of the widgets to use the same, shared event framework. Switch back to the Coaches view. For each coach, select it on the canvas, then select the Configuration tab at the bottom.
Locate the EventFramework configuration option; click the purple button next to the label. Then click the Select button to the right hand side. Find the variable you created in Step 2 (events) and select it. Do this for each of the widgets.
Step 4: Configure the visibility for each of the dialog coach views; this step should not be performed on the MDM Hierarchy coach view.
Select the coach view, then click the Visibility tab at the bottom. Leave source as “Value” then press the purple button next to the Visibility label. Press the Select button, then expand the events variable, expand the appropriate event, then select the visibility entry. Each of the different dialogs should be configured against its specific event. The MDM Hierarchy Dialog Add should be configured to use the addNode event; the MDM Hierarchy Dialog Details should be configured to use the nodeDetails event; the MDM Hierarchy Dialog MuliParent should be configured to use the multiParent event; the MDM Hierarchy Error Details should be configured to use the error event.
Step 5: Click the “Run” button in BPM.
The tree now has additional behaviour, if you right click on a node a pop-up dialog should now appear that will display additional data about the node. The add button on this dialog will launch the add dialog that can be used to add nodes into the hierarchy. If a node in the hierarchy has multiple parents in the hierarchy an icon indicating this is displayed to the right of the node, the MultiParent dialog will be launched if that icon is clicked and allows users to re-focus the hierarchy on the different parent nodes.
This brief post has demonstrated how to use the new MDM Hierarchy and associated coach views. In future posts more advanced topics, such as replacing the ajax service which supply the data to the hierarchy, and how to create custom widgets that use the event framework will be explored.
When you first connect from MDM Workbench to WebSphere Application Server (AppServer) where MDM Server is installed, for example to deploy a configuration project or to run a virtual job, you might see this error:
Job Manager Error - Failed to connect to the JMX port on server
There can be several reasons why the connection might fail, for background, here is the stack you are relying on when you connect to the JMX port.
In order for the JMX port connection to be successful, you need every component in this diagram to be in a fully functioning healthy state. And yes, that means there are a lot of places you can check! As a result, it's not practical here to explain every possible area to review, but this should give you some idea of where to start investigating.
To begin, cut the problem in half: there is a message associated with blueprint virtual bridge. Look for this, and it will help you decide whether the problem is more likely to be a runtime issue (below and to the right of the blueprint virtual bridge component) or a configuration issue
1. Look for virtual bridge messages
On the Application Server where MDM is hosted, open SystemOut.log or HPEL logs: if possible restart the AppServer first to make sure you have startup messages:
Success scenario
When the MBean starts successfully, you will see messages like these:
RMIConnectorC A ADMC0026I: The RMI Connector is available at port <xxxx>
JMXConnectors I ADMC0058I: The JMX JSR160RMI connector is available at port <xxxx>
BlueprintCont I org.apache.aries.blueprint.container.BlueprintContainerImpl doRun Bundle com.ibm.mdm.server.virtualbridge is waiting for dependencies [(objectClass=com.ibm.mdm.server.config.api.ConfigManager), (objectClass=com.initiatesystems.hub.mpinet.MPINetProtocolLogic)]
Note that these messages will only appear on startup, so they may not be visible if the logs have wrapped
If you have these success messages the Blueprint virtual bridge is available for JMX requests, and everything to the right of the diagram (MPIJNI, JMS, databases, filesystems) is healthy.
In this case the likely cause of the problems is to the left of the diagram, and probably relates to a configuration issue. More information is available in section 3. When the virtual bridge has started successfully
Failure scenario
When the MBean has not started, you see messages like this:
BlueprintCont E org.apache.aries.blueprint.container.BlueprintContainerImpl$1 run Unable to start blueprint container for bundle com.ibm.mdm.server.virtualbridge due to unresolved dependencies [(objectClass=com.ibm.mdm.server.config.api.ConfigManager), (objectClass=com.initiatesystems.hub.mpinet.MPINetProtocolLogic)]
This message identifies a problem with the JMX MBean listener, but does not in itself identify the root cause: to find the source, look for other messages in the logs with earlier timestamps.
If you have these failure messages the Blueprint virtual bridge is not available. More information is available in the next section, 2. When the virtual bridge has not started
No messages found
If you don't find any messages relating to com.ibm.mdm.server.virtualbridge, the most likely reason is that there were messages when the server started, but the logs have wrapped the virtual bridge messages are no longer current. The recommended action is to restart the server and collect the new logs.
2. When the virtual bridge has not started
When the blueprint virtual bridge has not started, the next step is to investigate potential runtime issues in one or more of the components on the right side of the diagram.
Look for Database errors, search the Application Server logs for
SQL
DSRA
Look for JMS errors, the JMX MBean for virtual relies on the SIBus messaging engine, search the Application Server logs for
CWSIS
Note that you can choose whether you use a datastore or filestore for the messaging engine data store: the default is datastore (database).
There may be file system errors, these will usually be reported by the component that depends on the file system, for example the database or the JMS filestore.
In many cases you will be able to find technotes or other links on the internet with information about how to resolve the errors, or if not, contact IBM support and provide the logs that show the errors.
These related links have information about resolving blueprint errors:
3. When the virtual bridge has started successfully
Once you have found the success message, the next step is to investigate the configuration in both WebSphere Application Server and MDM Workbench.
com.ibm.mdm.server.virtualbridge is waiting for dependencies
Review the server logs for authorization errors
On the Application Server where MDM is hosted, open SystemOut.log or HPEL logs. Look for errors that reference one or more of:
LDAP
LTPA
SECJ
Errors with any of these codes suggest that you need to re-visit the security configuration in the WebSphere Application Server administrative console, and check userid and password settings in the workbench client. Review the error messages, in many cases you will be able to find technotes or other links on the internet with information about how to resolve them, or if not, contact IBM support and provide the logs that show them.
Review the firewall settings
Verify that you can ping from the Workbench machine to the machine that hosts WebSphere Application Server and MDM Server, using your preferred ping tool.
Optionally you can use "Test Connection" from MDM Workbench, although note that in an ND configuration this tool only checks the dmgr, so it may not be the correct status for the actual server where MDM is hosted.
If you can not connect to the target MDM server, the JMX connection will not work and you need to contact your networking support team to make sure the network is available and if necessary, that appropriate firewall ports are opened.
Review the port and host configuration
In the WebSphere application console
Go to Servers -> Server Types -> WebSphere application servers
Select the server where the MDM runtime is installed
Scroll down to the ports section and open it (see the screenshot below)
Make a note of the ports for
BOOTSTRAP_ADDRESS
SOAP_CONNECTOR_ADDRESS
ORB_LISTENER_ADDRESS (if not 0)
In the Workbench
Go to the Servers tab (you may need to add it from Window -> Show View)
In the Overview panel, under Server select Manually provide connection settings
Set the RMI port to BOOTSTRAP_ADDRESS
Set SOAP port to SOAP_CONNECTOR_ADDRESS
Test these settings initially without any IPC port configuration
If you are still not able to connect, also configure the IPC port with the ORB_LISTENER_ADDRESS and retest
Have you ever tried to start a BPM server on linux only to be greeted by the following incomprehensible error?
java.lang.NoClassDefFoundError: org.eclipse.emf.ecore.EFactory
at java.lang.ClassLoader.defineClassImpl(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:275)
at java.security.SecureClassLoader.defineClass
(SecureClassLoader.jav4)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:540)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:451)
at java.net.URLClassLoader.access$300(URLClassLoader.java:79)
at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:
1038)
at java.security.AccessController.doPrivileged(AccessController.
java2)
at java.net.URLClassLoader.findClass(URLClassLoader.java:429)
at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.
java4)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:703)
at java.lang.ClassLoader.loadCl
