bakleks 270007PVJ3 Visits (2772)
This document outlines how Physical MDM customisations can be built from source artefacts in an automated build and test system. This document does not aim to be a complete guide on this topic, but rather to point the way to how some detailed steps can be implemented using examples.
The MDM Advanced or Standard editions both include the MDM Workbench. In version 11.0 and beyond the MDM Workbench is used by solution developers to create artefacts which customise the MDM solution for the physical, virtual and hybrid implementation styles. These source code artefacts are typically built into a Composite Bundle Archive (CBA) and deployed to WebSphere where they augment the functionality already available in the MDM Server Enterprise Business Application (EBA).
A good practice amongst MDM solution developers is to create an automated build process such that customisation source code is checked-in to a code control repository, and an automated build process takes those source files and builds the CBA ready for deploying onto post-build test systems, placing built artefacts into a second repository or shared file system.
Some automated systems take this “build” concept further, by automating the deployment of such built artefacts to test systems, which in turn report back on the “health” of the build, how many tests passed and failed, and generally quickly provide valuable feedback to developers whether recent changes broke the solution or not. Project managers overseeing such projects are able to reduce project risk by adopting this continuous delivery processes, and changes to MDM solutions become more reliable and safer as a result.
To add MDM solutions to such a continuous build environment it is necessary to:
This article is mostly concerned with step #7 – building source artefacts.
2. Materials and prerequisites
This article is accompanied by a collection of example scripts. We do not intend that these are used directly, but as an example of how you may wish to implement your own automated build process.
The current solution consists of four main files:
In order for the scripts to work, the machine running the scripts needs to have the following products installed:
To run the Ant scripts the user needs to run mdm_wb_build.xml as a build file.
The script contains only the “runBuild” target.
The target checks that necessary properties, such as Eclipse Home, date and time stamps and output folder prefix are set. Provided these properties do exist, it creates a folder based on OutputFolderPrefix and date and time, within which “logs”, “CBAExport” and “workspace” folders are created.
The logs folder contains “MDM
generateDevProject: BUILD SUCCESSFUL
workspaceBuild: BUILD SUCCESSFUL
exportCBA: BUILD SUCCESSFUL
End of report.
The CBAExport folder contains all of the exported CBAs.
The workspace folder contains a local copy of build artefacts.
After the directories have been created, the script checks which operating system it is running on and sets the isLinux or isWindows property to “true” as appropriate and calls either runAnt.sh or runAnt.bat to run a headless Eclipse process. The relevant file (either the batch or the shell script) should be available by default in the bin directory in the Eclipse installation directory.
The runAnt script then sets up the log files, environment variables and runs a second script “mdm
3. Step breakdown of the automated build and test system
Given that automated building and testing of MDM solutions is a worthwhile goal, the following sections provide some guidance in some of these areas where actions specific to the MDM tools and development/build environment are necessary, and some points of discussion are presented where choices exist.
3.1 Identify the pieces of the solution which represent the “source code” for the solution.
The source code for an MDM solution will be made up of a collection of Eclipse projects and their contents. MDM development, MDM configuration, MDM hybrid mapping, MDM service tailoring, MDM custom interface, MDM metadata and other MDM-specific projects types. CBA projects will add to the list.
MDM Development projects contain a “module.mdmxmi” file, which contains a model of the customizations which the project aims to create. This file should always be considered to be source code.
At some point the mdmxmi file will be used to generate Java, XML, SQL and other file artefacts, and there are a few different approaches you can take for these files:
The current solution is to only consider files which have been manually changed as “source code”, and “generate artefacts” from the mdmxmi model as part of the automated build process itself. This approach demands that the MDM workbench tools are installed as part of the build environment, because the “generate artefacts” process that turns .mdmxmi files into other artefacts will be a necessary part of the build process.
A project “MDM
3.2 Create a source code repository.
There are many choices regarding which product to use as a source code repository and covering them is not the aim of this document.
3.3 Recognize when a consistent set of code has been checked-in, at which point a “build” is started.
This event may be triggered manually, automated overnight, or whenever a change-set is delivered to the code stream. The capturing of this event is often specific to the code control system being used, though some solution teams augment this by adding a web page that enables build requests to be manually requested.
3.4 Create a build environment.
A build environment should include RAD (or RSA) which can be called in a “headless” manner such that functionality within RAD can be used without a user-interface being present.
MDM Workbench will be required in addition to RAD to perform a complete build of “module.mdmxmi” files.
For the list of platforms that MDM Workbench v11.0 and onwards support – refer to the product release documentation.
3.5. The build environment “boot-straps” itself.
A small script is responsible for “boot-strapping” the process by it checking-out the other build scripts which in turn build the artefacts from solution developers.
3.6 The build scripts check out the artefacts from code control to the local file system.
These actions are specific to the code control system so will not be discussed further here.
3.7. The source artefacts are processed, transforming them into built artefacts.
This phase of the automated system typically consists of a hierarchy of Ant files which decomposes the overall build process into many smaller steps and “Ant targets”. The Maven framework is a common choice of technology to oversee this phase.
These Ant files can be categorized into two types:
For a detailed walkthrough of specific implementations of the build process refer to the Ant scripts provided with this blog entry.
3.8. The build process often executes “unit tests” to further validate that the solution artefacts are healthy and do what they are expected to do.
The tools and approaches used to execute unit tests vary widely dependent on technology choice. Simple Java JUnit tests offer one simple solution, which can be invoked with scripts once the tests and tested code are built.
3.9 Built artefacts are published to a repository.
Every build against which build metadata can be gathered and reported is versioned by the publish process. Build logs, unit test results and results of other tests indicating the “health” of the build are gathered and published to the repository as well.
Products such as Rational Asset Manager can be used here, or for a really basic solution a simple shared folder on a network drive may suffice.
3.10 Automated deploy and test health of overall build.
If the build is considered “good” then further automation can be added to deploy the built solution to a test environment, with higher-level tests (functional and end-to-end system tests) exercising the solution further. Such tests can also report back to the build repository on the health of each build.
The automated deployment of entire systems for testing is often one of the most complex areas of the whole continuous development process. Products such as Rational Urban Code Deploy (UCD) can be used for this stage of the process, though for some environments a set of (reasonably complex) scripts might be sufficient.
For the MDM pieces, we are mostly concerned with deploying and un-deploying SQL scripts, depl
Prior to deploying extensions to the server, it is often necessary to modify the database. This is possible using the SQL scripts found in the MDMSharedResources project in the built workspace. Rollback scripts in the same location should be applied once testing is complete to reset the database back to a known good state.
For CBA deployment, Jython scripts can be used to manipulate the WebSphere server. Detailed documentation of these steps can be found in the WebSphere documentation.
ELizBeth 270005BAKK Visits (2681)
You must have faced quite a good amount of code generation issues if you are working on MDM Server v8 or v8.5. One of the main problems that I have faced was whenever we do not have a hold on the OOTB code generation technique. Say if you need to have OOTB TCRM
Errors logged during code generation
Errors occurred during execution
Error executing tag handler: java
Another problem which we face is, MDM does not support the Transient Object code generation in v8 or 8.5 Hence we have land up with no other option other than to create an Entity, its corresponding classes and will not be persisted. It would be used as a wrapper object. Therefore, your project may have unnecessary files which you may not use in MDM framework.
By this approach it helps the developer to have a strong hold on how your service description should be by writing your own WSDL and generating the java code based on the definition in the WSDL. Definitely, this approach would be challenging as it will not be as simple as clicking the Generate Code in the MDM Model Editor
RAD : 18.104.22.168
MDM : MDM Server v 8.5
WAS : v6.1
Creating Top-Down Web Service
Web services can be created using two methods: top-down development and bottom-up development. Top-down Web services development involves creating a Web service from a WSDL file.
When creating a Web service using a top-down approach, first you design the implementation of the Web service by creating a WSDL file. You can do this using the WSDL Editor. You can then use the Web services wizard to create the Web service and skeleton Java™ classes to which you can add the required code.
Although bottom-up Web service development may be faster and easier, the top-down approach is the recommended way of creating a Web service. By creating the WSDL file first you will ultimately have more control over the Web service, and can eliminate interoperability issues that may arise when creating a Web service using the bottom-up method.
The tools that help to generate the web service artifacts are WSDL2JAVA.bat and JAVA2WSDL.bat.
WSDL2JAVA generates the web service skeletons and the deployment descriptor templates against a WSDL. This is used for the top down approach
JAVA2WSDL generates WSDL from the Java class. This is used for bottom up approach.
This document helps you to create a top to down EJB implementation of Web Service for MDM services. It is quite simple to develop a web service with the help of Web Service editor that is available with the RAD.
By the end of this document, you will be able to create an ejb web service implementation from WSDL to invoke MDM services.
Note: We need to have manual effort to bring up the web service project structure in sync with the classes and the class names that gets generated with the help of MDM model editor.
RAD Preference Settings
Prior to developing the Web Service, you need to set certain Preferences in RAD.
1. Windows > Preferences > Web Services > Resource Management
Note: It is a good practice to select this option when you use utility JAR files or third party libraries, in order to avoid these loadable Java Classes that have to be regenerated.
Below are the steps which explain how to create a top down EJB web service implementation:-
Click Next > Generates the Web Service skeleton from the WSDL and publishes the project to the server
Note:- Start the server before generating the web service
i. Move the SearchBindingImpl to Proj
ii. Rename the below files:-
SearchBindingImpl -> SearchServiceBean
iii. ejb-jar.xml ->
a) Modify the mapping of
ix. If you have defined wrapper object in WSDL to hold any OOTB objects, make sure you create the BObj, Converter and Constructor for the same.
vwilburn 100000F865 Visits (2649)
Skip the fluff and go deep into the technical details of InfoSphere MDM. In a series we're calling MDM Tech TV, several IBM engineers catch you up on technical topics. These informal video demonstrations give you a quick taste of various features.
The three new videos complement the existing MDM videos:
jtonline 110000B6Y8 Visits (2626)
Before the MDM Developers group got going a few authors from the IBM development team posted on the MDM Workbench blog. Now that this group blog has a few posts as well it makes sense to combine the two, so we won't be posting to the separate MDM Workbench blog in the future.
This blog is for more than just the workbench but the biggest advantage is that it is much easier for anyone to contribute: any member of the group can write a blog post! If you have something to share that you think the community would find useful, give it a try. It's a great way to get early feedback on something that may turn in to a full developerWorks article over time. If you do write a post, just let me know as soon as you are ready for it to be published.
These are just a selection of some of the popular posts from the old blog:
And don't forget, there are still forums available for questions and discussions:
ThomasRogers 270006MA78 Visits (2609)
Whether you're already familiar with the Workbench or completely new to it there are a lot of new changes in v11. So why not check out the MDM
There's also a major new addition to the Wiki if you're looking for more information on installing MDM on Linux head over t
vwilburn 100000F865 Visits (2557)
Currently in an open beta until February 2014, IBM
Dave_Kelsey 100000BGBR Visits (2512)
No results found is a common response to a search request, but how do you detect this in your BPM process ?
A “No Results Found” situation causes an exception to be thrown which means you can detect it using an intermediate error event, but you need to be able to handle a real errors as well as a no results found which is a special kind of exception.
Here we have a simple example of a search human service which applies equally to Physical and Virtual MDM Server.
There is an intermediate error event attached to the do search nested service that calls the MDM Search integration service. The gateway decision determines whether it is a no results found or not.
To determine how to configure the decision gateway we need to have a look at the format of the exception we get when a no results found is generated and this differs slightly between virtual and physical MDM Server.
Handling a Virtual MDM Server response
A sample of part of the xml format of the exception is shown here
Here is is the reason code we are interested in. So the decision gateway configuration looks like this
The decision logic being
extracts the reason code and checks for ENOREC.
Handling Physical MDM Response
The implementation remains the same, the test within the decision gateway needs to be altered as the location of the reason code is different to virtual. In this example the reason code that is checked is applicable to a Party or Person Search, however it is possible that the reason code will be different for different types of physical search.
A sample of part of the xml format of the exception is shown here
Here we see the reason code we are interested in is nested under the <errors> tag within an < element>.
So in this case we need the decision logic to be
Failed to connect to the JMX port on server
When you first connect from MDM Workbench to WebSphere Application Server (AppServer) where MDM Server is installed, for example to deploy a configuration project or to run a virtual job, you might see this error:
Job Manager Error - Failed to connect to the JMX port on server
There can be several reasons why the connection might fail, for background, here is the stack you are relying on when you connect to the JMX port.
In order for the JMX port connection to be successful, you need every component in this diagram to be in a fully functioning healthy state. And yes, that means there are a lot of places you can check! As a result, it's not practical here to explain every possible area to review, but this should give you some idea of where to start investigating.
To begin, cut the problem in half: there is a message associated with blueprint virtual bridge. Look for this, and it will help you decide whether the problem is more likely to be a runtime issue (below and to the right of the blueprint virtual bridge component) or a configuration issue
1. Look for virtual bridge messages
On the Application Server where MDM is hosted, open SystemOut.log or HPEL logs: if possible restart the AppServer first to make sure you have startup messages:
When the MBean starts successfully, you will see messages like these:
Note that these messages will only appear on startup, so they may not be visible if the logs have wrapped
If you have these success messages the Blueprint virtual bridge is available for JMX requests, and everything to the right of the diagram (MPIJNI, JMS, databases, filesystems) is healthy.
In this case the likely cause of the problems is to the left of the diagram, and probably relates to a configuration issue. More information is available in section 3. When the virtual bridge has started successfully
When the MBean has not started, you see messages like this:
If you have these failure messages the Blueprint virtual bridge is not available. More information is available in the next section, 2. When the virtual bridge has not started
No messages found
If you don't find any messages relating to com.
2. When the virtual bridge has not started
When the blueprint virtual bridge has not started, the next step is to investigate potential runtime issues in one or more of the components on the right side of the diagram.
Note that you can choose whether you use a datastore or filestore for the messaging engine data store: the default is datastore (database).
There may be file system errors, these will usually be reported by the component that depends on the file system, for example the database or the JMS filestore.
In many cases you will be able to find technotes or other links on the internet with information about how to resolve the errors, or if not, contact IBM support and provide the logs that show the errors.
These related links have information about resolving blueprint errors:
3. When the virtual bridge has started successfully
Once you have found the success message, the next step is to investigate the configuration in both WebSphere Application Server and MDM Workbench.
Review the server logs for authorization errors
On the Application Server where MDM is hosted, open SystemOut.log or HPEL logs. Look for errors that reference one or more of:
Errors with any of these codes suggest that you need to re-visit the security configuration in the WebSphere Application Server administrative console, and check userid and password settings in the workbench client. Review the error messages, in many cases you will be able to find technotes or other links on the internet with information about how to resolve them, or if not, contact IBM support and provide the logs that show them.
Review the firewall settings
Verify that you can ping from the Workbench machine to the machine that hosts WebSphere Application Server and MDM Server, using your preferred ping tool.
Optionally you can use "Test Connection" from MDM Workbench, although note that in an ND configuration this tool only checks the dmgr, so it may not be the correct status for the actual server where MDM is hosted.
If you can not connect to the target MDM server, the JMX connection will not work and you need to contact your networking support team to make sure the network is available and if necessary, that appropriate firewall ports are opened.
Review the port and host configuration
jaylimburn 2700028UUJ Visits (2384)
Are you attending the IBM IMPACT conference this year?
If you are then be sure to check out the sessions covering MDM and BPM alignment.
The first session hosted by IBM MDM Product Manager Trey Anderson and entitled: Improving Business Processes by Aligning BPM and MDM will cover the value proposition of combing these two technologies and cover a series of easy to obtain business benefits from running a su
Trey, Jay and Allen have been the driving force behind the integrated capabilities now available within IBM BPM and IBM InfoSphere MDM. Take the opportunity to meet with them at IMPACT and discuss your projects and how MDM and BPM alignment can help.
Nic Townsend 2700051ED4 Visits (2368)
Changing log levels
jtonline 110000B6Y8 Visits (2366)
To avoid errors being produced in code generated from module models, the workbench validates the model contents. This post summarises the naming constraints that must be observed in the model to avoid validation errors. This is not intended to be a complete list of constraints but if there are any others that you think are worth being aware of up front, please leave a comment.
The entity name must,
The entity name must not,
Database table name:
The database table name must,
The entity attribute name must,
The entity attribute name must not,
Entity field name:
The entity field name must,
The entity field name must not,
Database field name:The database field name must,
The database field name must not,
The code table name must,
The entity name must not,
Database table name:
The database table name must;
Code Table Attribute
The code table attribute name must,
The code table attribute name must not,
Database field name:
The database field name must,
The database field name must not,
Transient Data Object (TDO)
The TDO name must,
The TDO name must not,
Transient Data Object attribute
The TDO attribute name must,
The TDO attribute name must not,
The transaction name must,
The folder name must,
The folder name must not,
Hub base name:
The hub base name must,
The hub base name must not,
Details on valid Java identifiers can be found in sections 3.8 and 3.9 of the Java language specification however, due to the other naming constraints imposed in the model, the only extra restriction is the use of any of the following reserved words-
The name must not,
jaylimburn 2700028UUJ Visits (2232)
The team here are pleased to announce that after many nights researching, writing and editing, a new IBM Redbook is due to be published entitled 'Designing and Operating a Data Reservoir'.
The book takes you through the steps an organization should go through when designing and building a data reservoir solution, In this book we base the scenario around a pharmaceutical company, however the discussions, principals and patterns included in the book are relevant for any industry.
A data reservoir provides a platform to share trusted, governed data across an organization. It empowers users to engage in the sharing and reuse of data to ensure that an organization can fully leverage their most important asset. - Data. A data reservoir allows for collection of vast sets of data that can be curated, shaped and monitored to allow advanced analytics to be constructed offering new insights to an organization about their data.
See this blog post for more background on the need for a data reservoir:
The new IBM Redbook, as been authored by thought leaders in the data management space and will be available as a full IBM Redbook publication shortly. In the mean time the draft is available directly from the IBM Redbooks website:
We'd love to hear your feedback on the book and would be keen to hear your stories around data reservoir solutions.
Make Data Work
jtonline 110000B6Y8 Visits (2213)
There's a new seri
jtonline 110000B6Y8 Visits (2200)
There are a huge range of topics and speakers on offer at the Information On Demand conference later this month, but what if there's a topic you care about missing from the conference programme? Well this year there is a conference within a conference where you get to set the agenda; you won’t want to miss these sessions!! Check out the unconference site for all the details, including what an unconference is, and how to suggest topics:
Information On Demand Technical Unconference
If you've not been to an IBM unconference before, there's a Worklight demo from the IBM Impact Unconference on YouTube, which may give you some idea of what to expect. Of course, since the participants get to lead the event, every unconference is different!
RaymondMari 0600011YW5 Visits (2110)
Requirements for Developer Machine
Developers working with the MIH Workbench require a large software stack. This requirement can
have resource implications for the hardware used by the development team. It is recommended that
each developer's machine meet the following specifications:
One of the following;
· Windows 7
· Windows XP
One of the following;
· IBM Rational Software Architect v7.5.4
· IBM Rational Application Developer v7
· InfoSphere Master Information Hub (MIH) Workbench plugin
· InfoSphere Master Information Hub distribution file MIH9
o VMware workstation v8
o One of the following
When possible, it is recommended that developers use a 64-bit version of Windows
running on a Virtual machine with up to 4GB of memory. MIH Workbench can work
with less powerful hardware, but productivity may be impacted by the demands of the
large software stack.