This document outlines how Physical MDM customisations can be built from source artefacts in an automated build and test system. This document does not aim to be a complete guide on this topic, but rather to point the way to how some detailed steps can be implemented using examples.
The MDM Advanced or Standard editions both include the MDM Workbench. In version 11.0 and beyond the MDM Workbench is used by solution developers to create artefacts which customise the MDM solution for the physical, virtual and hybrid implementation styles. These source code artefacts are typically built into a Composite Bundle Archive (CBA) and deployed to WebSphere where they augment the functionality already available in the MDM Server Enterprise Business Application (EBA).
A good practice amongst MDM solution developers is to create an automated build process such that customisation source code is checked-in to a code control repository, and an automated build process takes those source files and builds the CBA ready for deploying onto post-build test systems, placing built artefacts into a second repository or shared file system.
Some automated systems take this “build” concept further, by automating the deployment of such built artefacts to test systems, which in turn report back on the “health” of the build, how many tests passed and failed, and generally quickly provide valuable feedback to developers whether recent changes broke the solution or not. Project managers overseeing such projects are able to reduce project risk by adopting this continuous delivery processes, and changes to MDM solutions become more reliable and safer as a result.
To add MDM solutions to such a continuous build environment it is necessary to:
- Identify the pieces of the solution which represent the “source code” for the solution.
- Create a source code repository which identified “source code” can be checked into. GIT is a popular choice here, though there are many alternatives such as Rational Team Concert from IBM, CVS, SVN, etc.
- Recognize when a consistent set of code has been checked-in, at which point a “build” is started.
- Create a build environment.
- The build environment often “boot-straps” itself via a simple initial script which checks-out the rest of the build scripts which in turn build the artefacts from solution developers.
- The build scripts check out the artefacts from code control to the local file system.
- The source artefacts are processed, transforming them into built artefacts
- The build process often executes “unit tests” to further validate that the solution artefacts are healthy and do what they are expected to do.
- Built artefacts are published to a repository which versions every build and against which build metadata can be gathered and reported. Such build logs, unit tests results, and results of other tests indicate the “health” of the build.
- If the build is considered “good” then further automation can be added to deploy the built solution to a test environment, with higher-level tests (functional and end-to-end system tests) exercising the solution further. Such tests can also report back to the build repository on the health of each build.
This article is mostly concerned with step #7 – building source artefacts.
2. Materials and prerequisites
This article is accompanied by a collection of example scripts. We do not intend that these are used directly, but as an example of how you may wish to implement your own automated build process.
The current solution consists of four main files:
In order for the scripts to work, the machine running the scripts needs to have the following products installed:
Rational Application Developer (RAD) or Rational Software Architect (RSA)
WebSphere Application Server (WAS) with a profile set up
MDM Workbench (the scripts accommodate version 11.0 and above)
mdm_wb_build.properties contains a list of properties that need to be defined before running the scripts:
“InputFolder” defines the folder containing the code to be imported/built.
“OutputFolderPrefix” property defines the folder to which the projects will be copied and CBAs exported. Currently the script generates a time-stamp that gets added to the defined output folder prefix.
“AntContribHome” should be set to the folder containing “ant-contrib.jar” which provides some of the functionality used within the scripts.
“EclipseHome” should be set to the installation folder of RAD or RSA.
“WASHome” should be set to the installation folder of WAS and both “WASRuntimeTypeId” and ““WASRuntimeId”should be specified as well.
To run the Ant scripts the user needs to run mdm_wb_build.xml as a build file.
The script contains only the “runBuild” target.
The target checks that necessary properties, such as Eclipse Home, date and time stamps and output folder prefix are set. Provided these properties do exist, it creates a folder based on OutputFolderPrefix and date and time, within which “logs”, “CBAExport” and “workspace” folders are created.
The logs folder contains “MDM_WB_BUILD_log.html” and “MDM_WB_BUILD_log.xml”. The XML file stores debug information. The HTML file contains general information on whether the main steps were successful, for example:
checkPropertiesExist: BUILD SUCCESSFUL
createServerRuntimeAndTargetPlatform: BUILD SUCCESSFUL
generateDevProject: BUILD SUCCESSFUL
workspaceBuild: BUILD SUCCESSFUL
exportCBA: BUILD SUCCESSFUL
End of report.
The CBAExport folder contains all of the exported CBAs.
The workspace folder contains a local copy of build artefacts.
After the directories have been created, the script checks which operating system it is running on and sets the isLinux or isWindows property to “true” as appropriate and calls either runAnt.sh or runAnt.bat to run a headless Eclipse process. The relevant file (either the batch or the shell script) should be available by default in the bin directory in the Eclipse installation directory.
The runAnt script then sets up the log files, environment variables and runs a second script “mdm_wb_build_inside_eclipse.xml” inside a headless RAD/RSA environment.
3. Step breakdown of the automated build and test system
Given that automated building and testing of MDM solutions is a worthwhile goal, the following sections provide some guidance in some of these areas where actions specific to the MDM tools and development/build environment are necessary, and some points of discussion are presented where choices exist.
3.1 Identify the pieces of the solution which represent the “source code” for the solution.
The source code for an MDM solution will be made up of a collection of Eclipse projects and their contents. MDM development, MDM configuration, MDM hybrid mapping, MDM service tailoring, MDM custom interface, MDM metadata and other MDM-specific projects types. CBA projects will add to the list.
MDM Development projects contain a “module.mdmxmi” file, which contains a model of the customizations which the project aims to create. This file should always be considered to be source code.
At some point the mdmxmi file will be used to generate Java, XML, SQL and other file artefacts, and there are a few different approaches you can take for these files:
The current solution is to only consider files which have been manually changed as “source code”, and “generate artefacts” from the mdmxmi model as part of the automated build process itself. This approach demands that the MDM workbench tools are installed as part of the build environment, because the “generate artefacts” process that turns .mdmxmi files into other artefacts will be a necessary part of the build process.
A project “MDMSharedResources” in the workspace can be considered “source code”, and it is recommended that this project is checked-into the code control system, together with its .project file and .mdmxmi file. The other contents of this project can generally be ignored from check-in, as the artefacts in this project often will be re-created by the automated build system.
3.2 Create a source code repository.
There are many choices regarding which product to use as a source code repository and covering them is not the aim of this document.
3.3 Recognize when a consistent set of code has been checked-in, at which point a “build” is started.
This event may be triggered manually, automated overnight, or whenever a change-set is delivered to the code stream. The capturing of this event is often specific to the code control system being used, though some solution teams augment this by adding a web page that enables build requests to be manually requested.
3.4 Create a build environment.
A build environment should include RAD (or RSA) which can be called in a “headless” manner such that functionality within RAD can be used without a user-interface being present.
MDM Workbench will be required in addition to RAD to perform a complete build of “module.mdmxmi” files.
For the list of platforms that MDM Workbench v11.0 and onwards support – refer to the product release documentation.
3.5. The build environment “boot-straps” itself.
A small script is responsible for “boot-strapping” the process by it checking-out the other build scripts which in turn build the artefacts from solution developers.
3.6 The build scripts check out the artefacts from code control to the local file system.
These actions are specific to the code control system so will not be discussed further here.
3.7. The source artefacts are processed, transforming them into built artefacts.
This phase of the automated system typically consists of a hierarchy of Ant files which decomposes the overall build process into many smaller steps and “Ant targets”. The Maven framework is a common choice of technology to oversee this phase.
These Ant files can be categorized into two types:
- Run outside of the RAD environment;
- Run inside a “headless” RAD instance.
For a detailed walkthrough of specific implementations of the build process refer to the Ant scripts provided with this blog entry.
3.8. The build process often executes “unit tests” to further validate that the solution artefacts are healthy and do what they are expected to do.
The tools and approaches used to execute unit tests vary widely dependent on technology choice. Simple Java JUnit tests offer one simple solution, which can be invoked with scripts once the tests and tested code are built.
3.9 Built artefacts are published to a repository.
Every build against which build metadata can be gathered and reported is versioned by the publish process. Build logs, unit test results and results of other tests indicating the “health” of the build are gathered and published to the repository as well.
Products such as Rational Asset Manager can be used here, or for a really basic solution a simple shared folder on a network drive may suffice.
3.10 Automated deploy and test health of overall build.
If the build is considered “good” then further automation can be added to deploy the built solution to a test environment, with higher-level tests (functional and end-to-end system tests) exercising the solution further. Such tests can also report back to the build repository on the health of each build.
The automated deployment of entire systems for testing is often one of the most complex areas of the whole continuous development process. Products such as Rational Urban Code Deploy (UCD) can be used for this stage of the process, though for some environments a set of (reasonably complex) scripts might be sufficient.
For the MDM pieces, we are mostly concerned with deploying and un-deploying SQL scripts, deploying/un-deploying CBA files, and starting and stopping the MDM solution and WebSphere server(s).
Prior to deploying extensions to the server, it is often necessary to modify the database. This is possible using the SQL scripts found in the MDMSharedResources project in the built workspace. Rollback scripts in the same location should be applied once testing is complete to reset the database back to a known good state.
For CBA deployment, Jython scripts can be used to manipulate the WebSphere server. Detailed documentation of these steps can be found in the WebSphere documentation.
Additional documentation on adding the CBAs to the WebSphere local bundle repository, extending the MDM server EBAs, starting and stopping the WAS server is available on the MDM Developer Site.