The design, creation, and maintenance of data transformations can require a significant amount of work. The prevalent approach to data transformation today is programmatic mapping. There are many tools on the market, such as XSLT tools from multiple vendors, mapping editors from IBM WebSphere Message Broker (Message Broker) and WebSphere Process Server (Process Server), and so on. Unfortunately, applicability of these tools is quite limited. They are great for demonstration of field-to-field transformation. For real-life examples requiring structural data transformations (such as structures merging and splitting), you usually still need to resort to general-purpose programming languages such as Java™, C, or ESQL.
To make matters worse, the actual implementation is typically done by developers, regardless of what is used for transformation implementation (specialized tools or general purpose programming languages). Implementing data transformations is already complex, and the following issues with this approach make it even more difficult.
- Knowledge of the data, both enterprise semantic models and application data models, usually
resides with business analysts, not developers.
Business analysts need to somehow convey transformation definitions (requirements) to developers. This is traditionally done using Excel spreadsheets, which are far from ideal for requirements specification. Every company is forced to introduce their own Excel-compatible notation.
- Ambiguity of Excel notation often leads to misinterpretation of transformation rules, resulting in prolonged development and testing cycles for transformation implementation.
- Excel-based definitions are not directly translated in the code, but are
interpreted by humans. It is extremely difficult to keep the two in synch.
Any transformation defect requires potentially lengthy investigation to determine whether it is a definition error, interpretation error (misunderstanding on the developer's part), or a code defect.
- Introducing changes in the transformation becomes very difficult.
Transformation definitions are owned by the business analysts. They tend to make changes in the original spreadsheets, which go back to developers. Depending on the organization of the transformation code, even seemingly trivial changes can require significant modifications in the implementation code.
- Developers often view data transformations as an integral part of the business logic, and consequently intertwine its implementation with the implementation of other business logic. This approach makes introduction of transformation changes even more complex. The impact of the changes can spread throughout an implementation.
In software engineering today, decomposition and encapsulation are widely adopted approaches to deal with the complexities. In our case, it means externalizing data transformation into separate components, with well defined interfaces (similar to the approach taken by Process Server), and separating it from the rest of the business logic. Though this approach simplifies introducing changes, it does not solve the bigger data transformation issue: requirements transfer between business analysts and developers.
An ideal solution to the problem would be a programmable transformation engine that provides tooling for business analysts to define transformations, and for developers to integrate them into their implementations. Requirements ambiguity would be eliminated by putting transformation definitions directly in the hands of the business analysts who actually design them. It also eliminates multiple implementations of the same transformations (Excel spreadsheets and code implementations), eliminating many of today's inconsistencies. IBM's WebSphere Transformation Extender (TX) is a tool that can be used for just such a purpose.
Introduced in 1994, TX is currently used in many financial, insurance, and major telecom institutions. Unfortunately there isn't a lot of technical information about it on the Web (even if you search on its previous names, DataStage TX and Mercator) besides IBM's product Web site (see Resources). This article is based on a quick tool evaluation, and is by no means complete.
Figure 1 shows the major components of WebSphere TX.
Figure 1. Major components of WebSphere TX
- Data definition
- Creation of the metadata about the data to be transformed. Type Designer, which is part of WebSphere TX, supports automatic creation of the metadata from the following data formats:
- XML schemas
- Cobol copybooks
- Database tables
- Java classes
- Positional files, including EDI, and so on
- Transformation rules design
- Creation of rules, defining transformation of the source structures into destination structures. WebSphere TX provides a wide set of standard transformation rules,
ranging from simple field moves to arithmetic and string operations, sorting, and more.
Map Designer, part of WebSphere TX, supports visual authoring of the transformation rules. It is integrated with transformation debugger and profiler, supporting productivity of rules designers.
- Transformation execution
- Run time support for transformation execution. Transformation map, created during the previous step, can be executed in multiple ways ranging from the standalone rules Launcher to SDK, allowing users to embed data transformation directly in the code.
The next section shows the WebSphere TX capabilities with a couple of examples.
The two examples are:
Input is the same in both cases. It is a set of XML files, with the first two defining information about Automobiles (Listing 1, Listing 2), and the other two defining information about Insurances (Listing 3, Listing 4).
In this example, the transformation result should be an XML document
adhering to the XML schema in Listing 5. An autopolicy element in the resulting
XML needs to be created for every VIN number that occurs in either document in
Listing 2 or Listing 4.
Listing 1. Automobiles.xsd
<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://examples.boris.com/Automobiles"
xmlns:tns="http://examples.boris.com/Automobiles">
<complexType name="automobile">
<all>
<element name="VinNumber" type="string"></element>
<element name="Make" type="string"></element>
<element name="Model" type="string"></element>
<element name="Year" type="integer"></element>
</all>
</complexType>
<complexType name="automobiles">
<sequence minOccurs="1" maxOccurs="unbounded">
<element name="automobile" type="tns:automobile"></element>
</sequence>
</complexType>
<element name="AutomobileList" type="tns:automobiles"></element>
</schema>.
|
Listing 2. Automobiles.xml
<?xml version="1.0" encoding="UTF-8"?>
<tns:AutomobileList xmlns:tns="http://examples.boris.com/Automobiles"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
<automobile>
<VinNumber>123456</VinNumber>
<Make>Buick</Make>
<Model>LeSabre</Model>
<Year>1993</Year>
</automobile>
<automobile>
<VinNumber>123457</VinNumber>
<Make>Honda</Make>
<Model>Accord</Model>
<Year>1995</Year>
</automobile>
<automobile>
<VinNumber>123458</VinNumber>
<Make>Toyota</Make>
<Model>Corolla</Model>
<Year>1997</Year>
</automobile>
<automobile>
<VinNumber>123459</VinNumber>
<Make>Toyota</Make>
<Model>Camry</Model>
<Year>2000</Year>
</automobile>
<automobile>
<VinNumber>123450</VinNumber>
<Make>Ford</Make>
<Model>Escort</Model>
<Year>2002</Year>
</automobile>
</tns:AutomobileList>
|
Listing 3. Insurances.xsd
<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://examples.boris.com/Insurances"
xmlns:tns="http://examples.boris.com/Insurances">
<complexType name="InsuredAutomobiles">
<sequence minOccurs="1" maxOccurs="unbounded">
<element name="VIN" type="string"></element>
</sequence>
</complexType>
<complexType name="Policy">
<all>
<element name="Number" type="string"></element>
<element name="Agent" type="string"></element>
<element name="automobiles"
type="tns:InsuredAutomobiles"></element>
</all>
</complexType>
<complexType name="Policies">
<sequence minOccurs="1" maxOccurs="unbounded">
<element name="policy" type="tns:Policy"></element>
</sequence>
</complexType>
<element name="Policies" type="tns:Policies"></element>
</schema>
|
Listing 4. Insurances.xml
<?xml version="1.0" encoding="UTF-8"?>
<tns:Policies xmlns:tns="http://examples.boris.com/Insurances"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >
<policy>
<Number>654321</Number>
<Agent>SuperAgent</Agent>
<automobiles>
<VIN>123458</VIN>
<VIN>123450</VIN>
<VIN>123459</VIN>
</automobiles>
</policy>
<policy>
<Number>654322</Number>
<Agent>UnderAgent</Agent>
<automobiles>
<VIN>123457</VIN>
</automobiles>
</policy>
<policy>
<Number>654323</Number>
<Agent>QuasiAgent</Agent>
<automobiles>
<VIN>123455</VIN>
</automobiles>
</policy>
</tns:Policies>
|
Listing 5. Autopolicies.xsd
<?xml version="1.0" encoding="UTF-8"?> <schema xmlns="http://www.w3.org/2001/XMLSchema" targetNamespace="http://examples.boris.com/InsuredAutomobiles" xmlns:tns="http://examples.boris.com/InsuredAutomibles"> <complexType name="AutoPolicy"> <sequence> <element name="PolicyNumber" type="string" minOccurs="0" maxOccurs="1"></element> <element name="Agent" type="string" minOccurs="0" maxOccurs="1"></element> <element name="VIN" type="string" minOccurs="0 maxOccurs="1"></element> <element name="Make" type="string" minOccurs="0 maxOccurs="1"></element> <element name="Model" type="string" minOccurs="0 maxOccurs="1"></element> <element name="Year" type="string" minOccurs="0" maxOccurs="1"></element> </sequence> </complexType> <complexType name="AutoPolicies"> <sequence minOccurs="1" maxOccurs="unbounded"> <element name="Policy" type="tns:AutoPolicy"></element> </sequence> </complexType> <element name="Autopolicies" type="tns:AutoPolicies"></element> </schema> |
In the XML to Java mapping example, the output of transformation is an Autopolicies Java class, as shown in Listing 6. In this case, the result of the transformation can be used directly by the Java code (unlike the first example, where the result is an XML stream and additional marshaling is required).
An autopolicy class needs to be present for every VIN number that occurs in either document
in Listing 2 or Listing 4.
Listing 6. Autopolicy and Autopolicies classes
package com.boris.support.transform;
public class autoPolicy {
private String PolicyNumber = null;
private String Agent = null;
private String VIN = null;
private String Make = null;
private String Model = null;
private String Year = null;
public String getAgent() {
return Agent;
}
public void setAgent(String agent) {
Agent = agent;
}
public String getMake() {
return Make;
}
public void setMake(String make) {
Make = make;
}
public String getModel() {
return Model;
}
public void setModel(String model) {
Model = model;
}
public String getPolicyNumber() {
return PolicyNumber;
}
public void setPolicyNumber(String policyNumber) {
PolicyNumber = policyNumber;
}
public String getVIN() {
return VIN;
}
public void setVIN(String VIN) {
VIN = VIN;
}
public String getYear() {
return Year;
}
public void setYear(String year) {
Year = year;
}
}
package com.boris.support.transform;
import java.util.Iterator;
import java.util.LinkedList;
public class autoPolicies {
private LinkedList autopolicies = new LinkedList();
public void addautopolicy(autoPolicy p){
autopolicies.add(p);
}
public Iterator getPolicies(){
return autopolicies.iterator();
}
}
|
The examples are implemented using the following steps:
WebSphere TX V 8.1 and Java SDK were used for transformation invocation.
Type trees are created using WebSphere TX Type Designer, as shown in Figure 2. It lets you directly import COBOL copybooks, XML schemas or DTDs, Java classes, Java Message Service (JMS) definitions, or CORBA IDL.
A separate tool, Database Interface Designer, allows connecting to the existing database to create type trees out of the database tables, queries, or stored procedures.
Figure 2. WebSphere TX Type Designer
In the case of the XML schema that we're using as the input and output parameters in the first scenario, the import wizard asks for the schema file location, language (multiple languages are supported), and parser type. Two parser types are supported by the tool: classic and Xerces. When choosing the parser type, keep in mind that Xerces generally works faster but requires schema definition at the run time for the input validation. Xerces is used in the examples. The author created three type definitions for XSDs (Listing 1, Listing 3, and Listing 5).
In the case of Java, output for the XML to Java example JAR file containing processed classes has to be defined in the dtx.ini file (or on CLASSPATH) so that the tool can find them. For the example, three type trees for Java classes were created (Listing 6). For the Autopolicies class, two different type trees were created to simplify transformation implementation.
Type Designer also lets you manually create type trees that can be used in the transformations. This option is used to create the VINs type tree used in the transformations.
All of the type definitions are in the code in Downloads.
Transformation maps are created using the Map Designer tool shown in Figure 3.
Figure 3. WebSphere TX Map designer tool
A typical implementation of mapping in WebSphere TX involves a loop through all of the instances of the structure that needs to be created, and building every structure. In our case this translates to creating the list of the VIN numbers that are unique between two input documents (Listing 2, Listing 4), then building Autopolicy structure (for XML to XML mapping) or Autopolicy class (for XML to Java mapping) for every VIN number.
WebSphere TX distinguishes two types of maps:
- Execution
- A standalone unit of execution that defines a complete transformation. It is a unit of compilation, debugging, and profiling. WebSphere TX transformation contains at least one (or more) execution maps. An execution map can invoke another execution map.
- Functional
- Similar to a subroutine, it maps a portion of data at a time. A functional map belongs to a particular execution map.
A map, either execution or functional, contains input and output cards -- input/output parameters. A functional map can contain only one output card. Each output card represents a single transformation, which can use data from both input and output (defined prior to this one) cards.
Transformation for XML to XML mapping contains two execution maps. The main one shown in Figure 3 calculates VIN sets for both automobiles and insurances, and invokes makePolicies map, as shown in Figure 4, which builds Autopolicy structure (using the F_EACH_PPLICY functional map) for every unique VIN. The complete transformation implementation for this example is in the code in Downloads.
Figure 4. MakePolicies map
A map debugger, part of Map Designer, allows step by step execution of the transformation and viewing the results of each step, as shown in Figure 5.
Figure 5. Debugging Map
Finally, Map Designer lets you run a created map and display both results of the execution and its profiling information, as shown in Figure 6.
Figure 6. Execution results
Implementation of the XML to Java mapping example, as shown in Figure 7, is very similar to the first one.
Again we use the first executable map (InsuredAutos_java) to collect VINs used in both input files. The second executable map (R_MakePolicies_Java) first builds the container class, based on the key passed to the map, and then loops through all of the VINs.
Unlike the first example, where the loop was on the sequence of the Autopolicy element, in this case it is necessary to
use a list of VINs to ensure looping.
Within each step of a loop you must do two things: create an autopolicy class, and attach it to the container class. Because a functional map can have only one output card, a third executable map (R_BuildClasses_java) was introduced, which is invoked for every VIN number to do the actual work. Instances of Java classes in WebSphere TX are managed in a hash map that is associated with the map execution. A specific object, or class instance, has a unique ID associated with it. This allows passing of a reference to the object (between maps or between cards) by passing an ID of an object.
Figure 7. Transformation map for XML to Java example
In this case the execution result is a Java class, so the tool does not allow you to view execution results directly (similar to Figure 6). One way to analyze the results of execution is by looking at the log file, as shown in Listing 7.
Listing 7. Log file of the transformation execution
<2712-3412>: Trace started
<2712-3412>: Trace file | ContainerClass.log
<2712-3412>: Trace append | false
<2712-3412>: Trace error | false
<2712-3412>: Serialize | false
<2712-3412>: Deserialize | false
<2712-6268>: [compareConnection]
<2712-6268>: [compareConnection] (rc = 0) OK
<2712-6268>: [onNotify]
<2712-6268>: | OnNotify called for PUTSTART
<2712-6268>: [onNotify] (rc = 0) OK
<2712-6268>: [put]
<2712-6268>: | [performClassOperation]
<2712-6268>: | | Class name : com.boris.support.transform.autoPolicies
<2712-6268>: | | Include inherited members : no
<2712-6268>: | | Return public fields : no
<2712-6268>: | | Reference to the top level object : IDContainer
<2712-6268>: | | Instantiating root object
<2712-6268>: | | [instantiateObjectFromConstructorNode]
<2712-6268>: | | | Object with reference IDContainer in map
instance 3
obtained from the pool successfully
<2712-6268>: | | [instantiateObjectFromConstructorNode] (rc = 0) OK
<2712-6268>: | | Root object instantiated
<2712-6268>: | | Number of methods to call : 1
<2712-6268>: | | Calling methods/obtaining public fields
<2712-6268>: | | [invokeMethodsAndGenerateOutput]
<2712-6268>: | | | Number of methods to invoke : 1
<2712-6268>: | | | Preparing method : addautopolicy
<2712-6268>: | | | [convertXmlNodeToClass]
<2712-6268>: | | | [convertXmlNodeToClass] (rc = 0) OK
<2712-6268>: | | | [convertXmlNodeToObject]
<2712-6268>: | | | | Retrieving object from the pool,
MapInstance:3 Reference:123455
<2712-6268>: | | | | Object obtained from the pool successfully
<2712-6268>: | | | [convertXmlNodeToObject] (rc = 0) OK
<2712-6268>: | | |Method parameter 0 : com.boris.support.transform.autoPolicy
prepared successfully
<2712-6268>: | | [invokeMethodsAndGenerateOutput] (rc = 0) OK
<2712-6268>: | | Methods called/public fields obtained
<2712-6268>: | [performClassOperation] (rc = 0) OK
<2712-6268>: [put] (rc = 0) OK
<2712-6268>: [onNotify]
<2712-6268>: | OnNotify called for PUTSTOP
<2712-6268>: | Closing trace file and exiting function
<2712-6268>: [onNotify] (rc = 0) OK
|
For transformation implementation, Java APIs provided as a part of WebSphere TX SDK were used to build Java implementation of the examples. This Java code can then be used, for example, as part of WebSphere Application Server implementation.
The workhorse of the XML to XML mapping example is the StreamMapper class, as shown in Listing 8. It is a generic class using the WebSphere TX run time engine to do transformation of multiple input streams into multiple output streams.
Listing 8. Streammapper implementation
package com.boris.websphereTX;
import java.io.InputStream;
import java.io.ByteArrayInputStream;
import com.ibm.websphere.dtx.dtxpi.MAdapter;
import com.ibm.websphere.dtx.dtxpi.MCard;
import com.ibm.websphere.dtx.dtxpi.MConstants;
import com.ibm.websphere.dtx.dtxpi.MMap;
import com.ibm.websphere.dtx.dtxpi.MStream;
// This class takes inputs as stream of data and returns
// mapping results as a set of streams.
public class streamMapper {
private static boolean initialized = false;
private boolean ready = false;
public streamMapper(){
if(!initialized){
try{
long start = System.currentTimeMillis();
// Initialize the API
MMap.initializeAPI(null);
long duration = System.currentTimeMillis() - start;
System.out.println("WebSphere TX APIs initialization " +
duration + " millisec");
initialized = true;
ready = true;
}
catch( Exception e ){
System.out.println("Error Initializing WebSphere TX APIs");
e.printStackTrace();
}
}
}
public boolean isReady() {
return ready;
}
public InputStream [] map(InputStream mapStream, InputStream[] inputStreams){
// Make sure we are ready
if(!ready){
System.out.println("WebSphere TX APIs are not initialized");
return null;
}
MMap map = null;
int nOutputCards = 0;
try{
// peg the time
long start = System.currentTimeMillis();
byte[] mapData = new byte[mapStream.available()];
mapStream.read(mapData);
map = new MMap("Map", null, mapData);
// Override the input cards
int nInputs = inputStreams.length;
int nInputCards = map.getInputCardCount();
if(nInputs != nInputCards){
System.out.println("Map inputs mismatch. Map requires " +
nInputCards + " received " + nInputs);
// Clean up
map.unload();
return null;
}
for(int i=0; i < nInputs; i++){
InputStream current = inputStreams[i];
byte[] inputData = new byte[current.available()];
current.read(inputData);
MCard card = map.getInputCardObject(i+1);
card.overrideAdapter(null, MConstants.MPI_ADAPTYPE_STREAM);
MAdapter adapter = card.getAdapter();
MStream stream = adapter.getOutputStream();
stream.write(inputData, 0, inputData.length);
}
// Override output cards
nOutputCards = map.getOutputCardCount();
for(int i = 0; i < nOutputCards; i++){
MCard card = map.getOutputCardObject(i+1);
card.overrideAdapter(null, MConstants.MPI_ADAPTYPE_STREAM);
}
long duration = System.currentTimeMillis() - start;
System.out.println("Loading map and inputs " + duration + " millisec");
// Run the map
start = System.currentTimeMillis();
map.run();
duration = System.currentTimeMillis() - start;
System.out.println("Map execution " + duration + " millisec");
// Check the return status (need check for success)
int iRC = map.getIntegerProperty(MConstants.MPIP_OBJECT_ERROR_CODE, 0);
if(iRC != 0){
String szMsg = map.getTextProperty(MConstants.MPIP_OBJECT_ERROR_MSG, 0);
System.out.println("Map status: " + szMsg + " (" + iRC + ")");
}
//Get results
start = System.currentTimeMillis();
InputStream [] results = new InputStream [nOutputCards];
for(int i = 0; i < nOutputCards; i++){
MCard card = map.getOutputCardObject(i+1);
MAdapter adapter = card.getAdapter();
MStream stream = adapter.getInputStream();
int streamSize = stream.getSize();
byte[] data = new byte[streamSize];
stream.seek(0, MConstants.MPI_SEEK_SET);
for(int offset = 0, len = streamSize; len > 0; len -= offset){
int bRead = stream.read(data,offset,len);
len -= bRead;
offset += bRead;
}
results[i] = new ByteArrayInputStream(data);
}
duration = System.currentTimeMillis() - start;
System.out.println("Getting results " + duration + " millisec");
return results;
}
catch( Exception e ){
System.out.println("Mapping Error");
e.printStackTrace();
return null;
}
finally{
// Clean up
long start = System.currentTimeMillis();
try{
map.unload();
MMap.terminateAPI();
}
catch(Exception e){}
long duration = System.currentTimeMillis() - start;
System.out.println("Clean up " + duration + " millisec");
}
}
}
|
The class constructor initializes WebSphere TX run time APIs. For it to work, WebSphere TX DLL (in the case of Windows) has to be defined on the execution path. The actual mapping is done by invoking the map method, which takes transformation map (in the form of the input stream) and an array of the input streams of map inputs and returns the array of the output streams of map outputs. Transformation map, passed to this method, can invoke other execution maps. As long as all of these maps are available to the run time, it will load them as they are invoked. As a result only the map that starts the execution has to be passed in.
After the map is loaded, the implementation goes through all of the input cards for the map and overrides their adapters to get the input from the streams, which are passed in. Because we used XML inputs, and the Xerces parser to create type trees in addition to the XML inputs, XSDs for these inputs should be available to the run time as well. The XSDs are used by the run time for the validation of the input parameters.
After running the map, the execution result is checked. All of the execution errors are reported. If there are no errors, the implementation goes through all of the map output cards and extracts output streams from their adapters. These output streams are returned back. See Listing 9 below.
Listing 9. Test for the Streammapper class
package com.boris.websphereTX.test;
import java.io.FileInputStream;
import java.io.InputStream;
import com.boris.websphereTX.*;
public class streamMapTest {
public static void main(String[] args) {
FileInputStream mis1 = null;
FileInputStream mis2 = null;
FileInputStream param1 = null;
FileInputStream param2 = null;
try{
// Load map file
Mis1 = new FileInputStream("InsuredAutos_ori.mmc");
Mis2 = new FileInputStream("R_MakePolicies_ori.mmc");
param1 = new FileInputStream("Automobiles.xml");
param2 = new FileInputStream("Insurances.xml");
}
catch( Exception e ){
System.out.println("Error loading data");
e.printStackTrace();
System.exit(1);
}
InputStream[] mapData = new InputStream[2];
mapData[0] = param1;
mapData[1] = param2;
// create mapper
streamMapper mapper = new streamMapper();
// map
if(mapper.isReady()){
InputStream[] results = mapper.map(mis1,mapData);
// Print results
int nOutputs = results.length;
for(int i = 0; i < nOutputs; i++){
System.out.println("Execution result " + i + "\n");
InputStream output = results[i];
try{
int streamSize = output.available();
byte[] data = new byte[streamSize];
for(int offset = 0, len = streamSize; len > 0; len -= offset){
int bRead = output.read(data,offset,len);
len -= bRead;
offset += bRead;
}
System.out.println(new String(data));
}
catch( Exception e ){
System.out.println("Error Reading result stream");
e.printStackTrace();
}
}
}
else{
System.out.println("Problem loading WebSphere TX APIs");
}
//clean up
try{
mis1.close();
mis2.close();
param1.close();
param2.close();
}
catch( Exception e ){
System.out.println("Error cleaning up");
e.printStackTrace();
System.exit(1);
}
}
}
|
Using a simple driver for the Streammapper class (in Listing 9 above) produces the results in Listing 10.
Listing 10. Execution results for Streammapper
WebSphere TX APIs initialization 219 millisec
Loading map and inputs 31 millisec
Map execution 156 millisec
Getting results 0 millisec
Clean up 0 millisec
Execution result 0
123456,123457,123458,123459,123450,
Execution result 1
123458,123450,123459,123457,123455,
Execution result 2
<?xml version="1.0" encoding="UTF-8"?>
<tns:Autopolicies xmlns:tns="http://examples.boris.com/InsuredAutomobiles">
<Policy>
<VIN>123456</VIN>
<Make>Buick</Make>
<Model>LeSabre</Model>
<Year>1993</Year>
</Policy>
<Policy>
<PolicyNumber>654322</PolicyNumber>
<Agent>UnderAgent</Agent>
<VIN>123457</VIN>
<Make>Honda</Make>
<Model>Accord</Model>
<Year>1995</Year>
</Policy>
<Policy>
<PolicyNumber>654321</PolicyNumber>
<Agent>SuperAgent</Agent>
<VIN>123458</VIN>
<Make>Toyota</Make>
<Model>Corolla</Model>
<Year>1997</Year>
</Policy>
<Policy>
<PolicyNumber>654321</PolicyNumber>
<Agent>SuperAgent</Agent>
<VIN>123459</VIN>
<Make>Toyota</Make>
<Model>Camry</Model>
<Year>2000</Year>
</Policy>
<Policy>
<PolicyNumber>654321</PolicyNumber>
<Agent>SuperAgent</Agent>
<VIN>123450</VIN>
<Make>Ford</Make>
<Model>Escort</Model>
<Year>2002</Year>
</Policy>
<Policy>
<PolicyNumber>654323</PolicyNumber>
<Agent>QuasiAgent</Agent>
<VIN>123455</VIN>
</Policy>
</tns:Autopolicies>
|
The workhorse of the XML to Java example is the ClassMapper class, as shown in Listing 11. It is a generic class using the WebSphere TX run time engine to do a transformation of multiple input streams into a class file.
Listing 11. ClassMapper implementation
package com.boris.websphereTX;
import java.io.InputStream;
import com.ibm.websphere.dtx.dtxpi.MAdapter;
import com.ibm.websphere.dtx.dtxpi.MCard;
import com.ibm.websphere.dtx.dtxpi.MConstants;
import com.ibm.websphere.dtx.dtxpi.MMap;
import com.ibm.websphere.dtx.dtxpi.MStream;
import com.ibm.websphere.dtx.dtxpi.tools.MObjectPool;
//This class takes inputs as stream of data and returns
//mapping results as a java class. It can take several
//maps. It always uses the first map for inputs.
public class classMapper{
private static int mapInstance = 1;
private static boolean initialized = false;
private boolean ready = false;
public classMapper(){
if( !initialized ){
try{
long start = System.currentTimeMillis();
// Initialize the API
MMap.initializeAPI(null);
long duration = System.currentTimeMillis() - start;
System.out.println("Websphere TX APIs initialization " +
duration + " millisec");
initialized = true;
ready = true;
}
catch( Exception e ){
System.out.println("Error Initializing Websphere TX APIs");
e.printStackTrace();
}
}
}
public boolean isReady(){
return ready;
}
public Object map(InputStream mapStream, InputStream[] inputStreams,
String cName, String key){
// Make sure we are ready
if( !ready ){
System.out.println("Websphere TX APIs are not initialized");
return null;
}
MMap map = null;
try{
// peg the time
long start = System.currentTimeMillis();
// Create a map
byte[] mapData = new byte[mapStream.available()];
mapStream.read(mapData);
map = new MMap("Map", null, mapData);
// Set map instance to value other than 0
map.setIntegerProperty(MConstants.MPIP_MAP_INSTANCE, 0, mapInstance++);
map.setIntegerProperty(MConstants.MPIP_MAP_DESTROY_OBJECT_POOL, 0,
MConstants.MPI_FALSE);
// Override the input cards
int nInputs = inputStreams.length;
int nInputCards = map.getInputCardCount();
if( nInputs != (nInputCards - 1) ){
System.out.println("Map inputs mismatch. Map requires " +
nInputCards + " received " + nInputs);
// Clean up
map.unload();
return null;
}
for( int i=0; i <= nInputs; i++ ){
MCard card = map.getInputCardObject(i+1);
card.overrideAdapter(null, MConstants.MPI_ADAPTYPE_STREAM);
MAdapter adapter = card.getAdapter();
MStream stream = adapter.getOutputStream();
if(i < nInputs){
InputStream current = inputStreams[i];
byte[] inputData = new byte[current.available()];
current.read(inputData);
stream.write(inputData, 0, inputData.length);
}
else{
// The last inputStream is the key
byte[]inputData = key.getBytes();
stream.write(inputData, 0, inputData.length);
}
}
long duration = System.currentTimeMillis() - start;
System.out.println("Loading map and inputs " + duration + " millisec");
// Run the map
start = System.currentTimeMillis();
//Do not destroy the object pool
map.run();
duration = System.currentTimeMillis() - start;
System.out.println("Map execution " + duration + " millisec");
int currMap = map.getIntegerProperty(MConstants.MPIP_MAP_INSTANCE, 0);
// Check the return status (need check for success)
int iRC = map.getIntegerProperty(MConstants.MPIP_OBJECT_ERROR_CODE, 0);
if( iRC != 0 ){
String szMsg = map.getTextProperty(MConstants.MPIP_OBJECT_ERROR_MSG, 0);
System.out.println("Map status: " + szMsg + " (" + iRC + ")");
}
//Get results
start = System.currentTimeMillis();
//Get Object pool
// Get an Object from the Pool
Object results = null;
MObjectPool pool = MObjectPool.getInstance();
if( pool.isMapInstanceHashtableInThePool(currMap) )
results = pool.retrieveObjectFromThePool(currMap,key);
duration = System.currentTimeMillis() - start;
System.out.println("Getting results " + duration + " millisec");
return results;
}
catch( Exception e ){
System.out.println("Mapping Error");
e.printStackTrace();
return null;
}
finally{
// Clean up
long start = System.currentTimeMillis();
try{
map.unload();
}
catch( Exception e ){}
long duration = System.currentTimeMillis() - start;
System.out.println("Clean up " + duration + " millisec");
}
}
}
|
The structure of this class is identical to the structure of the Streammapper class in Listing 8.
The difference here is that it obtains results not from the output adapter but from the TX engine's object pool.
This is achieved by passing an additional parameter to the map—a key for the object that needs to be returned as a result
of the execution.
To locate the right part of the Object Pool to retrieve the resulting object,
a unique non-zero map number is needed. You can do this by using the static mapInstance variable,
which is incremented during every class invocation, and assigning this number to the running map instance.
After running the map, its execution result is checked. All of the execution errors are reported. If there are no errors, the implementation extracts the object with the predefined key from the map. This object is then returned back to the method invoker.
Listing 12. Test for classMapper class
package com.boris.websphereTX.test;
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.Iterator;
import com.ibm.websphere.dtx.dtxpi.MMap;
import com.boris.websphereTX.*;
import com.cna.support.transform.autoPolicies;
import com.cna.support.transform.autoPolicy;
public class classMapTest{
/**
* @param args
*/
public static void main(String[] args){
FileInputStream mis1 = null;
InputStream param1 = null;
InputStream param2 = null;
try{
// Load map file
mis1 = new FileInputStream("InsuredAutos_java.mmc");
param1 = new FileInputStream("Automobiles.xml");
param2 = new FileInputStream("Insurances.xml");
}
catch( Exception e ){
System.out.println("Error loading data");
e.printStackTrace();
System.exit(1);
}
InputStream[] mapData = new InputStream[2];
mapData[0] = param1;
mapData[1] = param2;
// create map
classMapper mapper = new classMapper();
// map
if( mapper.isReady() ){
Object results = mapper.map(mis1,mapData,
"com.cna.support.transform.autoPolicies", "IDContainer");
System.out.println(results);
if( results != null ){
autoPolicies policies = (autoPolicies)results;
System.out.println("Resulting set of policies");
Iterator pset = policies.getPolicies();
while( pset.hasNext() ){
autoPolicy policy = (autoPolicy)pset.next();
System.out.println(
"Number " + policy.getPolicyNumber() +
"Agent " + policy.getAgent() +
" Vin " + policy.getVin() +
" Make " + policy.getMake() +
" Model " + policy.getModel() +
" Year " + policy.getYear());
}
}
}
else{
System.out.println("Problem loading Websphere TX APIs");
}
//clean up
try{
mis1.close();
param1.close();
param2.close();
MMap.terminateAPI();
}
catch( Exception e ){
System.out.println("Error cleaning up");
e.printStackTrace();
System.exit(1);
}
}
}
|
Using a simple driver for the ClassMapper class (Listing 12) produces the results shown here in
Listing 13.
Listing 13. Execution results for ClassMapper
WebSphere TX APIs initialization 219 millisec Loading map and inputs 0 millisec Map execution 625 millisec Getting results 0 millisec Clean up 0 millisec com.boris.support.transform.autoPolicies@69306930 Resulting set of policies Number Agent VIN 123456 Make Buick Model LeSabre Year 1993 Number 654322Agent UnderAgent VIN 123457 Make Honda Model Accord Year 1995 Number 654321Agent SuperAgent VIN 123458 Make Toyota Model Corolla Year 1997 Number 654321Agent SuperAgent VIN 123459 Make Toyota Model Camry Year 2000 Number 654321Agent SuperAgent VIN 123450 Make Ford Model Escort Year 2002 Number 654323Agent QuasiAgent VIN 123455 Make Model Year |
For this code to work, the command string in Listing 14 has to be passed to the Java Virtual Machine.
Listing 14. Java command string for the execution
-Dmpi.map.destroy_object_pool=false |
Summary: Map once, transform everywhere
The WebSphere TX transformation engine is available on multiple platforms, including Windows®, different versions of UNIX®, and the mainframe. It also provides a software development kit (SDK) for invoking the transformation engine using Java, C, C+, CORBA, Remote Method Invocation (RMI), and so on. A WebSphere TX node for Message Broker is also available (see Resources).
Such wide platform and language coverage, along with complete separation of design and run-time execution, provides architects the freedom to move data transformation processing to the platform most appropriate for a given implementation, without any changes to the transformation logic. Depending on the architectural requirements, transformation can be:
- Embedded in the WebSphere application server or mainframe transactions execution, optimizing performance
- Bundled with the WebSphere Business Integration processing, preserving current topology
- Externalized to the separate host, maximizing decoupling
The author wishes to extend many thanks to Michael Hudson on the WebSphere TX team for providing clear insight about how WebSphere TX works and for helping to put the code examples together.
| Description | Name | Size | Download method |
|---|---|---|---|
| WebSphere TX code | examplesTX.zip | 33KB | HTTP |
| Java code | examplesJava.zip | 30KB | HTTP |
Information about download methods
Learn
- WebSphere Transformation Extender product overview.
- WebSphere Transformation Extender for Message Broker product overview.
- IBM on demand demos to learn about various software products and technologies from IBM.
- Stay current with
developerWorks technical events and webcasts.
- In the
Architecture area on developerWorks, get the resources you need
to advance your skills in the IT architecture arena.
-
Browse the technology bookstore for books on these and other technical topics.
Get products and technologies
-
Get started now! Download more IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere.
Discuss
- Participate in the discussion forum.
- Check out
developerWorks
blogs and
get involved in the
developerWorks community.
Boris Lublinsky has more than 25 years of experience in software engineering and technical architecture. For the past several years he has focused on enterprise architecture, SOA, and process management. Dr. Lublinsky is a technical speaker and author, with more than 50 technical publications in different magazines, including Avtomatika i telemechanica, IEEE Transactions on Automatic Control, Distributed Computing, Nuclear Instruments and Methods, Java Developer's Journal, XML Journal, Web Services Journal, JavaPro Journal, Enterprise Architect Journal, and EAI Journal. Currently Dr. Lublinsky works as a principal architect at Herzum Software, where he is responsible for the design of component-based development factories and consulting with customers on component-based development and SOA. Boris can be reached at blublinsky@herzumsoftware.com.
Comments (Undergoing maintenance)
Table of contents
Next steps from IBM
Rational Tester for SOA Quality is an automated web services testing and SOA testing tool that supports Web services SOAP, HTTP(S), JMS, WS-Security, UDDI, and RESTful standards.
- Try: The no-charge Rational Service Tester for SOA Quality helps quality assurance professionals validate web service based SOA application..
- Article: See the whats new with Rational Service Tester for SOA Quality, and its Generic Service Client, which provides a single client to interact with any SOA service type.
- Tutorial: Get hands-on experience with this Transport for SOAP feature tutorial using Rational Tester for SOA Quality.
- Buy: Rational Service Tester for SOA Quality
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
