Integrating systems of highly complex message formats with WebSphere Message Broker

This article uses a real-world example from a banking system to show you how to use WebSphere Message Broker data modeling to reduce the cost of integrating highly complex message formats.

Dr. Hesham Soultan (hsoultan@eg.ibm.com), IT Architect, Cairo Technology Development Center, IBM

Photo of Dr. Hesham SoultanDr. Hesham Soultan is an IT Architect at the Cairo Technology Development Center in Egypt. His development expertise includes automotive embedded software, machine vision, machine learning, and business integration. Since joining the IBM Business Integration team, he has worked on WebSphere DataPower, WebSphere Message Broker, and WebSphere MQ. You can contact Hesham at hsoultan@eg.ibm.com.



15 August 2012

Also available in Chinese

Introduction

Data transformation and format conversion are core tasks of business integration. For data transformation, IBM® WebSphere® Message Broker (hereafter called Message Broker) provides its Mapping node, and for the physical format conversions, data modeling using the message sets plays an important role, in conjunction with different message parsers.

After creating the message model, you can use it to convert the format of any request message received by Message Broker to another physical format required by a back-end system. In the other direction, you can use the message model to parse the response message from the back-end system to generate the logical tree of the message, and easily convert it back to the format of the front-end system. The main physical formats that are widely used and supported by Message Broker are the text, binary, and XML. Text format can be fixed-length, tagged delimited, or delimited only, and delimited elements can either be ordered or unordered. This article has four sections:

Using message sets to reduce code size

Sometimes you need to integrate with a back-end system that has a complex interface. In that case, the message modeling capability in the Message Broker can significantly speed up the integration process, as shown in the example used in this article. The example architecture is shown in Figure 1. The banking system requires request and response messages for some functions to be in a complex fixed-length text format, and in a tagged delimited unordered complex text format for other functions.

Figure 1. Example architecture
Example architecture

In such cases, where you do not know the strength of message modeling in Message Broker, you would need to write complex custom code to do both the physical format conversion and the message parsing. Creating message models and then using the MRM parser gives you the flexibility to generate or parse any message, regardless of its complexity. Furthermore, using the proven code in the MRM parser increases the overall reliability of your middleware solution. Figure 2 below shows the banking system adapter message flow as it receives an XML MQ message from the front end through the MQInput node. To convert this message to text format according to the message model, you simply write the three lines of ESQL code shown in Listing 1 to select the message model. Any created logical tree (the last two statements in the listing create the MRM tree) is converted to the selected physical format. The code is associated with the Message Broker compute node Extract backend Msg.

Figure 2. Banking system adapter message flow
Banking system adapter message flow

The response from the back-end banking system is read by the MQGet node, and then the compute node MapToesbXML is parsed by some simple ESQL code. Just one statement calls the MRM parser according to the message model parameters, as shown in Listing 2 below.

Listing 1. Back-end adapter request generation code
DECLARE inRef REFERENCE TO InputRoot.XMLNSC.BE_esbXML.Body;

SET OutputRoot.Properties.MessageSet='MRM_Binary_CSV';
SET OutputRoot.Properties.MessageType=FIELDNAME(inRef.*[1]);
SET OutputRoot.Properties.MessageFormat='Text_CSV';
		
-- <<<<Copy the Backend message from the BE_esbXML>>>> 
CREATE LASTCHILD OF OutputRoot DOMAIN ('MRM');  
SET OutputRoot.MRM = inRef.*[1];
Listing 2. Back-end response parsing code
CREATE LASTCHILD OF OutputLocalEnvironment DOMAIN('MRM') PARSE(BlobBitstream 	
    SET 'MRM_Binary_CSV'
    TYPE funName
    FORMAT 'Text_CSV');

It is very important to test each message model separately before using it to make sure it gives you the right message in the right format. For this purpose, you use the simple flow shown below:

Figure 3. Simple flow to test message set
Simple flow to test message set

As shown, the flow has an MQInput node with the Input Message Parsing properties tab set as shown at the bottom. A breakpoint after the input node enables you to review the logical tree and make sure that the parsing is successful. You can set the output properties in the Compute node, as shown in Listing 3 below, to get the output message in XML format for easy review:

Listing 3. Simple code for converting the message physical format
SET OutputRoot.Properties.MessageSet='MRM_Binary_CSV';
SET OutputRoot.Properties.MessageType='Account_135a_Response';
SET OutputRoot.Properties.MessageFormat='XML';

Fixed-length text messages

This section shows you how to create the message model that can generate or parse one of the banking system messages. The example is shown in Listing 4 below. The first part of it, before the three commas, is a function ID with a fixed value. The second part, between the three commas and the hat character, is the message header with a defined number of elements that will be found in every message. The last part, after the hat character, contains the business parameters of the function:

Listing 4. Simple code for converting the message physical format
TAM.S.CHA.ACC.822,,,,
CRM 123456789012341234567890123456789012345678901234567890123456789
01234567890123456789012382202012345678901234567890123450123456789012340000^
0015881800240001362371

To create the message model, first create the complex type of the entire message as Account_822b_Request_t , as shown in Figure 4:

Figure 4. Message model in Message Broker message set
Message model in Message Broker message set

Set the logical and physical properties of this complex type as shown in Figure 5, start with the logical on the left, and leave the default values for the rest of the properties:

Figure 5. Logical and physical properties of the message type
Logical and physical properties of the message type

For the first part of the message, the function ID, create the simple element Prefix and set its logical properties, as shown in Figure 6 below, and leave the default for the rest of properties. The function ID text is set as a fixed value. If the element is missing from the message tree, the fixed value is inserted into the bit stream so that the message structure is preserved.

Figure 6. Logical properties of the Prefix element
Logical properties of the Prefix element

After adding the Prefix element, add three dummy local elements with zero length as placeholders to force the system generate the three commas after the delimiting fourth comma.

For the other two parts of the message, create a complex type containing the other two parts of the message, the header and the business parameters. Assuming the type name is _822b_Req_Body_t, set the physical properties of that complex type as shown in Figure 7 below. Add a local element to the main message with the type _822b_Req_Body_t (the element is named ReqVariables in Figure 4).

Figure 7. Physical properties of the complex type _822b_Req_Body_t
Physical properties of the complex type _822b_Req_Body_t

Each of the last two parts of the message is modeled in two elements of complex types. Each of them contains an ordered sequence of fixed-length elements. Figure 8 shows the physical properties of one of them -- the one for the business parameters, _822_Request_businessParams_t:

Figure 8. Physical properties of the complex type _822_Request_businessParams_t
Physical properties of the complex type _822_Request_businessParams_t

Tagged delimited ordered or unordered text messages

Listing 5 below shows one of the banking system messages of the Tagged Delimited String (TDS) format with unordered elements. One important specification of the message is that any of the tagged elements would be missed according to the pattern of the parameter values. The first part of the message, from the start to the three commas, represents the function ID. But this time it contains a comma that divides the ID into two subparts. Therefore, it is represented in the message model by two elements rather than one, Prefix1 and Prefix2. The extra two commas between the function ID and the second part of the message are handled the same way as with the fixed-length message sample, by inserting dummy elements:

Listing 5. Message with tagged delimited, unordered elements
FUNDS.TRANSFER,/I/PROCESS,,,
SRC.CHANNEL::=IVR,INT.TIME.STAMP::=00000000000000,MIDDLE.WARE.ID::=
0000000000000000000000021345678900987654321100450635090876523212131167890,MESS.TYPE::=
255,MSG.SUB.TYPE::=01,MSG.QUALIFIER::=0,INT.MSG.NUM::=2134567890098765432110045000
9090876523212131167890,FORCE.POST.FLAG::=0,CUSTOMER.NO::=703154,DEBIT.ACCT.NO::=
1588180024,DEBIT.CURRENCY::=AED,DEBIT.AMOUNT::=10000,CREDIT.ACCT.NO::=
0580079572,CREDIT.CURRENCY::=EGP,EQU.EGP.AMT::=10000,CHARGES.ACCT.NO::=
1588650075,CHARGE.AMT::=GBP10,CHARGE.TYPE::=CORRBKCHG

Figure 9 below shows the message model for the TDS sample. As both the message header and the business parameters are tagged and delimited using the same delimiters, they are encapsulated in the same complex type of the element TagValFields:

Figure 9. Message model in the Message Broker message set
Message model in the Message Broker message set

Figure 10 below shows the logical and physical properties of the complex type that contains both the message header and the business parameters. As shown, for the logical properties, set the Composition to unorderedSet. For the Physical properties, set the Data Element Separation to Tagged Delimited, and select a comma as the Delimiter (separator of the different elements). Then set the Tag Data Separator to the three characters shown in the message sample (::=). Leave the defaults for the rest of both the logical and physical parameters:

Figure 10. Logical and physical properties of the TDS complex type
Logical and physical properties of the TDS complex type

In Figure 11 below, you can see the physical properties for a sample element of the elements constituting the complex type that contains both the header and the business parameters -- the element IntTimeStamp. Here you can set the tag and the rest of the physical representation properties. The tag here is the element name that will appear in the message rather than the element name shown in Figure 9, which will appear only in the logical tree, in other words, internally in Message Broker.

Figure 11. Physical properties of the simple element IntTimeStamp
Physical properties of the simple element IntTimeStamp

ISO 8583 messages

ISO 8583 is an ISO standard for the banking and financial services sector, and it specifies a common interface for interchanging credit and debit card messages between devices and card issuers. ISO 8583 is typically used to define the message format for data exchanged with a point-of-sale (POS) device or an automated teller machine (ATM).

The developerWorks article Integrating with TCP/IP using WebSphere Message Broker explains the concepts and usage of ISO 8583. Another developerWorks article, Integrating secure ATM banking systems using WebSphere Message Broker presents additional details, include extension of 64-element ISO 8583 messages, with recommendations to avoid integration issues.

Business integration developers or designers who use ISO 8583 messages must know how the message model is created, so that they can create it themselves or customize a reusable model for a specific use. This section describes ISO 8583 from the message set perspective.

ISO 8583 messages are physical fixed-length text messages that contain some binary hexadecimal elements. Whenever you receive an ISO 8583 message, you must interpret the primary bitmap so that you can to use it to interpret the rest of the message. The primary bitmap is an 8-character element. The message model, as shown in Figure 12 below, contains an element of complex type named PrimaryBitmap as the first element. This complex type, as shown in Figure 13 below, contains 64 integer elements, each of them corresponding to one bit of the 8-character bitmap. Your ESQL code must check each bit of the received 8-character element and fill in the corresponding integer element. If you replace the binary bitmap by the generated 64-integer complex element in the received message, you can use the message model in Figure 12 to parse it.

Figure 12. Part of the ISO 8583 message model
Part of the ISO 8583 message model

How does the parser use the model to interpret the message? To understand this process, first look at how the properties of the model elements are set. Take the element PrimaryAccountnumber as an example. With elements of variable length where you let the user determine the length, ISO 8583 creates an element of complex type that contains an integer element for the length followed by the element in question. The element PrimaryAccountnumber is of that pattern -- a length element followed by the required element itself.

Figure 13. Part of the PrimaryBitmap complex type
Part of the PrimaryBitmap complex type

Figure 14 shows the part of the physical properties of the complex-type element PrimaryAccountnumber. The property to focus on is the Repeat Reference, whose value is selected to point to the sub-element of the PrimaryBitmap element that has the same index as the index of the element in question (PrimaryAccountnumber). Since this element is the third element in the message model, it corresponds to the Bit2 element of the bitmap (starting with Bit0). This selection makes the parser dedicate the third element in the message to be parsed as the PrimaryAccountnumber element when the Bit2 sub-element of the PrimaryBitmap element has a nonzero value.

Figure 14. Part of physical properties of ISO 8583 message element
Part of physical properties of ISO 8583 message element

To understand how the parser detects the length of the length-value elements such as the PrimaryAccountnumber element, see Figure 15 below, which shows the important physical properties of the sub-element PrimaryAccountnumber. For the property Length Reference, select the element name that contains the length value -- the element Length. The parser gets the length of the PrimaryAccountnumber element from the value of the fixed-length element Length.

Figure 15. Part of physical properties of a variable-length element
Part of physical properties of a variable-length element

Conclusion

This article showed how message modeling in WebSphere Message Broker can simplify integration efforts and take advantage of its highly efficient parsers. It focused on three physical message types and showed how you can easily create the message model for these complex-format message samples -- fixed-length text messages, tagged delimited string messages, and ISO 8583 standard messages (a special case of a fixed-length text messages).

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=830722
ArticleTitle=Integrating systems of highly complex message formats with WebSphere Message Broker
publish-date=08152012