Integrating systems of highly complex message formats with WebSphere Message Broker
Data transformation and format conversion are core tasks of business integration. For data transformation, IBM® WebSphere® Message Broker (hereafter called Message Broker) provides its Mapping node, and for the physical format conversions, data modeling using the message sets plays an important role, in conjunction with different message parsers.
After creating the message model, you can use it to convert the format of any request message received by Message Broker to another physical format required by a back-end system. In the other direction, you can use the message model to parse the response message from the back-end system to generate the logical tree of the message, and easily convert it back to the format of the front-end system. The main physical formats that are widely used and supported by Message Broker are the text, binary, and XML. Text format can be fixed-length, tagged delimited, or delimited only, and delimited elements can either be ordered or unordered. This article has four sections:
- Using message sets to reduce code size uses a real-world example to show you how message sets can significantly reduce programming effort and time to market.
- Fixed-length text messages uses the example to show you how to use fixed-length text message.
- Tagged delimited ordered or unordered text messages explains this format and the ISO 8583 standard message structure.
- ISO 8583 messages shows you how to create an ISO 8583 message model and us it to facilitate message parsing regardless of the physical format.
Using message sets to reduce code size
Sometimes you need to integrate with a back-end system that has a complex interface. In that case, the message modeling capability in the Message Broker can significantly speed up the integration process, as shown in the example used in this article. The example architecture is shown in Figure 1. The banking system requires request and response messages for some functions to be in a complex fixed-length text format, and in a tagged delimited unordered complex text format for other functions.
Figure 1. Example architecture
In such cases, where you do not know the strength of message modeling in Message Broker, you would need to write complex custom code to do both the physical format conversion and the message parsing. Creating message models and then using the MRM parser gives you the flexibility to generate or parse any message, regardless of its complexity. Furthermore, using the proven code in the MRM parser increases the overall reliability of your middleware solution. Figure 2 below shows the banking system adapter message flow as it receives an XML MQ message from the front end through the MQInput node. To convert this message to text format according to the message model, you simply write the three lines of ESQL code shown in Listing 1 to select the message model. Any created logical tree (the last two statements in the listing create the MRM tree) is converted to the selected physical format. The code is associated with the Message Broker compute node Extract backend Msg.
Figure 2. Banking system adapter message flow
The response from the back-end banking system is read by the MQGet node, and then the compute node MapToesbXML is parsed by some simple ESQL code. Just one statement calls the MRM parser according to the message model parameters, as shown in Listing 2 below.
Listing 1. Back-end adapter request generation code
DECLARE inRef REFERENCE TO InputRoot.XMLNSC.BE_esbXML.Body; SET OutputRoot.Properties.MessageSet='MRM_Binary_CSV'; SET OutputRoot.Properties.MessageType=FIELDNAME(inRef.*); SET OutputRoot.Properties.MessageFormat='Text_CSV'; -- <<<<Copy the Backend message from the BE_esbXML>>>> CREATE LASTCHILD OF OutputRoot DOMAIN ('MRM'); SET OutputRoot.MRM = inRef.*;
Listing 2. Back-end response parsing code
CREATE LASTCHILD OF OutputLocalEnvironment DOMAIN('MRM') PARSE(BlobBitstream SET 'MRM_Binary_CSV' TYPE funName FORMAT 'Text_CSV');
It is very important to test each message model separately before using it to make sure it gives you the right message in the right format. For this purpose, you use the simple flow shown below:
Figure 3. Simple flow to test message set
As shown, the flow has an MQInput node with the Input Message Parsing properties tab set as shown at the bottom. A breakpoint after the input node enables you to review the logical tree and make sure that the parsing is successful. You can set the output properties in the Compute node, as shown in Listing 3 below, to get the output message in XML format for easy review:
Listing 3. Simple code for converting the message physical format
SET OutputRoot.Properties.MessageSet='MRM_Binary_CSV'; SET OutputRoot.Properties.MessageType='Account_135a_Response'; SET OutputRoot.Properties.MessageFormat='XML';
Fixed-length text messages
This section shows you how to create the message model that can generate or parse one of the banking system messages. The example is shown in Listing 4 below. The first part of it, before the three commas, is a function ID with a fixed value. The second part, between the three commas and the hat character, is the message header with a defined number of elements that will be found in every message. The last part, after the hat character, contains the business parameters of the function:
Listing 4. Simple code for converting the message physical format
TAM.S.CHA.ACC.822,,,, CRM 123456789012341234567890123456789012345678901234567890123456789 01234567890123456789012382202012345678901234567890123450123456789012340000^ 0015881800240001362371
To create the message model, first create the complex type of the entire message as Account_822b_Request_t , as shown in Figure 4:
Figure 4. Message model in Message Broker message set
Set the logical and physical properties of this complex type as shown in Figure 5, start with the logical on the left, and leave the default values for the rest of the properties:
Figure 5. Logical and physical properties of the message type
For the first part of the message, the function ID, create the simple element Prefix and set its logical properties, as shown in Figure 6 below, and leave the default for the rest of properties. The function ID text is set as a fixed value. If the element is missing from the message tree, the fixed value is inserted into the bit stream so that the message structure is preserved.
Figure 6. Logical properties of the Prefix element
After adding the Prefix element, add three dummy local elements with zero length as placeholders to force the system generate the three commas after the delimiting fourth comma.
For the other two parts of the message, create a complex type containing the other two parts of the message, the header and the business parameters. Assuming the type name is _822b_Req_Body_t, set the physical properties of that complex type as shown in Figure 7 below. Add a local element to the main message with the type _822b_Req_Body_t (the element is named ReqVariables in Figure 4).
Figure 7. Physical properties of the complex type _822b_Req_Body_t
Each of the last two parts of the message is modeled in two elements of complex types. Each of them contains an ordered sequence of fixed-length elements. Figure 8 shows the physical properties of one of them -- the one for the business parameters, _822_Request_businessParams_t:
Figure 8. Physical properties of the complex type _822_Request_businessParams_t
Tagged delimited ordered or unordered text messages
Listing 5 below shows one of the banking system messages of the Tagged Delimited String (TDS) format with unordered elements. One important specification of the message is that any of the tagged elements would be missed according to the pattern of the parameter values. The first part of the message, from the start to the three commas, represents the function ID. But this time it contains a comma that divides the ID into two subparts. Therefore, it is represented in the message model by two elements rather than one, Prefix1 and Prefix2. The extra two commas between the function ID and the second part of the message are handled the same way as with the fixed-length message sample, by inserting dummy elements:
Listing 5. Message with tagged delimited, unordered elements
FUNDS.TRANSFER,/I/PROCESS,,, SRC.CHANNEL::=IVR,INT.TIME.STAMP::=00000000000000,MIDDLE.WARE.ID::= 0000000000000000000000021345678900987654321100450635090876523212131167890,MESS.TYPE::= 255,MSG.SUB.TYPE::=01,MSG.QUALIFIER::=0,INT.MSG.NUM::=2134567890098765432110045000 9090876523212131167890,FORCE.POST.FLAG::=0,CUSTOMER.NO::=703154,DEBIT.ACCT.NO::= 1588180024,DEBIT.CURRENCY::=AED,DEBIT.AMOUNT::=10000,CREDIT.ACCT.NO::= 0580079572,CREDIT.CURRENCY::=EGP,EQU.EGP.AMT::=10000,CHARGES.ACCT.NO::= 1588650075,CHARGE.AMT::=GBP10,CHARGE.TYPE::=CORRBKCHG
Figure 9 below shows the message model for the TDS sample. As both the message header and the business parameters are tagged and delimited using the same delimiters, they are encapsulated in the same complex type of the element TagValFields:
Figure 9. Message model in the Message Broker message set
Figure 10 below shows the logical and physical properties of the complex type that contains both the message header and the business parameters. As shown, for the logical properties, set the Composition to unorderedSet. For the Physical properties, set the Data Element Separation to Tagged Delimited, and select a comma as the Delimiter (separator of the different elements). Then set the Tag Data Separator to the three characters shown in the message sample (::=). Leave the defaults for the rest of both the logical and physical parameters:
Figure 10. Logical and physical properties of the TDS complex type
In Figure 11 below, you can see the physical properties for a sample element of the elements constituting the complex type that contains both the header and the business parameters -- the element IntTimeStamp. Here you can set the tag and the rest of the physical representation properties. The tag here is the element name that will appear in the message rather than the element name shown in Figure 9, which will appear only in the logical tree, in other words, internally in Message Broker.
Figure 11. Physical properties of the simple element IntTimeStamp
ISO 8583 messages
ISO 8583 is an ISO standard for the banking and financial services sector, and it specifies a common interface for interchanging credit and debit card messages between devices and card issuers. ISO 8583 is typically used to define the message format for data exchanged with a point-of-sale (POS) device or an automated teller machine (ATM).
The developerWorks article Integrating with TCP/IP using WebSphere Message Broker explains the concepts and usage of ISO 8583. Another developerWorks article, Integrating secure ATM banking systems using WebSphere Message Broker presents additional details, include extension of 64-element ISO 8583 messages, with recommendations to avoid integration issues.
Business integration developers or designers who use ISO 8583 messages must know how the message model is created, so that they can create it themselves or customize a reusable model for a specific use. This section describes ISO 8583 from the message set perspective.
ISO 8583 messages are physical fixed-length text messages that contain some binary hexadecimal elements. Whenever you receive an ISO 8583 message, you must interpret the primary bitmap so that you can to use it to interpret the rest of the message. The primary bitmap is an 8-character element. The message model, as shown in Figure 12 below, contains an element of complex type named PrimaryBitmap as the first element. This complex type, as shown in Figure 13 below, contains 64 integer elements, each of them corresponding to one bit of the 8-character bitmap. Your ESQL code must check each bit of the received 8-character element and fill in the corresponding integer element. If you replace the binary bitmap by the generated 64-integer complex element in the received message, you can use the message model in Figure 12 to parse it.
Figure 12. Part of the ISO 8583 message model
How does the parser use the model to interpret the message? To understand this process, first look at how the properties of the model elements are set. Take the element PrimaryAccountnumber as an example. With elements of variable length where you let the user determine the length, ISO 8583 creates an element of complex type that contains an integer element for the length followed by the element in question. The element PrimaryAccountnumber is of that pattern -- a length element followed by the required element itself.
Figure 13. Part of the PrimaryBitmap complex type
Figure 14 shows the part of the physical properties of the complex-type element PrimaryAccountnumber. The property to focus on is the Repeat Reference, whose value is selected to point to the sub-element of the PrimaryBitmap element that has the same index as the index of the element in question (PrimaryAccountnumber). Since this element is the third element in the message model, it corresponds to the Bit2 element of the bitmap (starting with Bit0). This selection makes the parser dedicate the third element in the message to be parsed as the PrimaryAccountnumber element when the Bit2 sub-element of the PrimaryBitmap element has a nonzero value.
Figure 14. Part of physical properties of ISO 8583 message element
To understand how the parser detects the length of the length-value elements such as the PrimaryAccountnumber element, see Figure 15 below, which shows the important physical properties of the sub-element PrimaryAccountnumber. For the property Length Reference, select the element name that contains the length value -- the element Length. The parser gets the length of the PrimaryAccountnumber element from the value of the fixed-length element Length.
Figure 15. Part of physical properties of a variable-length element
This article showed how message modeling in WebSphere Message Broker can simplify integration efforts and take advantage of its highly efficient parsers. It focused on three physical message types and showed how you can easily create the message model for these complex-format message samples -- fixed-length text messages, tagged delimited string messages, and ISO 8583 standard messages (a special case of a fixed-length text messages).
- WebSphere Message Broker
- WebSphere Message Broker V8 information center
A single Web portal to all WebSphere Message Broker V8 documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere Message Broker environment.
- WebSphere Message Broker developer resources
Technical resources to help you use WebSphere Message Broker for connectivity, universal data transformation, and enterprise-level integration of disparate services, applications, and platforms to power your SOA.
- WebSphere Message Broker product page
Product descriptions, product news, training information, support information, and more.
- Using message sets in WebSphere Business Integration Message
Broker to implement an ESB in an SOA
This IBM Redpaper shows you how to use WebSphere Message Broker to build an ESB that enables services communication and loose coupling between service providers and service consumers.
- Tutorial: WebSphere Message Broker message sets
Open the file IA96_0604/html/mrm.html in your browser.
- Tutorial: WebSphere Message Broker message sets and WSDL
An IBM Education Assistant tutorial
- Message modeling and parsing enhancements in WebSphere
Message Broker V6.1
Message modeling lets you predefine message formats so that WebSphere Message Broker can then automatically parse and write them. This developerWorks article shows you how to use the enhanced message modeling and parsing in WebSphere Message Broker.
- Download free trial version of WebSphere Message
WebSphere Message Broker is an ESB built for universal connectivity and transformation in heterogeneous IT environments. It distributes information and data generated by business events in real time to people, applications, and devices throughout your extended enterprise and beyond.
- WebSphere Message Broker documentation
WebSphere Message Broker specifications and manuals.
- WebSphere Message Broker forum
Get answers to your technical questions and share your expertise with other WebSphere Message Broker users.
- WebSphere Message Broker support page
A searchable database of support problems and their solutions, plus downloads, fixes, and problem tracking.
- WebSphere Message Broker V8 Development Training
In this IBM Training course, you will learn about the components of the WebSphere Message Broker development and runtime environments. The course also explores message flow problem determination, and shows you how to construct message flows that use ESQL, Java, and PHP to transform messages.
- Youtube tutorial: Integrating Microsoft .NET code in a
WebSphere Message Broker V8 message flow
This five-minute youtube tutorial shows you how simple it is to use WebSphere Message Broker V8 to build a message flow that includes Microsoft .NET code. Microsoft Visual Studio is used to build .NET code in C#, which is then integrated into a message flow using Message Broker and an HTTP RESTful interface.
- WebSphere Message Broker V8 information center
- WebSphere resources
- developerWorks WebSphere developer
Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
- developerWorks WebSphere application integration developer
How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions.
- Most popular WebSphere trial downloads
No-charge trial downloads for key WebSphere products.
- WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
- WebSphere on-demand demos
Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
- developerWorks WebSphere weekly newsletter
The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
- WebSphere-related books from IBM Press
Convenient online ordering through Barnes & Noble.
- WebSphere-related events
Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
- developerWorks WebSphere developer resources
- developerWorks resources
downloads for IBM software products
No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
Join a conversation with developerWorks users and authors, and IBM editors and developers.
cloud computing resources
Access the IBM or Amazon EC2 cloud, test an IBM cloud computing product in a sandbox, see demos of cloud computing products and services, read cloud articles, and access other cloud resources.
- developerWorks tech briefings
Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
- developerWorks podcasts
Listen to interesting and offbeat interviews and discussions with software innovators.
- developerWorks on
Check out recent Twitter messages and URLs.
- IBM Education Assistant
A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.
- Trial downloads for IBM software products