Enhancing flexibility of Websphere Message Broker applications by deploying application resources as source files

WebSphere Message Broker V8 (hereafter called "Message Broker") Toolkit Fix Pack 1 makes your message flows more flexible by enabling you to update and deploy parts of a flow without recompiling and redeploying the entire flow. You can also assemble your broker archive (BAR) file on the same server where the application is deployed but where the Message Broker Toolkit (hereafter called the "Toolkit") is not available. This article shows you how to use Toolkit Fix Pack 1 to construct and configure message flows using deployable subflows, ESQL modules, and maps without compiling a flow into a compiled message flow (CMF) format, and how to modify and assemble Message Broker artifacts on a server without a Toolkit.

Share:

Julia Canella (jcanella@ca.ibm.com), Software Engineer, WebSphere Message Broker Toolkit team, IBM

Photo of Julia CanellaJulia Canella is A Software Engineer on the WebSphere Message Broker Toolkit team at the IBM Software Lab in Toronto, Canada. She works on the BAR and Navigator components of the Toolkit. You can contact Julia at jcanella@ca.ibm.com.



Eugene Kharlamov (eugenek@ca.ibm.com), Software Developer, IBM Integration Bus Development Team, IBM

Photo of Eugene KharlamovEugene Kharlamov is a Software Developer on the IBM Integration Bus Development Team. He has 15 years of experience with IBM in product development and consultant roles for different WebSphere products. You can contact Eugene at eugenek@ca.ibm.com.



19 September 2012

Also available in Chinese

Introduction

This article shows you how to use the new features in IBM® WebSphere® Message Broker Toolkit Fix Pack 1 to deploy and manage Message Broker applications in source format without compiling them in the Toolkit. These features improve application flexibility and make code updates much easier.

Designing Message Broker applications to use source code deployment of message flows, subflows, ESQL modules, and maps makes a lot of sense in many scenarios. For example, an ESQL module used by multiple flows may need to be updated after the referencing flows have been deployed. In the past, all main message flows that contain the updated ESQL module would have had to be recompiled and retested. In addition you would have to build a BAR file on the machine where the Toolkit is installed and then copy the created BAR file to the target server and deploy it to the target Message Broker server. And since the Toolkit is supported only on a subset of the platforms supported by Message Broker, the BAR file could not be created on some Message Broker servers.

Using Toolkit Fix Pack 1, you can deploy message flows, subflows, ESQL modules, and maps without compiling them in the Toolkit independently from each other. Therefore a BAR file including these components can be built on any Message Broker server, whether or not the Toolkit is available. The source deployment examples in this article show you how to use the new Broker and Toolkit features to achieve these goals.

You can download a sample application at the bottom of this article that shows you how to design message flows for maximum flexibility by using independently deployed reusable components such as message flows, subflows, ESQL modules, and maps. The sample application also shows you how to assemble a BAR file on a target server when you need to include message sets or Java® code that have to be compiled before being added to the BAR file and deployed to the Message Broker runtime. The sample application has been simplified to focus on the main topics of the article.

In the past, all message flows and subflows were implemented in files with a .msgflow extension. The difference between a deployable message flow and a subflow was determined by message flow contents. If the flow had an input node that could be invoked externally (such as an MQ Input or Web Service Input node), then it was a deployable message flow that could be compiled into a CMF file and deployed to the broker.

When the BAR file was assembled, all deployable message flow resources were compiled. The subflows referenced by the flow were in-lined into the parent CMF file – a process where the executable code is inserted directly into the file. If more than one subflow node in the parent flow referenced the same subflow implementation, then each node would be replaced by an in-lined copy of the subflow in the compiled CMF. If a subflow referenced another subflow, then the nested subflows were flattened and inserted into the parent flow CMF. ESQL modules or mapping code was also in-lined into the CMF file during this process.

Therefore, an application that consisted of a message flow, a subflow, an ESQL module, and a map turned into a single CMF file when the BAR file was built and deployed. When any change to the container message flow, child subflow, ESQL module, or mapping module was required, then the entire application had to be recompiled, retested, and redeployed.

The BAR file also had to be created on a machine where the Toolkit was installed, since assembling the BAR file required compiling the broker application resources, and the compilation process was dependent on the Toolkit workspace metadata. The Toolkit can be installed only on Linux® or Microsoft® Windows® machines, and therefore when the Broker server operating system was not supported by the Toolkit, the BAR file could not be assembled directly on the Broker server.

Figure 1 shows a simple example of a broker application that includes a main deployable message flow (Flow1.msgflow) referencing a subflow (Subflow1.msgflow) and an ESQL module (esqlflow.esql). The subflow also references esqlsub.esql. When deployed, there is only one Flow1.cmf file, which contains the flow with both the ESQL modules and the subflow in-lined:.

Traditional application – Single CMF
Traditional application – Single CMF

To make your application more flexible and easier to maintain, you do not want to have to recompile, retest, or redeploy the entire application when updates are made. When a change is needed, for example only to the ESQL module esqlsub.esql, then you just want to redeploy the affected ESQL file without redeploying all resources that reference esqlsub.esql. Toolkit Fix Pack 1 makes your applications more flexible by enabling you to deploy message flows (.msgflow), subflows (.subflow), ESQL modules (.esql), and maps (.msgmap) as source files instead of compiled versions in the CMF file.

In addition to the above mentioned resources, maps (.map), WSDL files (.wsdl), and message models defined by XML schemas (.xsd) can be deployed as source files as well. The message models in the broker can be defined by the XML schemas and WSDL files. In V8 they do not have to be defined using only message sets anymore (although the message models defined by message sets are fully supported). Figure 2 shows that the same source files authored in the Toolkit are added to the BAR file and deployed to the broker independently as source files:

New application – Resources deployed as source files
New application – Resources deployed as source files

By deploying the source files directly to the broker instead of compiling them into a CMF file, you are making the application more flexible since the changed components can be independently deployed to the broker. When only one of the files is updated, simply create a new BAR file containing only the updated source file and deploy it to the execution group. This new source file will replace the old one in the execution group and the referencing flows will automatically restart to pick up the changes. For example. in Figure 2, if esqlsub.esql is updated, it is the only file that needs to be added to the new BAR file. When that BAR file is deployed to the same default execution group and when Flow1 is run, the flow will stop, pick up the updated ESQL code, and restart.

Packaging the BAR file on the target server

When the broker application authoring is complete, the source code is usually stored in a source control system such as CVS, RTC, or Clear Case, and then to deploy the broker application to a target server, the application artifacts are packaged into a BAR file. When the BAR creation process is automated, the MQSICREATEBAR command is usually called from an automated script. MQSICREATEBAR command implementation is dependent of the Toolkit, and therefore BAR packaging scripts using this command can be run only on a machine where the Toolkit is installed. The Broker server often runs on a server with an operating system that is not supported by the Toolkit. Therefore you would have to build the BAR file using the MQSICREATEBAR command on a machine where the Toolkit is installed, move the BAR file to the Broker server machine, and deploy the BAR file to the broker server. Figure 3 shows the BAR packaging process distributed across multiple machines:

Traditional deployment process
Traditional deployment process

The option to deploy resources as source files lets you bypass the compilation process in the Toolkit or MQSICREATEBAR command. Instead you can use the MQSIPACKAGEBAR command to build the BAR file directly on the broker server. This command is installed together with the Broker server and can be invoked without having the Toolkit installed. In this case, as shown in Figure 4, the deployable source files can be transferred from the source control server directly to the Message Broker Server. Then you can run the MQSIPACKAGEBAR command on the target Broker server to create the BAR file. Using the MQSIPACKAGEBAR command simplifies the broker resources deployment process and makes it independent from the Toolkit.

Source deployment process
Source deployment process

While using the MQSIPACKAGEBAR tool is convenient because it runs on the Broker server, creating a script that includes all of the required artifacts and their dependencies without Toolkit assistance may be not be as simple, especially for complex applications. To simplify this task, you can use the Toolkit to generate the MQSIPACKAGEBAR script during development. This script can be used on the Broker server later as long as the application file structure does not change. To generate the script, open the BAR editor in the Toolkit. Select the desired contents on the Build tab and build the BAR file, making sure that the build option Compile and in-line resources is selected. Once the BAR file is built, open the Manage editor tab and expand the group Command for packaging the BAR contents. Figure 5 shows that the expanded section contains the MQSIPACKAGEBAR command needed to create the same BAR file on the server. You can save this script to a file on the file system or workspace. After the development is complete, the source files and the script can be checked into the Source Control Server. Then you can check the script out on the Message Broker server, update the target workspace location pointing to the local file system, and run the script on the server.

MQSIPACKAGEBAR command
MQSIPACKAGEBAR command

Figure 6 shows the application life cycle from authoring your applications to deploying the BAR file on the Message Broker Server:

Application life cycle
Application life cycle

Deploying message sets and Java components with source deploy

When the BAR file is created using source artifacts, some components of broker applications, like message sets and Java code, have to be compiled before being added to the BAR file. If your application uses a message set or a Java Compute node referencing a Java class in a Java project on the workspace, then you need to generate the deployable files for those components.

If you are building a BAR file using the Toolkit BAR Editor or the MQSICREATEBAR command, then these deployable files are generated for you. However if you use the MQSIPACKAGEBAR command, you must compile these artifacts in the Toolkit before adding them to the BAR file on the server. It may not be possible to compile these resources on the Broker Server, since it may not have the Toolkit and necessary compilers installed.

There are two ways to compile these resources. In the Toolkit, you can right-click on the Java or Message Set project in the Broker Development view and select Build for mqsipackagebar. The compiled files for Message Sets and Java projects will then be generated and saved on the workspace. You can check these compiled files into the source control repository, retrieve them on the server, and add then to the BAR file using the MQSIPACKAGEBAR command.

Another way is to use the Toolkit MQSICREATEBAR command with the special flag -compileOnly:

mqsicreatebar -data {workspaceWithJavaOrMsgSetProjects} -compileOnly

User-defined and promoted properties with source deploy

It is common for broker application flows to use promoted and user-defined properties defined on a subflow. An example will show how to set the values of these properties and how the values are used by the runtime. Figure 7 shows how to set the default values of the user-defined and promoted properties in the Properties view when you open the subflow in the Flow Editor:

Promoted and user-defined properties for subflow in Flow Editor
Promoted and user-defined properties for subflow in Flow Editor

You can also override the default value from the main flow by setting the value after clicking on the subflow node on the main flow opened in the Flow Editor:

Default value for subflow property overridden in Flow Editor
Default value for subflow property overridden in Flow Editor

Another way you can override the user-defined or configurable property value of the subflow is in the BAR Editor. By selecting the subflow element under the parent subflow file element on the Manage tab's tree, you can then override the default value in the Properties view:

Changing default value of promoted and user-defined properties for subflow in BAR Editor
Changing default value of promoted and user-defined properties for subflow in BAR Editor

In addition to overriding the default values in the BAR editor when the subflow source is selected, you can also override the values specified on the subflow node in the BAR Editor when the subflow node element is selected under the main flow parent element:

Promoted and user-defined default properties for all instances of subflow can be set in BAR Editor
Promoted and user-defined default properties for all instances of subflow can be set in BAR Editor

When the property value is updated in the BAR Editor, the value is stored in a broker.xml file whose values override the values stored in the source flows.

The runtime usage of the values of the user-defined and promoted properties follow a set of rules. The value of a subflow property set in the Flow Editor (Figure 7) will be the default value for that property for all message flows that use that subflow if it is not overridden anywhere else. As shown in Figure 8, that value may be overridden from the referencing flow by selecting the subflow node in the Flow Editor. If the value has been overridden by the parent flow, then the default value defined on the subflow will not be used by the runtime and instead the value set on the subflow node will be used.

You can override both the default property value and the property value set on the subflow node using the BAR Editor, which adds to your ability to change subflow property values without changing the source code for the subflow or referencing flow, and avoids the need to rebuild the BAR file.

If the value has been set when the subflow node is selected on the parent flow in the Flow Editor, as shown in Figure 8 above, then the only way to override that value is to use the method shown in Figure 10. After you have set the value in this way, there is no other way to override the value anywhere else. If the default property value set on the subflow is not overridden on the subflow node from the referencing flow (Figure 7), than either method of changing the value in the BAR Editor will override this value.

Table 1 shows you how to override the user-defined or promoted property value. Three message flows (Flow1, Flow2, and Flow3) have a subflow node referencing the same subflow (Subflow1) with different property configurations. The subflow has one configurable property (property0). The property default value is set to 1 by the subflow itself opened in the Flow Editor. The actual value used by the runtime when the flows are deployed depends whether and where the property's value is overridden.

Table 1. Example of property value usage for user-defined and configurable properties across Toolkit and runtime
Property setting (default value = 1)Flow1 referencing Subflow1Flow2 referencing Subflow1Flow3 referencing Subflow1
Override default property when subflow node is selected on parent flow in Flow EditorSet value on subflow node to 2. Effective property0 value: 2.No overrides. Effective property0 value is default value: 1.No overrides. Effective property0 value is default value: 1.
Subflow element selected on BAR Editor Manage tab. Subflow property0 default value overridden with: 3.BAR override does not apply as node override takes precedence. Effective property0 value: 2.Uses overridden value by BAR Editor. Effective property0 value: 3.Uses overridden value by BAR Editor. Effective property0 value: 3.
Subflow node element selected under referencing flow element on BAR Editor Manage tabNo overrides. Value specified on subflow node in Flow Editor is used. Effective property0 value: 2.No overrides. Default value overridden by BAR Editor on subflow is used. Effective property0 value: 3.Subflow node property value set to 4. Effective property0 value: 4.
Value used by broker runtimeEffective property0 value: 2Effective property0 value: 3Effective property0 value: 4

Sample overview

The sample application used in this article (hereafter called the "application") shows you how to design and deploy message flows as source files to achieve better structural and logical flexibility. The application is based on the Address Book sample provided in the Toolkit.

The application contains six projects. The Message Broker project AddressBook contains the two flows and references three other projects: the library LogMessageData, the Java project AddressBookJava, and Message Set project AddressBookMessageSet. The BARfiles project simply contains the BAR files in the sample. TestClientBarFiles contains the file TestBroker_default.tst, which is an MQSC script that you can run to create the following queues in your broker: ADDRESSBOOK_FAULT, ADDRESSBOOK_IN, ADDRESSBOOK_OUT.

In addition to the Address Book Sample, the application contains the LogTransaction.subflow invoked from the AddressBookProviderFlow.msgflow to log the received data to the file system using a custom Java logging component packaged in loggerLib.jar. The custom Java logger interface requires the input to be in the format modeled by the XML schema stored in the LogMessageData library, FileMessageDefinition.xsd:

Sample application
Sample application

Implementing the message flow

There are two message flows in the source deploy sample: AddressBookConsumerFlow.msgflow and AddressBookProvider.msgflow. Figure 12 shows the updated AddressBookProviderFlow.msgflow that includes the custom logging subflow. The subflow LogTransaction.subflow has been added to the flow and will be run when the operation saveAddress occurs. The next section explains how the LogTransaction subflow works.

Address book provider flow
Address book provider flow

Implementing the deployable subflow

The subflow LogTransaction.subflow is called before the record is saved. The subflow carries out two main operations: it uses a map to change the message format from the format used by the main flow to the format required by the custom logger interface, and it then passes the changed message to a Java Compute node, which saves the message into a file on the file system:

Subflow diagram
Subflow diagram

Running the application

The first step in running the application is to build the BAR file with the necessary source files without compiling the flows. Open AddressBookConsumer.bar in the BAR Editor and make sure that the Compile and in-line resources option is not selected:

Build options
Build options

Select both message flows, the library (LogMessageData), and the message set (messageSet.mset) in the Editor. Click Build and Save to add the artifacts to the BAR file. If you select the Manage tab on the BAR editor, you can see all the artifacts that were packaged into the BAR file, as shown in Figure 15. You can also see what the MQSIPACKAGEBAR command would look like if you were to create the same BAR file on a machine without the Toolkit installed:

Manage tab
Manage tab

To deploy the AddressBookConsumer.bar file, drag and drop it onto an execution group in the Brokers view. Once AddressBookConsumer.bar is deployed, all the source artifacts (message flows, subflow, ESQL modules,and library with XML schema file) are displayed as child elements of the execution group in the Brokers view, as shown in Figure 16.

Since message sets and Java projects cannot be deployed as source files, the compiled versions of these artifacts have been deployed to the execution group: AddressBookJava.jar for the AddressBookJava Java project, AddressBookMessageSet.xsdzip for the AddressBookMessageSet message set and the custom Java logging library loggerLib.jar.

Brokers view
Brokers view

Testing the sample with the test client

To test the application, open the AddressBookConsumerFlow.mbtest file in the AddressBook project in the Test Client Editor (Figure 17). By default, the test client is configured to invoke the flow with a message that contains an address that needs to be saved. Make sure the Configuration tab of the test client editor has the correct Deployment location information for your broker.

Address Book Consumer flow test client
Address Book Consumer flow test client

To send the message, click Send Message. Once the message is received by the Provider flow, it is forwarded to the path that saves the address from the message. The Flow Order node routes the message to both the Custom Logger subflow and to the Compute node that saves the message.

After transforming the message into the format required by the custom Java logger, the subflow invokes the Java Compute node Custom logger that internally extracts the required data from the message and then invokes the custom Java class LogFileWriter packaged in loggerLib.jar. This class creates a custom log file that contains the data from the message in the default system directory to store the temporary files (java.io.tmpdir). You will be able to see the path the message will travel in the test client in the Message Flow Test Events section.

Redeploying artifacts using Source Deploy and the Toolkit

To see the new flexibility of message flows in Toolkit Fix Pack 1, you can replace the custom Java logger with the out-of-the-box Trace node. Open the LogTransaction.subflow file, remove the existing Java Compute node, and configure a new Trace node, as shown in Figure 18. Here is the code to paste into the Pattern field:

Message saved: Name: ${Body.FileDataElement.name}
Contents: ${Body.FileDataElement.contents} 
Time is: ${EXTRACT(HOUR FROM CURRENT_TIMESTAMP)}
:${EXTRACT(MINUTE FROM CURRENT_TIMESTAMP)}
New Trace node configuration
New Trace node configuration

Save your subflow. To redeploy the subflow, create a new BAR file in the BARfiles project and name it updatedSubflow.bar. Check the LogTransaction.subflow file in the BAR Editor tree and click Build and Save. Since the subflow references LogMap.map and ToFileData.map, these files are also added to the BAR file, as shown below:

BAR file with updated subflow
BAR file with updated subflow

Drag and drop updatedSubflow.bar onto the execution group. Once the updated LogTransaction.subflow and referenced maps are deployed, all of the referencing flows will restart automatically. When they are invoked again, the new subflow implementation will be used by the runtime. In fact, you can manually remove the maps from the BAR file using the BAR Editor Manage tab, as the maps have not changed. In any case, your Brokers view should still look like Figure 16 above.

However, when you click on the log.LogTransaction component in the Brokers view, the Properties view indicates that the subflow file has been redeployed using the newly generated BAR file:

Redeployed subflow
Redeployed subflow

If you execute the same test you have just run, you should notice that the trace data is now written to the trace file specified in the Trace node (C:\tempLog\trace.log). You can deploy updated message flows, subflows, ESQL modules, and maps in the same independent fashion without the need to redeploy the referencing or referenced components.

Redeploying artifacts using Source Deploy and MQSIPACKAGEBAR

To make a change and redeploy a message set using the MQSIPACKAGEBAR command -- for example, to add a Country element to the address being stored -- open AddressBook_InlineSchema1.mxsd in AddressBookMessageSet. Add a local element to the message Address and name it Country. Also, change the minimum occurrences to 0 to avoid complaints that the sample message lacks a country element. Save the file.

As discussed, message set and Java projects cannot be deployed as source files, and therefore must be precompiled before being added to the BAR file. In order to save a compiled version of the message set (.xsdzip), right-click on AddressBookMessageSet in the Application Development navigator and click Build for mqsipackagebar. This action creates the file AddressBookMessageSet.xsdzip in the AddressBookMessageSet project. If you were to check the AddressBookMessageSet.xsdzip file into a source control system and then check it out on a machine without the Toolkit, you could use the MQSIPACKAGEBAR command to create a BAR file to redeploy this message set.

Open the Broker command console and type the command:

mqsipackagebar -w "{workspace path}\AddressBookMessageSet" -a 
"{path of directory where you want bar file to be saved}\updatedMSet.bar" -o 
AddressBookMessageSet.xsdzip

This command creates a new BAR file in the specified directory. After the message set is compiled and added to the BAR file, you can redeploy the message set by deploying this new BAR file to the execution group.

Conclusion

You have learned how deploying source files for flows, subflows, ESQL modules, and maps makes Message Broker applications more flexible and easier to maintain. The article showed how to redeploy only the updated resources of your application without recompiling and retesting your main message flows. It also showed how to create the BAR file on a Broker server where the toolkit is not available, and how to use the Toolkit features to deploy resources that require precompilation steps, such as message sets and Java projects. The section about user-defined and promoted properties described what property values are used by the runtime in different configuration scenarios.


Download

DescriptionNameSize
Code sampleSourceDeploy.zip30 KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=836465
ArticleTitle=Enhancing flexibility of Websphere Message Broker applications by deploying application resources as source files
publish-date=09192012