Creating a cloud to consume WebSphere MQ messages using IBM Workload Deployer and WebSphere TX

This article shows you how to use IBM Workload Deployer and WebSphere Transformation Extender with Launcher Hypervisor Edition to develop a cloud-based system that can dynamically scale to meet volume and reliability requirements when reading a WebSphere MQ queue.

Michael Hudson (mhudson@us.ibm.com), Senior Technical Staff Member, Data Transformation Strategy and Architecture, IBM

Photo of Michael HudsonMichael Hudson is the IBM Senior Technical Staff Member (STSM) for WebSphere Transformation Extender products at the IBM Software Lab in Boca Raton, Florida. You can contact Michael at mhudson@us.ibm.com.



14 March 2012

Introduction

IBM® WebSphere® Transformation Extender (hereafter called WebSphere TX) now offers its Launcher in a Hypervisor Edition. It is essentially a copy of WebSphere TX with Launcher pre-installed onto a Red Hat Linux® image that has be designed to work using virtualization managers such as IBM Workload Deployer.

The benefits of Hypervisor editions include simplified versioning, administration, scalability, and redundancy. This article will illustrate these benefits by showing you how to build, deploy, and run a collection of launchers that work together to read messages from a WebSphere MQ queue and write them to another queue. It will extend the basic WebSphere TX with Launcher Hypervisor Edition image by installing a WebSphere MQ client onto it, and then capture the extended image as a pattern based on a specific version (which could be a patch level) of WebSphere TX, with a specific version of a map and launcher system deployed to it. This pattern will be deployed multiple times to a cloud, and each launcher image will work independently of the other images to share the load and increase overall throughput. This article will end with a demonstration of the effect of abruptly removing active instances and bringing new instances online.

Prerequisites

In order to implement the system described in this article, you will need WebSphere TX with Launcher Hypervisor Edition V8.4.0.1, IBM Workload Deployer V3.1 and WebSphere MQ V7.

Before beginning to build this system, you need to have the required infrastructure in place, and you will need to have performed the basic steps to configure IBM Workload Deployer with a cloud group. For more information on this procedure, see the IBM Workload Deployer documentation.

After IBM Workload Deployer and its cloud group have been configured, import WebSphere TX with Launcher Hypervisor Edition into the IBM Workload Deployer catalog. Steps to do this are explained in the WebSphere TX with Launcher Hypervisor Edition documentation.

Finally, you will need a WebSphere MQ queue manager configured with a listener and two local queues called IN and OUT. You do not need an MQ cluster because you will use the cooperative browsing feature introduced in WebSphere MQ V7.

Cloning and extending WebSphere TX with Launcher virtual image

The first task is to clone and extend the basic WebSphere TX with Launcher virtual image by installing a WebSphere MQ client and deploying a WebSphere TX map and launcher system.

  1. Log on to IBM Workload Deployer using a web browser and navigate to the Catalog / Virtual Images view. Select the WebSphere TX with Launcher virtual image and click the Clone and extend icon (which looks like two CDs).
  2. Name the new virtual system and give it a description and version:
    Figure 1. Clone and extend
    Clone and extend
  3. The image must be deployed to the cloud before it can be extended. You will need to specify the deployment configuration for the chosen cloud group:
    Figure 2. Deployment configuration
    Deployment configuration
  4. Specify the hardware configuration:
    Figure 3. Hardware configuration
    Hardware configuration
  5. A new virtual image clone is created in the catalog and automatically deployed as a virtual system instance to the cloud:
    Figure 4. New virtual image
    New virtual image
  6. Once deployed, the virtual system instance is automatically started:
    Figure 5. Virtual image is started
    Virtual image is started
  7. Once started, the virtual image moves to Draft status, indicating that it is ready to be extended:
    Figure 6. Draft status
    Draft status
  8. Here is the new virtual system instance in the Instances view:
    Figure 7. Instances view
    Instances view
  9. Select the virtual system instance to view its virtual machines. From here, you can find the IP address or host name of the virtual machine running the new virtual system:
    Figure 8. Finding the IP address
    Finding the IP address
  10. Log in to the virtual machine using SSH:
    Figure 9. SSH session
    SSH session
  11. Extend the virtual image by installing a WebSphere MQ V7 client, which can be pushed to the image or pulled from the image using FTP:
    Figure 10. Install an MQ client
    Install an MQ client
  12. Set LD_LIBRARY_PATH to include the MQ 64-bit lib folder by adding the following statement to the wtxuser profile: export LD_LIBRARY_PATH=/opt/mqm/lib64/:$LD_LIBRARY_PATH:
    Figure 11. Edit the LD_LIBRARY_PATH
    Edit the LD_LIBRARY_PATH
  13. A known issue with IBM Workload Deployer V3.1 and/or WebSphere TX V8.4.0.1 requires the following step to be done manually. Edit /etc/virtualimage.properties file and add:
    RESET_VIRTUAL_IMAGE_COMMAND=/var/adm/ibmvmcoc-postinstall/resetvm.sh
    RESET_VIRTUAL_IMAGE_COMMAND_LOCATION=/var/adm/ibmvmcoc-postinstall

The next step is to create a WebSphere TX map with one input card that reads a message from one MQ queue and one output card that writes a transformed message to another MQ queue. Both cards should use the IBM WebSphere MQ (client) adapter.

  1. From a Microsoft® Windows® PC, open the WebSphere TX Design Studio and create a new map named simple. Use the same basic type tree for the input and output cards. All you need is one Text item, which will be the root type for both cards. The transformation rule is arbitrary. For example, you can convert the incoming message to upper case using the built-in UPPERCASE map rule:
    Figure 12. Creating a map
    Creating a map
  2. Configure the adapter command lines. The input adapter command line should be:
    -QN InputQueueName -QMN QueueManagerName -CD 
    SYSTEM.ADMIN.SVRCONN/TCP/QueueManagerHostName(QueueManagerPort)

    The output adapter command line should be:
    -QN OutputQueueName -QMN QueueManagerName -CD 
    SYSTEM.ADMIN.SVRCONN/TCP/QueueManagerHostName(QueueManagerPort)
  3. Now that you have a map, you must create a system that will run it. Open the WebSphere TX Integration Flow Designer and create a new system. Define a Linux server and set the Execution Mode to Launcher:
  4. Add a source map component and choose the map you created:
    Figure 13. Creating a system
    Creating a system
  5. Edit the Launcher settings. The MapServerLocation should point to a map within a path that exists on the virtual image, such as in a directory called /home/wtxuser/systems.
  6. Make sure that the SourceEvent trigger is ON. This setting that instructs the Launcher to run the specified map when a new message appears on the specified MQ queue:
    Figure 14. Launcher settings
    Launcher settings
  7. Build the map, generate the system, and securely FTP them to the location specified in the System's MapServerLocation:
    Figure 15. Upload the map and system
    Upload the map and system

Capturing the extended virtual image

  1. From the IBM Workload Deplorer's Catalog / Virtual Images view, click the Capture icon (the one that looks like a padlock):
    Figure 16. Capture the image
    Capture the image
  2. The extended virtual image is imported to the catalog:
    Figure 17. Virtual image catalog
    Virtual image catalog

Deploying the extended virtual image to the cloud

  1. Before you can deploy the extended image, you must create a new virtual systems pattern. From IBM Workload Deployer, navigate to the Patterns / Virtual Systems view, add a new virtual system, and give it a name and description:
    Figure 18. Create a pattern
    Create a pattern
  2. The extended virtual image can be added to the pattern as a part:
    Figure 19. Choose a part
    Choose a part
  3. Deploy this new pattern to the cloud by clicking on the Deploy in the cloud icon (the one that looks like a cloud):
    Figure 20. Deploy in the cloud
    Deploy in the cloud
  4. During deployment, make sure you add a deployment directory that points to the location where you FTP'd the system and map, /home/wtxuser/systems:
    Figure 21. Specify the deployment directory
    Specify the deployment directory
  5. A new virtual system instance is created:
    Figure 22. New virtual system
    New virtual system
  6. Deploy a second virtual system instance using the same pattern. Choose a new virtual system name, but use the same values as before for all of the other settings:
    Figure 23. Another virtual system
    Another virtual system
  7. Once again, make sure you add a deployment directory that points to the location where you FTP'd the system and map:
    Figure 24. Specify the deployment directory
    Specify the deployment directory
  8. A second virtual system instance is created:
    Figure 25. Another new virtual system
    Another new virtual system

    You now have two identical virtual system instances, each of which runs a launcher started with your system. Both instances are waiting for messages to appear on the specified input queue.

Running the systems

  1. Use an MQ client to populate a number of messages to the input queue and then connect the management console to both launchers. Notice that about half of the messages go to the first launcher and about half to the second launcher.
  2. Experiment by adding and removing additional cloned virtual system instances. Use the init pending high and init pending low options to control the launcher's listener activity and prevent a high number of pending maps, using Dtx.ini or the Pending Instance Thresholds setting:
    Figure 26. Management Console
    Management Console

Failover scenario

WebSphere TX with Launcher Hypervisor Edition provides a quick and easy way to implement this scenario and build a scalable solution. It is easy to see how adding additional cloned images can boost the overall consumption of messages from the queue. A second benefit from this implementation is failover -- if one of the clones disappears, the others will continue to consume messages, as shown below:

  1. To begin, make sure that both the input and the output card's transaction scopes are Map:
    Figure 27. Transaction scope
    Transaction scope
  2. Set the launcher's Pending Instance Thresholds to something reasonable, such as a low of 50 and a high of 100:
    Figure 28. Pending Instance Thresholds
    Pending Instance Thresholds
    Don't forget to redeploy the map and system if you modified them.
  3. Next, deploy three instances of the virtual system:
    Figure 29. Three virtual systems
    Three virtual systems
  4. Connect the Management Console to all three Launcher instances:
    Figure 30. Management Console
    Management Console
  5. Use an MQ client application to populate the input queue with a large number of messages. Use 99999 simple messages for this document. The three launchers start to get messages of the queue and transform them:
    Figure 31. History total maps
    History total maps
  6. Simulate a catastrophic failure on one of the launchers by stopping one of the virtual system instances from IBM Workload Deployer:
    Figure 32. Stopping a virtual system instance
    Stopping a virtual system instance
    Figure 33. Stopped virtual system instance
    Stopped virtual system instance
  7. One of the Launchers in the Management Console stops responding. Pay attention to how many messages were processed before it stopped, and note the pending instances. These messages have been browsed (and marked as such) but not retrieved from the queue. Eventually, WebSphere MQ unmarks these messages and makes them available to be browsed by one of the surviving launchers:
    Figure 34. Launcher not responding
    Launcher not responding
  8. The stopped virtual instance can be restarted to simulate another launcher coming online:
    Figure 35. Starting a virtual system image
    Starting a virtual system image
    Figure 36. Started virtual system image
    Started virtual system image
  9. The Management Console now sees three launchers. The history from the newly re-activated instance has reset itself:
    Figure 37. Management Console
    Management Console
  10. When all of the messages have been processed, you should be able to account for all of them:
    Figure 38. All messages processed
    All messages processed
  11. This example started with 99999 messages. Launcher 1 processed 45479, Launcher 2 processed 44752, and apparently Launcher 3 processed 7373. However, before Launcher 3 was stopped, it processed 2395 messages. 45479 + 44752 + 7373 + 2395 = 99999. All of the pending instances at the time the launcher was shut down were reclaimed by WebSphere MQ and processed by other launchers.

Conclusion

This article showed you how to use IBM Workload Deployer to version and administer WebSphere TX with Launcher Hypervisor images and deploy them to the cloud. By taking advantage of the cooperative browsing feature in WebSphere MQ, you have seen how WebSphere TX with Launcher Hypervisor images running in the cloud can create a scalable solution with built-in failover capability.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) is displayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=801955
ArticleTitle=Creating a cloud to consume WebSphere MQ messages using IBM Workload Deployer and WebSphere TX
publish-date=03142012