Queue Here: MQ, Integration, Services and Clouds
Mark Phillips 100000P36B 1,416 Visits
Melanie Rao and I have published an article on DeveloperWorks which demonstrates how you can push Java applications into BlueMix and connect them using Elastic MQ. The applications use the worker offload pattern described below.
IBM BlueMix has support for a wide variety of Java applications: You can write and deploy enterprise style Java web applications into the Liberty buildpack in the form of ear or war files; you can push an entire Liberty server package and the applications running in it; or you can deploy standalone JSEE applications packaged in JARs.
The article demonstrates two of these Java application styles - pushing a servlet web application in a Liberty server using the Liberty buildpack, and pushing a JSE application packaged in a jar file using the Java buildpack.
The application itself is deliberately kept simple in order to clearly demonstrate the "worker offload pattern" - using the ElasticMQ messaging service to distribute a workload from a Liberty front-end web application to a pool of back-end JSE worker applications. The processing workload in this example is trivial - converting words to uppercase - but the same principle and technique can be applied to any background task which can be offloaded to keep the user interface responsive. BlueMix makes it very simple to scale an application by increasing or decreasing the number of worker applications and the web front-end in this example displays the id of the worker applications which process each message, so you can see how the workload is distributed amongst the workers.
The application source code is available in a git repository on IBM DevOps Services so you can build it yourself and use it as the basis for your own Java worker offload application.
See the BlueMix Liberty documentation more information about pushing applications into BlueMix using the Liberty buildpack.
For the last few months the MQ Cloud team has been working on a new hosted cloud messaging service which is provisionally called "ElasticMQ". ElasticMQ has been available in limited beta on BlueMix since February and has just been refreshed with support for the MQLight API.
ElasticMQ offers a hosted messaging middleware service for Java or Node.js applications running in BlueMix (IBM's public cloud application platform). As we've worked on ElasticMQ we've taken lessons learned from developing WebSphere MQ over the years and distilled that experience into a messaging service which aims to be easy to learn and use and which offers great reliability and qualities of service.
To make ElasticMQ simple to use we have leveraged the BlueMix deployment model, and have worked to automate or eliminate the administrative steps that you typically associate with setting up a messaging system. To pick a couple of examples:
If you'd like more information about the new APIs take a look at the MQLight API and Concepts page. To experiment with ElasticMQ you can try it for free by signing up to the IBM BlueMix Beta and deploying the sample applications.
If you're going to IBM's 2014 Impact conference then Rob Nicholson will be presenting on MQLight and ElasticMQ, but don't worry if you can't make it... his slides are available on slideshare and he has posted a video of his demo on YouTube (with the demo code available on github).
We have plans to support more languages with ElasticMQ and for more connectivity options so watch this space for more news.
Mark Phillips 100000P36B 915 Visits
The MQ hypervisor edition provides an easy way to get a virtual machine deployed with a running queue manager in just a few minutes. Once you've deployed a VM you'll typically want to configure MQ further by creating queues and channels and authorising users to access the system. This post describes how you can automate the configuration of these resources by running a custom script package.
Script packages are a feature of IBM's SmartCloud family which greatly extend the power of patterns - allowing you to automate many of the complex setup tasks which you need to execute when building up a topology of virtual machines and middleware. If you're not familiar with script packages then this article (written when the concept was introduced in WebSphere Cloudburst) tells you all you need to know to get started.
To quickly summarise, a script package is simply the combination of a zip (or tgz) file and some metadata that tells the system what to do with the zip. You can put pretty much anything you like into the script package - including executable scripts, binary files, and data. In the metadata you specify when to run the script package (either when;a VM is created, destroyed, or on demand through the life of the VM), which executable should be run, and what parameters should be passed to the executable. When the script package is executed then the zip file is transferred to the VM and unpacked and the named executable is run.
Trying it out
The example script package configures a newly deployed queue manager so that a MQ client application can connect to it. iI shows how you can automate the creation of queues and users and add a server connection (SVRCONN) channel called USER.SVRCONN. It grants authority to the new users so that they can connect to the queue manager and so that they can browse, put, get, and inquire the new queues. The queues can be optionally added into an MQ Cluster.
Uploading the Script Package
Follow these steps to upload the script package:
Using the Script Package in a Pattern
To use the script package follow these steps to create a virtual system pattern:
Deploying the Script
To test the script, follow these steps to deploy the pattern specifying values for an user and queues.
Testing the Script Package
Once you have deployed a machine containing the script package, you can attached MQ clients to the queue manager and put and get messages to the queues you created using the users you specified.You will need to have installed the MQ samples and MQ clients packages on your computer to run the samples.
One of the things we've been working on for the last few months here in MQ development in Hursley is the latest version of MQ virtual application pattern - V18.104.22.168 of the IBM Messaging Extension for Web Application Pattern - so I'm delighted to say that we released it today.
Take a look at this post tor an introduction to the pattern. The full details about the latest update are in the announcement letter but here's a quick summary of the highlights :
The updated patterntype is available from the PureSystems Centre , Passport Advantage, or Fix Central (search for program number 5725-G55).
This post shows you how to load the MQ 7.0.1 Hypervisor Edition (HVE) image to a IBM Workload Deployer (IWD) appliance or Pure Application (PureApp) System. (Note that version 7.5 of the WebSphere MQ 7.5 Hypervisor Edition already includes CLI scripts to upload the Hypervisor image so this tip only applies to the original MQ 7.0.1 Hypervisor Edition image).
The MQ Hypervisor Edition is available from Passport Advantage - in this example we'll use MQ HVE 22.214.171.124 which is packed in a tarball called CI6PWML.tgz. Download the tarball and unpack it into an empty directory on the same machine as the CLI. You should end up with a directory like this...
The files with a *.zip suffix are examples of script packages (useful for tailoring your patterns) and the large file with a *.ova suffix is the MQ HVE image itself. The ova file contains the disk images for the virtual machine, the machine description, plus the metadata and part descriptions that make the Hypervisor Edition work well with IWD and Pure App Systems.
To complete your preparation, download the MQHVE-RHEL-loader-scripts.zip file by clicking this link and unpack it into the directory where you unpacked CI6PWML.tgz.
Loading the HVE and script packages and creating patterns
The Pure (or IWD) CLI you downloaded earlier contains the python runtime and everything else you need to run the scripts. Before you go any further make sure you know the hostname of your appliance, and have a user Id with sufficient permission to create catalogue content.
To load the MQ HVE image to the appliance, switch to the directory which contains the HVE image and the loader scripts and run the following command. This uploads the ~3Gb HVE ova file so it takes a while to run.
[PATH_TO_CLI]/pure.cli/bin/pure -a -h [PURE_IP_ADDRESS] -u [USER] -p [PASSWORD] -f MQHVE-RHEL-AddVirtualImage-7018.py
NOTE: Before you can do anything with the image you need to log onto your appliance and accept the license(s).
This next script creates two very simple patterns that you can use to deploy the MQ HVE
[PATH_TO_CLI]/pure.cli/bin/pure -a -h [PURE_IP_ADDRESS] -u [USER] -p [PASSWORD] -f MQHVE-RHEL-createPatterns-7018.py
...and this final script uploads the two MQ script packages. Make sure that WMQExecuteMQSC.zip, and WMQConfigureClustering.zip are in the directory when you run this one.
[PATH_TO_CLI]/pure.cli/bin/pure -a -h [PURE_IP_ADDRESS] -u [USER] -p [PASSWORD] -f MQHVE-RHEL-addScriptPackages.py
You can use these loader scripts without modification for the MQ 126.96.36.199 Linux HVE but you will need to change them to upload alternative versions of the HVE, or to upload the PowerVM version of the HVE. If you want to load the scripts to IWD then you will need to substitute the word 'deployer' for the word 'pure' in the commands, and remove the '-a' flag.
Mark Phillips 100000P36B Tags:  vapp iwd podcast pureapplicationsystems patterns mq vsys cloud virtualization 1,592 Visits
My previous blog post gave an overview of the MQ Patterns available for IBM SmartCloud. If that interested you then you might like to listen to this podcast where the very talented John McNamara (from MQ User Technologies) interviews me on the same subject.
While you're listening to the pod I can recommend reading some of the other great material on the MQdev Community and listening to the other podcasts in the series.
Mark Phillips 100000P36B Tags:  patterns vapp virtualization cloud vsys mq 1 Comment 3,660 Visits
In this first blog entry I want to provide a brief overview of IBM's current cloud technologies,and then introduce the two types of WebSphere MQ products that we currently have available: the WebSphere MQ Hypervisor Edition; and the IBM Messaging Extension for Web Application Pattern Type.
Introduction to IBM SmartCloud
IBM's SmartCloud family and Pure Application systems provide a flexible cloud platform for Infrastructure-as-a-Service (IAAS) and Platform-as-a-Service (PAAS). You can use this platform to deploy compute resources (like virtual machines), and compute workloads (for example applications), and then to manage those deployed resources through an intuitive GUI.
The deployed resources can be placed on hardware owned by your organisation (that is your private cloud); on hardware hosted by IBM (making them part of your public cloud presence); or they can be deployed in an architecture which combines private cloud and public cloud deployments (making it a hybrid cloud).
There are effectively three tiers to the products that are available.
IBM Workload Deployer, Pure Application Systems and IBM Smart Cloud Application services all share the same “pattern” concept – described below – giving you access to a continuum of deployment options. You can deploy a pattern on your existing hypervisor technology; buy dedicated Pure Application Systems to run your own private cloud; or outsource the hosting of your deployed workloads to IBM. Thus your are able to distribute your compute resources and skills across private, hybrid, and public clouds in whichever way is quickest, most efficient, and cost effective.
Virtual System Patterns and MQ Hypervisor Editions
A number of IAAS technologies - both proprietary and open-source - are available to speed up your deployment and configuration of new computing resources. In the majority of these technologies the essential unit of deployment is a virtual machine - typically from a catalogue containing template images which define the software and configuration of the machine.
In common with other IAAS solutions, IBM's SmartCloud offers the ability to deploy single virtual machines as Virtual Appliances. A virtual appliance typically starts life as a standard operating system image or an IBM hypervisor edition image (which contains an operating system plus middleware). This base image can be deployed as a VM and extended with additional software. Once the extra software has been installed and configured, the machine is recaptured back into the catalogue, and is available as a new template for virtual machine deployment.
IBM SmartCloud extends this single-machine deployment concept with the idea of Virtual System Patterns, making it possible to model more complex topologies of machines and to deploy them together. A large WebSphere Application Server cluster is a good example of a virtual system pattern. It typically contains a number of machines running HTTP servers to act as a web tier; a number of machines running WebSphere Application Server to host the application(s); and a deployment manager to manage the application server instances. Individual machines in the virtual system pattern can be tailored with script packages which are executed on the virtual machine instances. Script packages can be run when the instance is deployed or removed or on an add-hoc basis through the lifetime of the virtual machine, so they can be used to configure a machine when it starts, or to clean up a machine before it is destroyed.
To make it simple to use WebSphere MQ in conjunction with virtual system patterns we developed the WebSphere MQ Hypervisor Edition (HVE) which combines MQ and an operating system in a single virtual machine image. Two variants of the MQ HVE are available – one for Red Hat Enterprise Linux and one for AIX. These can be deployed to VMware or PowerVM cloud groups respectively.
The MQ Hypervisor Edition has a number of configurable options,which can be set in the virtual system pattern and locked, or left open for the user to specify when the MQ HVE is deployed. For example the user can specify the amount of memory and virtual CPUs that the instance should have, the name of a queue manager to be created in the machine, and the port that the queue manager should listen on. When the machine is deployed, a queue manager is automatically configured and set to automatically start and stop with the VM. For a full list of properties see the MQ Hypervisor part description in the MQ Infocenter.
Once a virtual system pattern has been deployed, then the status of the machines inside it can be viewed from the IWD / Pure Application system console, and maintenance can be applied to the machines from the catalogue of available FixPacks.
For example, the screenshot on the left below shows a pattern containing three virtual system parts which have been customized so that they implement a multi-instance queue manager pair with a shared disk. The screenshot on the right shows the deployed instance of the virtual system pattern and the status of the virtual machines it contains. In future blog posts I will show some examples of virtual system patterns and script packages which implement messaging systems.
To summarise, you can use the MQ Hypervisor Edition with IBM SmartCloud to automate and standardize the traditional approach to provisioning messaging systems. The benefits of this approach include :
Virtual Application Patterns and the Messaging Extension for Web Application pattern type
The previous section describes how virtual system patterns provide a quick and easy way to provision topologies of virtual machines and to configure them with middleware and other software. The main focus for modelling virtual system patterns is the virtual machine.
IWD and Pure Application systems also offer another class of patterns called Virtual Application (vApp) Patterns where the emphasis is on modelling applications and their dependencies rather than systems. When you deploy a virtual application pattern to IWD or a Pure Application System then one or more virtual machines are created to host the application and all the associated middleware is installed and configured.
A range of application types are supported by different types of vApp patterns – for example there is a vApp pattern type that can deploy a stand-alone Java application, and another pattern type (the Web Application pattern type) for deploying a Java Enterprise application into a web or servlet container. The web application pattern type is most interesting from an WebSphere MQ perspective, because it is the web application pattern type that we have extended for connecting applications to MQ. The IWD and Pure Application System infrastructure is extensible so you can add your own vApp patterntypes if you find that there isn't already a patterntype that meets your requirements.
Creating or editing a virtual application pattern is simple. For instance, to model a JEE virtual application you start up the vApp pattern editor and effectively drop a WAR file or an EAR file containing the application onto the canvas. Once you've done this the pattern editor opens the WAR or EAR file and introspects the application's deployment descriptor to works out the list of resources that the application is dependent on. The pattern editor then prompts you to satisfy the application's dependencies by dragging and dropping the resources that the application needs and wiring them to the application.
The example in the pattern on the right shows a web application which has a dependency on a JMS Queue and a JMS ConnectionFactory. These dependencies have been expressed in the model as a Queue node and an "Existing Messaging Service" node. The Existing Messaging Service node represents a client connection to an MQ queue manager (so hostname, port, and channel details details are specified as properties of the node – not shown here). When the application is deployed, a virtual machine containing WebSphere Application Server is created to host the application, the application is deployed into the application server, and JNDI objects are created to represent the Queue and ConnectionFactory. When the application is started it can get the JNDI objects out of the JNDI repository and connect to the MQ Queue Manager.
Queues, Topics, and Existing Messaging Service nodes are all available as standard in the Web Application pattern type which is included with IWD and Pure Application Systems and they allow you to connect to a "existing" MQ queue manager which has been defined outside the pattern. If you would like an MQ queue manager to be created for you when you deploy an application pattern then you need to use the IBM Messaging Extension for Web Application Pattern Type. The messaging extension is a separate software product that you need to download from Passport Advantage and upload to your Pure Application System or IWD machine.
The IBM Messaging Extension adds a sample application pattern and template to the Web Application patterntype, and a new node (the Messaging Service) to the palette of the Web Application vApp pattern editior. The sample application is shown below.
When you deploy a pattern containing a Messaging Service node then a new VM is created containing WebSphere MQ, a queue manager is created, and the queues represented on the canvas are created. JNDI objects are then created for the ConnectionFactory, Queue, and Topic resources and they are linked into the application.
Virtual application patterns allow you to deploy an application without needing to worry about configuring the container that the application runs in or the middleware it uses. To help hide those configuration details, you can access the common operations that need to be performed on the application's middleware and application server via the pattern instance's management panels. Once the pattern has been deployed basic monitoring functions are also installed into the virtual machine so that MQ activity can be displayed graphically in the management panels.
The diagram below shows the machine topology which is created when the above application is deployed, along with the machine status display and MQ monitoring graphs.
This blog entry describes the two types of WebSphere MQ products that are available for IBM Cloud environments. The following table compares products to highlight the different use cases for each product:
Take a look at the following links if you'd like to know more about the products I've described: