SupportPacs are add-ons that complement various products in the IBM WebSphere MQ family. SupportPacs available for WebSphere MQ include product extensions, user and administration tools, exits, and much more. Many current WebSphere MQ features first appeared as SupportPacs, where they were refined and improved based on user feedback before finally being integrated into the base product. The Performance Report SupportPacs are created anew for each new product version, while many others have been available for more than a decade.
In short, SupportPacs exist to help you make the most out of your IBM WebSphere software. There are hundreds of SupportPacs available and it would be impossible to do justice to all of them in a single article, so I'm going to tell you about the ones I rely on most:
- IH03: WebSphere Message Broker V6-Message display, test & performance utilities (RFHutil)
- MA01: WebSphere MQ - Q program
- MA0W: WebSphere MQ API Trace
- MO71: WebSphere MQ for Windows - GUI Administrator (mqmon)
- MO72: MQSC Client for WebSphere MQ
- MS81: WebSphere MQ internet pass-thru (MQIPT)
- MH03: WebSphere MQ SSL Configuration Checker
- MO04: WebSphere MQ SSL Wizard
- MS03: WebSphere MQ - Save Queue Manager object definitions using PCFs (saveqmgr)
- MS0P: WebSphere MQ Explorer - Configuration and Display Extension Plug-ins
Links to these (and all other) SupportPacs are listed in the Resources section at the end of this article.
Feel free to use the comment box below to add your favorites to this article.
This SupportPac lives in the Message Broker section of the SupportPacs page. Better known by the name of one of the enclosed the executable modules, RFHUtil, it can be hard to find this utility if you do not know where to look.
IH03 contains a wide variety of programs, but the shining star of the bunch is RFHUtil and the client version RFHUtilc. This GUI utility is a little hard to learn but well worth the effort. Of course, it does the basics such as moving data from files to queues or from queues to files. But in addition to that, it has a number of tabbed dialogs that expose just about every parameter and option available in the WebSphere MQ API. Need to send or parse RFH headers? Check. Need something to quickly test pub/sub? Got it. Need to set or check report options? Confirmation options? Reply-to fields, bind options, persistence or expiry? Done, done, and done!
For programmers, this utility can be extremely useful when you need to test and the program at the other side of the interface is not available. Very frequently there is a need for a specific set of message options or a need to send a message that is not a simple string. RFHUtil can set those options or send messages with binary payloads or any of the various JMS formats.
Administrators can use RFHUtil to verify the contents of message headers and payloads while diagnosing strange behavior in the network. For example, I recently used RFHUtil to figure out why an application was not receiving confirmation messages. The request messages produced by the application had requested Confirmation on Delivery rather than Confirmation on Arrival. Because these types of confirmation require different authorizations, the expected confirmation messages were not produced. RFHUtil enabled me to pinpoint the problem within a few minutes.
There are a number of other utilities in IH03 that can generate load on a queue manager or broker, and I'm sure I will get around to trying them one of these days.
Behind the boring one-line abstract for SupportPac MA01 lies all the versatility of any top-of-the-line kitchen gadget. Many unsuspecting people read this SupportPac contains a simple pipe line program (Q) which takes messages from one source and outputs to a target, then yawn deeply and move on, not suspecting that Q slices, dices, shreds, chops, peels, grates, juices and makes julienne fries.
The power of Q comes from its design as a pipeline command program. (Star Trek fans will protest that the power of Q comes from the Q Continuum but I'll ask them to go with me on this one.) If you are not familiar with the concept of a pipeline program, the idea is to connect two or more programs up in a "pipeline" where the output of the first program becomes the input of the next program. Chain two or three simple programs together in a pipeline and you can build something useful that would otherwise require a lot of coding.
For example, you can pipe the output of any program into Q to create MQ messages, route the messages to a remote queue manager, and then use another instance of Q to convert the messages back into a file. But unlike a file transfer program, which requires a complete file, the pipeline method can accept the lines while they are being written. For example, a program that produces a log file could instead be piped into Q so that the log records were moved to a remote server in real time. With pipelining, the capabilities are limited only by your imagination.
But there's a lot to love about Q even if you don't need the pipeline capabilities. The basic command line options can dump messages in a variety of formats, using destructive reads or browses as needed. Message GET options enable the selection of messages by specific Message ID or Correlation ID, which can come in handy for removing poison messages or scanning to see if a particular message in a test data set was produced as expected. Often a requirement exists to mirror messages across two queues. That's easy to do: just specify multiple output queues and every message in goes to each output queue. The –h option acts as a filter enabling message selection by any arbitrary string. Want to generate test messages in volume? Q can do that too.
If you are among the people for whom tuning and troubleshooting WebSphere MQ is a bit of a black art, then SupportPac MA0W is your acceptance letter to wizard school. MA0W is very much like an MQ trace but with many more useful options and human-readable output.
One of the challenges in debugging is that there are often many layers of system code between the application and the WebSphere MQ API. For example, a Java™ EE server provides many functions that interact with WebSphere MQ outside of application control. Although the server exposes configurations that affect the behavior, the actual code is not available and so it is sometimes necessary to resort to deduction rather than straightforward inspection. MA0W opens up a window into the code by showing you each API call, complete with before-and-after states.
The most recent occasion I had to use MA0W was to figure out why a set of poison messages I was testing with was not requeued. A poison message is one that cannot be processed for some reason and, after having been read by the application under syncpoint, is subsequently backed out onto the queue. The next GET on the queue retrieves the same message. To prevent an infinite loop, IBM's JMS classes inspect the message's backout count, as well as the backout threshold and backout queue attributes of the input queue. Whenever the backout count exceeds the queue's backout threshold, the JMS classes will requeue to the backout queue. This removes the poison message so the application can continue to process other messages in the queue.
Because the poison message handling is built into the JMS classes, there is no code to inspect when the application does not behave as expected. In my testing with a set of poison messages, I could see the depth of the backout queue increase as the poison messages stacked up there -- but when I stopped the test program, all the messages ended up back on the input queue. This was definitely not what I expected and nothing I did caused the messages to end up in the blackout queue.
When I used MA0W to trace the API calls, it all made sense. In normal processing, poison messages are rare. Typically, behind each poison message is a perfectly valid message waiting to be processed. My test dataset however consisted entirely of poison messages. The API trace revealed that the poison messages were requeued under syncpoint and not committed. The unit of work was then extended to include the next message, which was also read under syncpoint. The process continued like this until there were no more messages, at which point my program ended, the unit of work ended, and all the messages were rolled back onto the input queue. Armed with this information, I added a single good message to the end of my test set. The good message was eventually delivered to the program, which then called COMMIT which caused all the poison messages to finally end up on the backout queue.
In my anecdote I referred to the "unexpected behavior" of the application, when in fact I just didn't understand how the system works. Once MA0W showed me the API calls used by the JMS classes, the behavior I saw was normal and expected.
Over the years I've used MA0W time and again to successfully diagnose some of the trickiest problems across a wide variety of platforms. Since it intercepts API calls within the queue manager itself, MA0W is indifferent to the programming language the application uses, and indeed whether the application is local or remotely connected. Among the other options are the ability to trace specific queues or specific processes, which can be extremely valuable on a busy, shared queue manager. MA0W is a highly recommended addition to your WebSphere MQ tool box.
This SupportPac, equally well known as MO71 and by the executable name mqmon, is a Windows® GUI administration tool for WebSphere MQ. Because it is compiled C code, it is small, light, and fast. All you need to do to run mqmon is to unzip it into a folder and double-click the executable. This will instantly give you access to any local queue managers and, assuming you have WebSphere MQ client installed, to any remote ones as well.
What I find most useful about mqmon is that it opens up a new, dedicated window for each function. Although it is possible to open up too many windows and bury your desktop in clutter, I find WebSphere MQ Explorer's single-task paradigm too limiting. If you set WebSphere MQ Explorer for Workbench mode, it is possible to open up many windows but each one is a full version of WebSphere MQ Explorer. With mqmon, I can open up individual windows that just show a single queue, or perhaps a subset of queues and nothing else. Or, I can have a channel status window open while I use another window to set triggering on a transmit queue. In mqmon, windows that show lists of objects have a row for each object and customizable columns for object attributes. Double-clicking on any object drills down to the details for that object, and there are usually several options related to the object type available. Clicking on any value enables you to easily change it, assuming you are suitably authorized.
Of course, mqmon is no slouch when it comes to features. Any object with a settable attribute or status has a corresponding dialog in mqmon. If you like a command line, mqmon has a dialog to enter runmqsc commands directly. There's a network view that shows you at a glance the status of all your queue managers and a lightweight monitoring platform. You can even enable an HTTP listener and connect to mqmon to perform inquiry and administration tasks using a browser.
One of the things that I use mqmon for is to test SSL configurations when I have a non-Java application that can't seem to connect. Whereas Java applications, such as WebSphere MQ Explorer use a Java keystore, other platforms use a Key Database format keystore. Using mqmon, I am able to test the SSL connection using the same keystore files that the application uses. This enables me to either isolate the keystore and certificates as the cause of the problem or rule them out.
MO71 has several other features that are not present in WebSphere MQ Explorer, like the ability to load a file to a queue or a queue to a file. But I won't spoil all of the fun, go download a copy and explore for yourself.
This is another SupportPac where the description is laughably understated. The three-sentence description is as follows:
|The MQSC Client for WebSphere MQ allows the use of MQSC commands either directly connected to a queue manager, or via a client connection to the queue manager. It also provides a number of formatting options for the display of the results to the commands. It may also be used to display or modify the client channel definition table.|
When I've talked about this SupportPac at conferences, many people have asked why not just use runmqsc to do the same things? Let's answer that by looking at the three sentence description again.
First, there is the ability of using client connections to perform script commands on the queue manager. This lets you to run scripted actions across the entire WebSphere MQ network from a single location.
For example, I use MO72 for compliance reporting. Starting with a script that iterates through a list of queue managers, I use MO72 to gather details of the security settings on the queue managers and compare these to a target baseline. Then, about once a week the script e-mails a compliance report letting me know which queue managers on the network, if any, have configuration issues I need to look at.
I have also used the same framework to write scripts that could implement a command across the entire network, look for all instances of a specific CONNAME, or perform automated instrumentation tasks such as failover to the disaster recovery site.
But it is not just the ability to use a client connection that enables this type of automation. The second sentence in the SupportPac description mentions "a number of formatting options" for the output. The output of the built-in runmqsc tool is intended to be human-readable and is printed in two columns, and with several lines for each object. Parsing this output with scripts requires a lot of string manipulation. For example, trying to identify the channels with "localhost" in the CONNAME requires that a script extract the attributes from each line, search for CONNAME attributes, and perform control break processing on each instance of the channel name attribute.
On the other hand, MO71 can generate output where all of the attributes for each object appear on a single line. The same process can now be reduced to using grep to identify all channels with "localhost" in the CONNAME, and then extracting the channel names from the remaining lines. The formatting capabilities of MO72 make scripts much easier to code and faster to run.
The last sentence of the SupportPac description mentions, almost as an afterthought, that MO72 can be used to generate a Client Channel Definition Table (CCDT). The textbook method of generating a CCDT is to use runmqsc to define several CLNTCONN channels on a running queue manager, and then distribute the resulting AMQCHL.TAB file. What is not widely known is that this method does not do any "garbage collection" inside the resulting file. When an entry is deleted, it leaves an empty hole in the file. It might be reused but it will never be compacted. As a result, the CCDT file has a tendency to grow over time when managed this way. MO71 is capable of generating a new file from scratch on every run. The result is that you always get the smallest possible CCDT file with exactly the set of definitions you need.
Even more compelling is that MO72 does not require a running queue manager to generate the CCDT file. One of my past instrumentation projects was to use some CGI scripts and an Apache Web server to build Web pages with a list of available queue managers. When the administrator selected the queue managers of interest and clicked the Submit button, the CGI scripts built a CCDT file on demand and downloaded it to the administrator's browser.
Of course, you could just as easily drive MO72 by hand or with simple shell scripts. Since the CCDT is a compiled artifact, it is appropriate to include the source definitions into a project's change management system. I know of one shop doing this and they use ant scripts to generate the CCDT as part of their application build. This type of automation results in improved consistency, repeatability, and reduced defects.
The "IPT" in MQIPT stands for “Internet Pass-Thru.” This SupportPac provides a control point in the DMZ for WebSphere MQ channels. A single MQIPT instance can manage connections from many external business partners and route them to any number of internal queue managers. Each connection path is separately managed and independently controlled.
MQIPT can also make firewall traversal easier by tunneling the WebSphere MQ channel protocols inside of HTTP or HTTPS. Typically, two business partners will each set up an MQIPT node in their DMZ such that the pair of MQIPT nodes sit between two queue managers and the two firewalls in the center. The queue managers see only a normal WebSphere MQ channel and the firewall sees only HTTP or HTTPS traffic.
I prefer MQIPT in the DMZ over placing a full queue manager there. My reasoning is that MQIPT does not queue any data to disk so in the event of a channel failure, there is no data piling up in the DMZ. As an administrator of WebSphere MQ, I also like the ability to enable or disable external access before the connection request ever reaches the queue manager. The same result can be achieved by closing the port on the firewall, but that requires a request to a separate team. With MQIPT, the MQ administrator can directly manage that access.
When MQIPT participates in an SSL connection, it makes some of the certificate details available to a user-written exit. What is especially helpful is that MQIPT exposes more certificate fields than WebSphere MQ does, so if you need to check the issuer's Distinguished Name, for example, you can do that in an MQIPT exit.
MQIPT is an excellent tool to help secure and control external connections on your WebSphere MQ network. Just remember that it augments but does not replace the hardening you would ordinarily do at the queue manager.
This SupportPac will examine the SSL configuration settings for a queue manager and, optionally, for a client as well, and report on any issues that it finds. There is no complicated installation to perform, just drop the self-contained executable on the host where the queue manager lives and run it. The program comes compiled for AIX®, HP-UX, Linux®, Solaris™, and Windows.
When the program finds an issue, a detailed report is printed, which typically includes a short description of the problem, an "advice" section with a more detailed description of the problem, suggested possible resolutions, and any known exceptions.
Because MH03 is a self-contained executable, it is very easy to set up and run. One of the reasons this is among my most used SupportPacs is how much I rely on it to help me with remote support. When I'm onsite, I prefer to have my client do the SSL verification tasks by hand to gain a better understanding of how the components all interact. But over the phone it often boils down to how fast you can get the problem solved. MH03 does all of the same verification tasks you would do by hand, but it does them faster and without forgetting anything. If you are using SSL on your WebSphere MQ network (and who isn't these days?), then you need MH03.
The MO04 SSL Wizard is one of my "most used" SupportPacs that I don't actually use that much. The way I use it these days is to give it to someone who is just getting started learning WebSphere MQ SSL so that they can learn from it.
MO04 is a Java-based GUI that walks you through an interview process to collect the requirements for connecting two queue managers (or a client and a queue manager) over SSL-enabled channels. Once the details for both sides of the SSL connection are collected, the SSL Wizard generates a very comprehensive process, including both narrative description and the necessary commands. The commands include the actual values for things like the queue manager name, channel names, and certificate details, and are intended to be run as-is.
The output produced is extremely helpful for understanding the process and answering questions like, Am I supposed to export or extract that certificate? The only caveat I have is that a few of the commands generated contain syntax errors. However, the output of the program is so useful and complete in other respects that the minor flaws can be forgiven.
My most-used SupportPac of all time is MS03, better known simply as saveqmgr. When it comes to backing up the queue manager, there are two schools of thought. Some people prefer to back up the filesystem that contains the queue manager configuration, while others prefer to back up the meta-data about the configuration. I fall into the latter group.
When WebSphere MQ is backed up at the filesystem level, the backup contains any messages that happened to be on the queue manager at that point in time. If the queue manager is restored, so are all of the messages that were on the queues. Unless every associated program and database upstream and downstream are restored to the same point in time, the restore puts the system into an undefined state which must be reconciled to a known point of synchronization to avoid duplicate or missing transactions.
That assumes the queue manager can be restored in the first place, which is not necessarily guaranteed. Many shops take that filesystem backup while the queue manager is running. This results in the backup of the queue files being slightly out of sync from the backup of the log files, and the possibility that one or more of the files will be corrupted.
For these reasons, I much prefer to back up the queue manager meta-data. This includes the object definitions, authorization profiles, SSL keystores, ini files, and the contents of any exit directories. The saveqmgr program captures the queue manager object definitions for all versions of WebSphere MQ. For version 6.0 and higher, saveqmgr can also capture the authorization profiles. Then, if it is necessary to recover a queue manager, you can recreate it empty so that there is no need to reconcile messages that might be days or weeks old and which were inadvertently restored with the queue manager.
Like MO72, saveqmgr also comes in a client version which I use extensively. Naturally, this version is saveqmgrc. I have seen many cases where saveqmgr was used locally to capture the queue manager configuration data. But for this data to be useful after a disk or server crash, it must be stored on another server. Since I am such a big fan of scripting and automation, my solution is to put saveqmgrc on a central server (probably the same one where MO72 lives) and use scripting to iterate through all of the queue managers and collect the object definitions and authorizations.
Typically, I store the output files on this central server in a tree structure with a separate directory for each queue manager. One of the benefits of this is that it becomes a flat-file database that is searchable with simple UNIX® find commands or some lightweight scripts. This makes it easy to answer questions such as, When exactly did that queue disappear?
Event messages are one of the best kept secrets in WebSphere MQ troubleshooting. When I need to debug an authorization problem, I enable Authorization Events and recreate the error. In years past, I would then pull out a copy of the C header files and my trusty hexadecimal calculator and parse the event messages field by field. Sure, it was tedious. But the event message told me the User ID that generated the error, the API call that failed, the object that was the target of the API call, and the options that were used. The ability to diagnose an authorization problem from a single event message was more than worth the trouble of parsing hexadecimal fields by hand.
Or at least it was until MS0P was released. MS0P will parse Authorization or any other type of event message into human-readable text. Thanks to this SupportPac, I have been able to retire my hex calculator. In fact, I've gotten so lazy, I'm not sure if I'd remember how to parse an event message by hand any more.
If that were the only thing this SupportPac did, it would be more than worth the download. In fact, there is a lot more here. This same plug-in also supplies a trace route function that can tell you which queue managers a message flows through. It also can display queue activity in a graph format.
A different plug-in in this bundle provides the ability to perform remote management of WebSphere MQ servers. The plug-in can manage Windows, UNIX, and Linux systems, and can start or stop the queue managers there and perform other tasks that are normally considered "command-line only."
A third plug-in has the capability of saving queue manager object definitions in a Comma Separated Values (CSV) format. This enables the object definitions to be processed by Microsoft Excel or loaded easily into a database.
Finally, the SupportPac contains two utility programs that can be quite helpful. The mqidcode utility translates decimal or hexadecimal numeric values into their human-readable counterparts. For example, you might have the output of a trace and need to know if a GET operation specified SYNCPOINT. Passing the numeric value of the Open Options field to mqidecode gets the answer without any need for C header files and a calculator. The other utility is qtune and its purpose is to expose the queue buffer settings so that you can inquire on them and set them. In high-volume situations where the queue manager must be tuned for optimum performance, it is very common for the queue buffers to have been changed from the default settings. Once they have been tuned, it is helpful to be able to inquire on them to verify the tuning or to capture the configuration in a backup. But until now there has been no easy way to inquire on the actual settings. In addition to making that inquiry easy, qtune can also make the changing the value easier as well.
I hope this guided tour of my favorite SupportPacs inspires you to look through the SupportPac Web site and try out a few new tools. There are literally hundreds of SupportPacs available so no article could ever hope to provide adequate coverage of them all, and no two people are likely to have the same list of favorites. Did I omit your favorite SupportPac? Use the comment form below to nominate your pick for World's Best SupportPac and be sure to say why you chose it.
Links to each of the SupportPacs discussed here are provided below in the Resources section.
SupportPac main page
Podcast: The Deep Queue
Author's Web page: T-Rob.net
Get products and technologies
Message Broker V6-Message display, test & performance utilities (RFHutil)
MA01: WebSphere MQ - Q program
MA0W: WebSphere MQ API Trace
MO71: WebSphere MQ for Windows - GUI Administrator (mqmon)
MO72: MQSC Client for WebSphere MQ
MS81: WebSphere MQ internet pass-thru (MQIPT)
MH03: WebSphere MQ SSL Configuration Checker
MO04: WebSphere MQ SSL Wizard
MS03: WebSphere MQ - Save Queue Manager object definitions using PCFs (saveqmgr)
MS0P: WebSphere MQ Explorer - Configuration and Display Extension Plug-ins
IBMers' Blog on Messaging
The Vienna WebSphere MQ List server
developerWorks WebSphere MQ forum
T.Rob Wyatt is a Senior Managing Consultant with IBM Software Services for WebSphere who assists customers with the administration, architecture, and security of WebSphere MQ. Recently he has focused on WebSphere MQ security, publishing in the IBM WebSphere Developer Technical Journal and presenting at the IMPACT and European Transaction and Messaging conferences. T.Rob also hosts The Deep Queue, a monthly WebSphere MQ security podcast.