This article describes the new capability in IBM® WebSphere® Message Broker (hereafter called Message Broker) for handling failover and providing high availability (HA). It will help you create and configure multi-instance queue managers and brokers. This article also provides step-by-step procedures to develop and test the Message Broker HA environment. The target audience for this article includes senior IT architects and IT specialists who design SOA solutions and need to implement HA ESB solutions.
To understand Message Broker multi-instance broker feature (hereafter called MI broker), you must first understand how the multi-instance queue manager (hereafter called MI queue manager) works. This article presents a conceptual overview of MI queue manager instances and the tasks for administering them, followed by a demonstration of how you can switch over between multiple instances. After that, the article describes the concepts of MI brokers, including their setup and how to configure and test them for high availability.
Multi-instance queue managers
The term MI queue manager refers to the combination of active and standby instances of the queue manager that share the queue manager data and logs. MI queue managers protect applications against the failure of queue manager processes by having one instance of the queue manager active on one server, and another instance on standby on another server, ready to take over automatically should the active instance fail. Replicating queue manager instances is an effective way to improve the availability of queue manager processes.
Examples in this article were run on a system using WebSphere MQ and Message Broker, with three servers running Red Hat Enterprise Linux 5.0. Figure 1 shows an overview of the MI queue manager and MI broker components on these three servers:
Figure 1. Overview of MI queue manager and MI broker components
The three servers are:
- Hosts the primary active instance of the MI queue manager IBMESBQM and the primary active instance of the MI broker IBMESBBRK
- Hosts the duplicate standby instance of the MI queue manager IBMESBQM and the duplicate standby instance of the MI broker IBMESBBRK
- Hosts the shared network file system /mqha through NFS V4
Though it's not mandatory to run the standby instance and host the shared file system on separate servers, it is advisable to prevent the queue manager from failing due to a hardware problem. In this example, WebSphere MQ is installed on two servers: wmbmi1.in.ibm.com for the primary MI queue manager, and wmbmi2.in.ibm.com for the standby MI queue manager. As shown in Figure 1, the active instance of IBMESBQM is running on wmbmi1.in.ibm.com; the other instance, running as standby on wmbmi2.in.ibm.com, is doing no active processing and is ready to take over from the active instance if it fails.
The tasks to administer these MI queue managers are described below.
Configuring the networked file system
An MI queue manager uses a networked file system to manage queue manager instances. The queue manager automates failover using a combination of file system locks and shared queue manager data and logs. Before V7.0.1, WebSphere MQ did not support networked storage accessed as a shared file system. If queue manager data was placed on shared network storage, you had to ensure that this data was not accessed by another instance of the queue manager running at the same time. The situation is improved in WebSphere MQ V7.0.1: queue managers can acquire read/write locks to provide access to their data; at any given time, only one instance of a queue manager is allowed to lock data, eliminating dirty reads.
Prior to configuring NFS, you need to ensure that the user ID (uid) and group ID (gid) of the user are the same for all the servers where MI queue manager instances reside -- wmbmi1.in.ibm.com, wmbmi2.in.ibm.com, and wmbmi3.in.ibm.com, in our example. This also applies to the NFS server. On Linux and UNIX systems, the uid used to run the WebSphere MQ and WebSphere Message Broker commands must be a member of both the mqbrkrs group and the mqm group, and the UNIX gids of the mqbrkrs and mqm groups must be the same on all the systems. Additionally, you should make sure that the uid and its accompanying gid for this uid are the same on all the systems, and is the same uid that created the first instance of the queue manager. Change uids and gids in Linux and UNIX environments with caution, because doing so affects the permission levels of files on the system. Changing a uid or gid causes the ownership of all the files previously owned by that user or group to change to the actual integer representing the file's previous owner. The system administrator must ensure that the ownership of the affected files and directories are manually restored.
In this example, the user mqm administers the MI queue manager and MI brokers. Use the commands in Listing 1 to match the uid and gid for the mqm user in /etc/passwd, then reboot the system if you need to change the values:
Listing 1. Matching the uid and gid of the mqm user on all member servers
[firstname.lastname@example.org]$ cat /etc/passwd | grep mqm mqm:x:500:501:MQSeries User:/home/mqm:/bin/bash [email@example.com]$ cat /etc/passwd | grep mqm mqm:x:500:501:MQSeries User:/home/mqm:/bin/bash [firstname.lastname@example.org]$ cat /etc/passwd | grep mqm mqm:x:500:501:MQSeries User:/home/mqm:/bin/bash
Additionally, you also need to set the uid and gid of the mqm group to be
identical on all the systems. Create log and data directories in a common,
shared folder named /mqha. Make sure that the mqha directory is owned by
the user and group mqm, and that the access permissions are set to
rwx for both user and group. The commands in Listing 2,
executed as root on wmbmi3.in.ibm.com, will achieve this:
Listing 2. Creating and setting ownership for directories under the shared folder /mqha
[email@example.com]$ mkdir -p /mqha/WMQ/IBMESBQM/data [firstname.lastname@example.org]$ mkdir -p /mqha/WMQ/IBMESBQM/logs [email@example.com]$ mkdir -p /mqha/WMB/IBMESBBRK [firstname.lastname@example.org]$ chown -R mqm:mqm /mqha [email@example.com]$ chmod -R ug+rwx /mqha
Next, you need to configure the NFS server on wmbmi3.in.ibm.com and then start it on the same machine. Add the excerpt in Listing 3 to the /etc/exports file, which should be executed as root on wmbmi3.in.ibm.com:
Listing 3. Configuring the NFS server
Start the NFS server by executing the command in Listing 4 as root on wmbmi3.in.ibm.com:
Listing 4. Starting the NFS server
If the NFS server is already running, refresh it using the command in Listing 5:
Listing 5. Refreshing the NFS server
Now the shared directory /mqha on wmbmi3.in.ibm.com should be mounted on both wmbmi1.in.ibm.com and wmbmi2.in.ibm.com. Use the command in Listing 6 to check whether the shared /mqha directory is visible from both instance hosting servers:
Listing 6. Checking to make sure /mqha is mounted
showmount –e wmbmi3.in.ibm.com
If the output of the command in Listing 6 is negative, you will need to mount the shared folder from the two instance hosting servers. Execute the command in Listing 7 as root on both of the servers to mount the exported file system:
Listing 7. Mounting the exported file system
mount -t nfs4 -o hard,intr wmbmi3.in.ibm.com:/ /mqha
Not all network file system locking protocols are robust, and a file system
might be configured for performance rather than data integrity. You must
amqmfsck command to test whether your network file
system will properly control access to queue manager data and logs:
amqmfsckwithout any options on each system to check basic locking.
amqmfsckon both WebSphere MQ systems simultaneously, using the
-coption, to test writing to the directory concurrently.
amqmfsckon both WebSphere MQ systems simultaneously, using the
-woption, to test waiting for and releasing a lock on the directory concurrently.
To work reliably with WebSphere MQ, a shared file system must provide:
- Data write integrity
- Guaranteed exclusive access to files
- The release of locks upon failure
If the file system does not provide these features, the queue manager data and logs may become corrupted when the shared file system is used in an MI queue manager configuration. Currently, NFS V4 provides all the above mentioned facilities, so it is the file system in use in this example.
Creating a multi-instance queue manager
Start by creating the MI queue manager on the first server, wmbmi1.in.ibm.com. Log on as the user mqm and issue the command in Listing 8:
Listing 8. Creating a queue manager
crtmqm -md /mqha/WMQ/IBMESBQM/data -ld /mqha/WMQ/IBMESBQM/logs IBMESBQM
Once the queue manager is created, display the properties of this queue manager using the command in Listing 9:
Listing 9. Displaying the properties of the queue manager
[firstname.lastname@example.org]$ dspmqinf -o command IBMESBQM addmqinf -s QueueManager -v Name=IBMESBQM -v Directory=IBMESBQM -v Prefix=/var/mqm –v DataPath=/mqha/WMQ/IBMESBQM/data/IBMESBQM
Copy the output from the
dspmqinf command and paste it on the
command line on wmbmi2.in.ibm.com from the console of user mqm, as shown
in Listing 10:
Listing 10. Configuring wmbmi2.in.ibm.com
[email@example.com]$ addmqinf -s QueueManager -v Name=IBMESBQM -v Directory=IBMESBQM -v Prefix=/var/mqm -v DataPath=/mqha/WMQ/IBMESBQM/data/IBMESBQM WebSphere MQ configuration information added.
Now display the queue managers on both servers using the
command on each. The results should look like Listing 11:
Listing 11. Displaying the queue managers on both servers
[firstname.lastname@example.org]$ dspmq QMNAME(IBMESBQM) STATUS(Ended immediately) [email@example.com]$ dspmq QMNAME(IBMESBQM) STATUS(Ended immediately)
The MI queue manager IBMESBQM has now been created both on the server wmbmi1.in.ibm.com and on the server wmbmi2.in.ibm.com.
Starting and stopping a multi-instance queue manager
Starting a queue manager in multi-instance mode is as simple as starting a
normal queue manager. To start the queue manager instances in either
order, issue the
strmqm command with the
parameter, as shown in Listing 12:
Listing 12. Starting the queue manager in multi-instance mode
[firstname.lastname@example.org]$ strmqm -x IBMESBQM WebSphere MQ queue manager 'IBMESBQM' starting. 5 log records accessed on queue manager 'IBMESBQM' during the log replay phase. Log replay for queue manager 'IBMESBQM' complete. Transaction manager state recovered for queue manager 'IBMESBQM'. WebSphere MQ queue manager 'IBMESBQM' started. [email@example.com]$ strmqm -x IBMESBQM WebSphere MQ queue manager 'IBMESBQM' starting. A standby instance of queue manager 'IBMESBQM' has been started. The active instance is running elsewhere.
The queue manager is first started on wmbmi1.in.ibm.com and later on wmbmi2.in.ibm.com. Hence, IBMESBQM starts as an active instance on wmbmi1, processing all requests, and as a standby instance on wmbmi2, ready to take over if the active instance fails. The active instance is granted exclusive access to the queue manager data and logs, while the standby waits to be granted such access; if the standby is granted exclusive access, it becomes the active instance.
You can also always start a queue manager configured with multiple
instances on different servers as a single-instance queue manager. Should
the queue manager be started using the
-x option), the instances of the queue manager
configured on other machines are prevented from starting as standby
instances. If an attempt is made to start another instance, the user would
receive the response that the queue manager instance is not permitted to
run as a standby.
Only two queue manager instances can run at the same time: an active instance and a standby. Should the two instances be started at the same time, WebSphere MQ has no control over which instance becomes the active instance; this is determined by the NFS server. The first instance to acquire exclusive access to the queue manager data becomes the active instance.
Now that the queue manager has been started, you can create and start the queue manager listener for applications to connect the active queue manager IBMESBQM, as shown in Listing 13. All administration commands in Listing 13 are issued on the active queue manager:
Listing 13. Creating and starting the queue manager listener
[firstname.lastname@example.org]$ runmqsc IBMESBQM 5724-H72 (C) Copyright IBM Corp. 1994, 2009. ALL RIGHTS RESERVED. Starting MQSC for queue manager IBMESBQM. define listener(IBMESBLISTENER) trptype(tcp) port(1414) control(qmgr) 1 : define listener(IBMESBLISTENER) trptype(tcp) port(1414) control(qmgr) AMQ8626: WebSphere MQ listener created. START LISTENER(IBMESBLISTENER) 2 : START LISTENER(IBMESBLISTENER) AMQ8021: Request to start WebSphere MQ Listener accepted.
There are two ways to stop the MI queue managers you've started. The first
is to switch the active and standby instances by using the
endmqm command with the
-s option. If you issue
endmqm -s IBMESBQM command on the active instance, you
will manually switch control to the standby instance. The
endmqm -s command shuts down the active instance without
shutting down the standby. The exclusive access lock on the queue manager
data and logs is released, and the standby queue manager takes over.
The second is to stop both the active and the standby instances of the
queue manager. If you issue the command
endmqm IBMESBQM on
the active instance of the queue manager -- without, as you will note, the
-s option -- you will stop both the active and the standby
If you are running multiple instances of the queue manager and decide that
you don't need to do so, then you can choose to stop only the standby
instance by issuing the
endmqm command with the
-x option. The standby instance will no longer serve as
standby, but the active instance will keep running.
Deleting a multi-instance queue manager
To delete multiple instances of a queue manager, you will first need to
delete its references from other standby queue manager servers. Run the
rmvmqinf command on all servers that have a definition of the
queue manager, as shown in Listing 14:
Listing 14. Deleting references to a queue manager
Next, you must delete the queue manager on all active queue manager
servers. Run the
dltmqm command, as shown in Listing 15, to
delete a queue manager that has instances defined on other servers.
Listing 15. Deleting references to a queue manager
You need not run this command on the same server where you created the
queue manager; you can run this command on any server where that queue
manager is defined. You could also delete MI queue managers just as well
by executing the
dltmqm IBMESBQM command on both servers.
Multi-instance queue manager runtime file system organization
Figure 2 shows a hierarchical representation of the MI queue manager /mqha file system. Queue manger-specific objects and active logs are stored in this location:
Figure 2. Hierarchical representation of MI queue manager file system
Managing a multi-instance queue manager
WebSphere MQ Explorer cannot run locally to more than one instance of an MI queue manager. You need to configure WebSphere MQ Explorer to manage MI queue managers remotely. Select Queue Managers => Add Remote Queue Manager menu item to add connections to an MI queue manager, as shown in Figure 3:
Figure 3. WebSphere MQ Explorer Navigator
The Add Queue Manager wizard will appear. Provide the queue manager name and select the appropriate connection method, as shown in Figure 4:
Figure 4. Add Queue Manager wizard
Click Next once you've made your selections.
Next, you will need to provide connection details for the active and standby queue managers, and instruct WebSphere MQ Explorer to auto-reconnect to the active queue manager in case of a failover. Figure 5 demonstrates how to enter this information properly:
Figure 5. Setting up an MI queue manager on the Add Queue Manager wizard
If no further client configuration is required, click Finish. WebSphere MQ Explorer will now connect the active queue manager instance, as illustrated in Figure 6. Remember, in this sample configuration, the active queue manager resides on the server wmbmi1.in.ibm.com:
Figure 6. Connected active queue manager in MQ Explorer Navigator
The next logical step is to verify the auto-reconnect functionality by performing a controlled failover of the IBMESBQM queue manager.
Failover between multi-instance queue managers
You should perform a failover test of the MI queue manager IBMESBQM in a controlled environment. (Later in this article, you'll see a more rigorous test for the results of a network or power failure when an MI broker has been implemented as well.) Controlled failover can be achieved by stopping the active queue manager and seeing if the standby queue manager acquires a lock on the /mqha file system and becomes the active queue manager.
Before you issue the command to stop the active instance of the MI queue
manager, it is important to identify the server on which the active
instance is running. You can do so using the
-x flag. In Listing 16, you can see that, in our
sample setup, the queue manager IBMESBQM is running as the active queue
manager on wmbmi1.in.ibm.com and as standby on wmbmi2.in.ibm.com:
Listing 16. Identifying the location of the active instance
[email@example.com$ dspmq -x QMNAME(IBMESBQM) STATUS(Running) INSTANCE(wmbmi1.in.ibm.com) MODE(Active) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby) [firstname.lastname@example.org]$ dspmq -x QMNAME(IBMESBQM) STATUS(Running as standby) INSTANCE(wmbmi1.in.ibm.com) MODE(Active) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby)
Now that you know where it's running, you can stop the active queue manager
on wmbmi1.in.ibm.com with the
endmqm -s IBMESBQM command, as
shown in Listing 17:
Listing 17. Stopping the active queue manager
[email@example.com]$ endmqm -s IBMESBQM Quiesce request accepted. The queue manager will stop when all outstanding work is complete, permitting switchover to a standby instance. [firstname.lastname@example.org]$ dspmq -x QMNAME(IBMESBQM) STATUS(Running elsewhere) INSTANCE(wmbmi2.in.ibm.com) MODE(Active) [email@example.com]$ dspmq -x QMNAME(IBMESBQM) STATUS(Running) INSTANCE(wmbmi2.in.ibm.com) MODE(Active)
In Listing 17 above, you can see that the queue manager IBMESBQM stopped on wmbmi1.in.ibm.com and is now running as an active queue manager on wmbmi2.in.ibm.com. Thus, the test shows that IBMESBQM failed over from the first server to the second. You can also determine this fact from WebSphere MQ Explorer, which auto-reconnected to the active instance of the queue manager now running on wmbmi2.in.ibm.com, as shown in Figure 7:
Figure 7. Auto-reconnect after failover
Now there is only one instance of the queue manager running, and there
would not be any failover if problems arise. Hence, to restore the standby
queue manager, you should execute the
strmqm -x IBMESBQM
command on wmbmi1.in.ibm.com. This will ensure that the queue manager
IBMESBQM runs actively on wmbmi2.in.ibm.com and passively on
wmbmi1.in.ibm.com. To test that the failover occurs properly from
wmbmi2.in.ibm.com to wmbmi1.in.ibm.com, and to restore the configuration
to its original arrangement, just repeat the steps in this section.
The MI broker feature of Message Broker works with WebSphere MQ in one of two ways. Each broker instance is embedded into a WebSphere MQ service so that, when the queue manager switches over to the standby system, the broker is automatically started on the standby node. This feature is available in Message Broker V126.96.36.199 or later.
You can configure MI brokers with a
-d option on the
commands, where the value of the option is
defined if you
want to create the WebSphere MQ service or
undefined if you
want to remove it. When you first create the broker instance with this
option, the queue manager has already been started and then the WebSphere
MQ service is created, so the broker will not be started by the service
until the queue manager has been stopped and restarted. This is true only
when you are creating the broker. Thereafter, the WebSphere MQ service has
Alternatively, the standby broker can run continuously in a semi-initialized state, waiting for the associated standby queue manager and shared broker configuration to become available.
In this section, you will create the MI broker IBMESBBRK on the sample system using the MI queue manager IBMESBQM that you set up previously. To do this, you'll need to use the same user (mqm), and will make this user a member of the mqbrkrs group. Ensure that the gid of mqbrkrs remains same across all the servers. Here are the tasks to administer the MI broker:
Creating a multi-instance broker
There are three sets of actions that must be performed to configure Message Broker in multi-instance mode:
- Create a shared work path directory on an NFS server
- Create a WebSphere MQ MI queue manager
- Create an MI broker
In the previous section, you configured a shared work path and set up an MI
queue manager, so you can move on to third step. First, you need to verify
the server on which the active queue manage is running; this can be
achieved using the
dspmq -x command. In our setup, the active
queue manager is on wmbmi1.in.ibm.com, so you should create the active
broker instance on that server. The command used to create a broker is
mqsicreatebroker, and its implementation and output for the
sample system are shown in Listing 18:
Listing 18. Creating a broker
[firstname.lastname@example.org]$ mqsicreatebroker IBMESBBRK -q IBMESBQM -e /mqha/WMB/IBMESBBRK AMQ8110: WebSphere MQ queue manager already exists. The setmqaut command completed successfully. The setmqaut command completed successfully. BIP8071I: Successful command completion.
Once the active instance of this broker is created, create the standby
instance on wmbmi2.in.ibm.com using the
Listing 19. Creating the standby instance of the broker
mqsiaddbrokerinstance IBMESBBRK -e /mqha/WMB/IBMESBBRK
mqsiaddbrokerinstance command is used to create an MI
broker on a UNIX/Linux or Windows server where Message Broker has been
mqsiaddbrokerinstance command is also used to
add a broker instance to any additional server on which you require
multi-instance support. The command expects that the user has first
created a multi-instance-enabled broker on an initial server using the
mqsicreatebroker command. The syntax for this command varies
by operating system:
mqsiaddbrokerinstance <brokerName> -e <sharedWorkPath> -w <Workpath> -i <serviceUserid> -a <servicePassword>
mqsiaddbrokerinstance <brokerName> -e <sharedWorkPath> -w <Workpath>
-w option is optional in both cases. It indicates
the directory in which working files specific to this broker instance
should be stored locally on the server where the broker instance is going
to run. On Linux and UNIX systems, the uid used to run this command must
be a member of both the mqbrkrs and the mqm groups. Additionally, you need
to make the uid and gid of mqbrkrs the same on all systems. Listing 20
shows the output of the
mqsiaddbrokerinstance command on
Listing 20. Output of mqsiaddbrokerinstance
[email@example.com]$ mqsiaddbrokerinstance IBMESBBRK -e /mqha/WMB/IBMESBBRK AMQ7272: WebSphere MQ configuration information already exists. BIP8071I: Successful command completion.
Starting and stopping a multi-instance broker
Now that duplicate instances of the broker IBMESBBRK have been created on
both servers, the next step is to start those instances. One will run as
the active instance and the other as the standby instance, based on the
state of the instances of the corresponding queue manager IBMESBQM. Start
the brokers by issuing the
mqsistart command on both servers,
as shown in Listing 21:
Listing 21. Starting the broker instances
[firstname.lastname@example.org]$ mqsistart IBMESBBRK BIP8096I: Successful command initiation. Check the system log to ensure that the component started without problems and that it continues to run without problems. [email@example.com]$ mqsistart IBMESBBRK BIP8236I: A standby instance of Broker IBMESBBRK has been started against the standby queue manager IBMESBQM. The active broker instance and active queue manager instance are running elsewhere. The multi-instance broker has started in standby mode. The broker will not become active until the current active broker instance and active queue manager instance end, or are stopped. No action required.
The broker IBMESBBRK starts as the active broker on wmbmi1.in.ibm.com, and
starts in standby mode on wmbmi2.in.ibm.com. Broker processes running as
active and standby instances are described below, but first, you need to
create an execution group on the active broker on wmbmi1.in.ibm.com by
mqsicreateexecutiongroup IBMESBBRK -e IBMESBEG
Listing 22. Creating an execution group
[firstname.lastname@example.org]$ mqsicreateexecutiongroup IBMESBBRK -e IBMESBEG BIP1124I: Creating execution group 'IBMESBEG' on broker 'IBMESBBRK'... BIP1117I: The execution group was created successfully. The broker has initialized the execution group. BIP8071I: Successful command completion. [email@example.com]$ ps -eaf | grep –I IBMESBEG mqm 364 32622 17 20:00 ? 00:00:07 DataFlowEngine IBMESBBRK 9a38eebe-2701-0000-0080-ad888bea8d47 IBMESBEG
Once the execution group has been created, it starts automatically. You can
verify this by using the
ps command and doing a
grep for the string
IBMESBEG (which is the
execution group name). Now that the broker and
are up on the active instance, you can examine the processes running on
the active and standby instances. On the active node, you will find the
following broker processes running:
DataFlowEngineprocess for each execution group created on each broker.
On the standby node, you will find following broker processes running:
bipservice(in standby mode)
bipbroker(in standby mode)
You can see the details for the sample systems in Listing 23:
Listing 23. Processes running on active and standby nodes
[firstname.lastname@example.org]$ ps -eaf | grep IBMESBBRK mqm 364 32622 1 20:00 ? 00:00:07 DataFlowEngine IBMESBBRK 9a38eebe-2701-0000-0080-ad888bea8d47 IBMESBEG mqm 32617 1 0 19:56 ? 00:00:00 bipservice IBMESBBRK mqm 32622 32617 0 19:56 ? 00:00:02 bipbroker IBMESBBRK mqm 32665 32622 0 19:56 ? 00:00:01 biphttplistener IBMESBBRK [email@example.com]$ ps -eaf | grep IBMESBBRK mqm 31916 1 0 20:04 ? 00:00:00 bipservice IBMESBBRK mqm 31921 31916 0 20:04 ? 00:00:00 bipbroker IBMESBBRK
It is important to know how to stop an MI broker. Using the
mqsistop command with the broker name will end an MI broker
or a broker instance. No switchover will happen upon execution of this
command; it will just bring down the running broker. The command must be
executed on both servers separately to bring down the active and standby
instances of the broker.
Deleting a multi-instance broker
Though you won't delete the MI brokers on your simple system at this point, it is important to know how to delete these MI components. The order of deletion of an MI broker and its associated instances is important, and you need to use the correct command for each step in the process. Here is an overview or how to delete MI brokers, using the sample setup:
- Stop the MI broker IBMESBBRK on the active node.
- Stop the MI broker IBMESBBRK on the standby node.
- Remove the broker instance IBMESBBRK on the standby node using the
- Remove the broker IBMESBBRK on the active node using the
Multi-instance broker runtime file system organization
Figure 8 shows a hierarchical representation of the MI broker's /mqha file system. The broker's components and registry-specific data are stored in this location:
Figure 8. Hierarchical representation of an MI broker file system
Backing up and restoring a multi-instance broker
No special techniques are needed to back up and restore an MI broker. You would perform these tasks in a standard way, using the commands in Listing 24 on the active instance:
Listing 24. Backing up and restoring a broker
mqsibackupbroker IBMESBBRK -d /backup/WMB mqsirestorebroker IBMESBBRK -d /backup/WMB -a <archive file name>
mqisbackupbroker command detects an MI broker and backs up
the registry and configuration from the shared work path of the broker.
mqsirestorebroker restores the registry and
configuration to the shared work path of the broker. Execute the backup
command with the active instance of the broker. While using the restore
command, stop all brokers with which the broker you're restoring shares
archive data (both active and standby instances). If you run this command
when a relevant broker is active the results may be unpredictable.
Failover switching between multi-instance brokers
Before starting with this section, you should have clear understanding what exactly failover means in the context of MI brokers. When a queue manager goes down for any reason, this will cause the associated broker to fail; Message Broker will then switch to the standby instance. At that time, the previously active instance of the broker sets itself to standby mode, and the standby instance becomes active. The MI broker relies on the status of the MI queue manager.
Consider two failover scenarios: a controlled switchover and an uncontrolled switchover due to a power or network failure.
Controlled switchover of a multi-instance broker
Begin the controlled switchover test by displaying the state of the MI broker on wmbmi1.in.ibm.com, to confirm that it's active. You can do this by issuing the command in Listing 25, and you should see output similar to what's shown in the listing:
Listing 25. Displaying the state of IBMESBBRK on wmbmi1.in.ibm.com
[mqm@wmbmi1 .in.ibm.com]$ mqsilist BIP1295I: Broker 'IBMESBBRK' is a multi-instance broker running in active mode on multi-instance queue manager 'IBMESBQM'. BIP8071I: Successful command completion
Next, do the same for the queue manager, as shown in Listing 26:
Listing 26. Displaying the state of IBMESBQM on wmbmi1.in.ibm.com
[mqm@wmbmi1 ~]$ dspmq –m IBMESBQM –x QMNAME(IBMESBQM) STATUS(Running) INSTANCE(wmbmi1.in.ibm.com) MODE(Active) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby)
Now switch to the standby server, wmbmi2.in.ibm.com, and execute the same commands. You should see the results shown in Listings 27 and 28, indicating that the queue manager and broker instances there are the standby instances.
Listing 27. Displaying the state of IBMESBBRK on wmbmi2.in.ibm.com
[mqm@wmbmi2 .in.ibm.com]$ mqsilist BIP1294I: Broker 'IBMESBBRK' is a multi-instance broker running in standby mode on multi-instance queue manager 'IBMESBQM'. More information will be available when the broker instance is active. BIP8071I: Successful command completion.
Listing 28. Displaying the state of IBMESBQM on wmbmi2.in.ibm.com
[firstname.lastname@example.org]$ dspmq –m IBMESBQM –x QMNAME(IBMESBQM) STATUS(Running as standby) INSTANCE(wmbmi2.in.ibm.com) MODE(Standby) INSTANCE(wmbmi1.in.ibm.com) MODE(Active)
Next, back on wmbmi1.in.ibm.com, stop the active instance of IBMESBQM, as shown in Listing 29:
Listing 29. Stopping the active instance of IBMESBQM
[mqm@wmbmi1 ~]$ endmqm -s IBMESBQM Quiesce request accepted. The queue manager will stop when all outstanding work is complete, permitting switchover to a standby instance.
Now display its state, as shown in Listing 30. The output should indicate that what had been the standby instance has become the active instance:
Listing 30. Confirming that IBMESBQM has stopped on wmbmi1.in.ibm.com
[email@example.com]$ dspmq -m IBMESBQM –x QMNAME(IBMESBQM) STATUS(Running elsewhere) INSTANCE(wmbmi2.in.ibm.com) MODE(Active)
Use the command in Listing 31 to list the state of the broker IBMESBBRK. You will see that it too has stopped running on wmbmi1.in.ibm.com:
Listing 31. Confirming that IBMESBBRK has stopped on wmbmi1.in.ibm.com
[firstname.lastname@example.org]$ mqsilist BIP1292I: The multi-instance Broker 'IBMESBBRK' on multi-instance queue manager 'IBMESBQM' has stopped. BIP8071I: Successful command completion.
Now switch back to wmbmi2.in.ibm.com again. Issue the command in Listing 32 to confirm that the instance of IBMESBQM there has become active:
Listing 32. Confirming that IBMESBQM has become active on wmbmi2.in.ibm.com
[email@example.com]$ dspmq -m IBMESBQM –x QMNAME(IBMESBQM) STATUS(Running) INSTANCE(wmbmi2.in.ibm.com) MODE(Active)
Finally, use the command in Listing 33 to confirm that IBMESBBRK on wmbmi2.in.ibm.com is active as well:
Listing 33. Confirming that IBMESBBRK is active on wmbmi2.in.ibm.com
[mqm@wmbmi2 ~]$ mqsilist BIP1295I: Broker 'IBMESBBRK' is a multi-instance broker running in active mode on multi-instance queue manager 'IBMESBQM'. BIP8071I: Successful command completion.
With IBMESBQM and IBMESBBRK active on wmbmi2.in.ibm.com, the switchover from one server to the other is now complete.
Uncontrolled failover due to power or network failure
In the previous section, you saw how processing could fail over from one server to another in a controlled environment. But what happens when one server's processing is abruptly terminated, or when it is unexpectedly disconnected from the network? Message Broker's high availability features can handle such a scenario. To perform this failover test, create a real-time scenario with message flow processing in the execution group IBMESBEG on the broker IBMESBBRK. Create and deploy a very simple message flow (shown in Figure 9) having following nodes:
- MQInput Node (IBMESB_IN)
- Compute Node (Copy Message)
- MQOutput Node (IBMESB_OUT)
Figure 9. Message flow
The input and output queues (IBMESBQM_IN and IBMESBQM_OUT) have their persistence property set to persistent, which means that all messages arriving on these queues are made persistent. This will avoid any message loss during failover.
In this section, you will see that the client connection also survives through the failover. This client would put messages on the input queue IBMESBQM_IN of the queue manager IBMESBQM.
amqsphac is a ready-to-use sample program that comes with
WebSphere MQ; you can use it to check out the connection status of the
message flow with the queue manager. It keeps putting messages on the
input queue and informs you about the status of the put. You can use this
sample program to check the connection status as well. Listing 34 shows
the results when the program is invoked well before wmbmi1.in.ibm.com goes
down due to a power or network failure. The program keeps running until
wmbmi2.in.ibm.com takes over from the failed server:
Listing 34. amqsphac output illustrating failover in action
[firstname.lastname@example.org]$ amqsphac IBMESBQM_IN IBMESBQM Sample AMQSPHAC start target queue is IBMESBQM_IN message <Message 1> // Client application is well connected here // and successfully putting message to Input Queue message <Message 2> message <Message 3> message <Message 4> message <Message 5> message <Message 6> message <Message 7> message <Message 8> message <Message 9> message <Message 10> message <Message 11> message <Message 12> message <Message 13> message <Message 14> message <Message 15> // wmbmi1 server has gone down here and the connection of client is lost 20:41:53 : EVENT : Connection Reconnecting (Delay: 168ms) 20:41:54 : EVENT : Connection Reconnecting (Delay: 1220ms) 20:41:55 : EVENT : Connection Reconnecting (Delay: 2324ms) 20:41:57 : EVENT : Connection Reconnecting (Delay: 4494ms) 20:42:02 : EVENT : Connection Reconnecting (Delay: 8731ms) 20:42:10 : EVENT : Connection Reconnecting (Delay: 19732ms) // wmbmi2 server has gained the control here and has become active. 20:42:31 : EVENT : Connection Reconnected // Client Connection to Queue Manager is re-established message <Message 16> // Client application is successfully putting the messages again message <Message 17> message <Message 18> message <Message 19> message <Message 20>
We got the results in Listing 34 from our own network failure test when we
disconnected a network cable from the wmbmi1.in.ibm.com server. The
standby queue manager IBMESBQM on wmbmi2.in.ibm.com became the active
queue manager, and IBMESBBRK became the active broker, in less than two
minutes. The client was then automatically reconnected to the active queue
amqsphac client produced 20 messages, all of
which were processed and put in the IBMESB_OUT queue by the copy message
flow. Here is the output:
Listing 35. Output demonstrating successful failover
[email@example.com]$ dspmq –m IBMESBQM -x QMNAME(IBMESBQM) STATUS(Running) INSTANCE(wmbmi2.in.ibm.com) MODE(Active) [firstname.lastname@example.org]$ echo "DISPLAY QLOCAL(IBMESBQM_OUT) CURDEPTH" | runmqsc IBMESBQM 5724-H72 (C) Copyright IBM Corp. 1994, 2009. ALL RIGHTS RESERVED. Starting MQSC for queue manager IBMESBQM. 1 : DISPLAY QLOCAL(IBMESBQM_OUT) CURDEPTH AMQ8409: Display Queue details. QUEUE(IBMESBQM_OUT) TYPE(QLOCAL) CURDEPTH(20) One MQSC command read. No commands have a syntax error. All valid MQSC commands were processed.
A limitation with Message Broker Explorer
There is currently a limitation when using WebSphere Message Broker Explorer to examine the standby instance of an MI broker. Although you can list and view all instances of an MI queue manager on WebSphere MQ Explorer, and switch the Explorer connection to the standby queue manager in case of a failover, you can't list or view the standby instance of an MI broker on the WebSphere Message Broker Explorer in a failover situation.
This article outlined the concepts and basic architecture of MI queue mangers and MI brokers, and explained and how to create, configure, manage, and test these MI components for high availability. The systems described here have included one broker or queue manager instance that is active and one standby. This represents an active-passive architecture for designing highly available messaging solutions. There is another type of HA architecture called an active-active scenario, in which you use multiple sets of MI queue managers and brokers overlayed on a WebSphere MQ cluster. Part 2 of this article will describe the other HA scenario in detail.
- WebSphere MQ resources
- Testing and support statement for WebSphere MQ multi-instance
This IBM Technote describes support for WebSphere MQ multi-instance queue managers on various operating systems, including IBM i5/OS and Solaris.
- Backing up a multi-instance queue manager
This IBM Technote provides detailed steps for backing up a multi-instance queue manager.
- Network File System
(NFS) V4 Protocol
This document from the Network Working Group specifies an Internet standards track protocol for the Internet community.
- WebSphere MQ V7.5 information center
A single Web portal to all WebSphere MQ documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere MQ environment.
- Testing and support statement for WebSphere MQ multi-instance queue managers,
- WebSphere Message Broker
- WebSphere Message Broker developer resources
Technical resources to help you use WebSphere Message Broker for connectivity, universal data transformation, and enterprise-level integration of disparate services, applications, and platforms to power your SOA.
- WebSphere Message Broker product page
Product descriptions, product news, training information, support information, and more.
- WebSphere Message Broker V8 information center
A single Web portal to all WebSphere Message Broker V8 documentation, with conceptual, task, and reference information on installing, configuring, and using your WebSphere Message Broker environment.
- What's new in WebSphere Message Broker V8
WebSphere Message Broker V8 provides universal connectivity with its ability to route and transform messages from anywhere to anywhere. Through its simple programming model and a powerful operational management interface, it makes complex application integration solutions much easier to develop, deploy, and maintain. This article describes the major enhancements in V8.
- Download free trial version of WebSphere Message Broker
WebSphere Message Broker V8 is an ESB built for universal connectivity and transformation in heterogeneous IT environments. It distributes information and data generated by business events in real time to people, applications, and devices throughout your extended enterprise and beyond.
- WebSphere Message Broker documentation
WebSphere Message Broker specifications and manuals.
- WebSphere Message Broker forum
Get answers to your technical questions and share your expertise with other WebSphere Message Broker users.
- WebSphere Message Broker support page
A searchable database of support problems and their solutions, plus downloads, fixes, problem tracking, and more.
- WebSphere Message Broker developer resources page
- WebSphere resources
- developerWorks WebSphere developer
Technical information and resources for developers who use WebSphere products. developerWorks WebSphere provides product downloads, how-to information, support resources, and a free technical library of more than 2000 technical articles, tutorials, best practices, IBM Redbooks, and online product manuals.
- developerWorks WebSphere application integration developer
How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application integration and business integration solutions.
- Most popular WebSphere trial downloads
No-charge trial downloads for key WebSphere products.
- WebSphere forums
Product-specific forums where you can get answers to your technical questions and share your expertise with other WebSphere users.
- WebSphere on-demand demos
Download and watch these self-running demos, and learn how WebSphere products and technologies can help your company respond to the rapidly changing and increasingly complex business environment.
- WebSphere-related articles on developerWorks
Over 3000 edited and categorized articles on WebSphere and related technologies by top practitioners and consultants inside and outside IBM. Search for what you need.
- developerWorks WebSphere weekly newsletter
The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
- WebSphere-related books from IBM Press
Convenient online ordering through Barnes & Noble.
- WebSphere-related events
Conferences, trade shows, Webcasts, and other events around the world of interest to WebSphere developers.
- developerWorks WebSphere developer resources
- developerWorks resources
downloads for IBM software products
No-charge trial downloads for selected IBM® DB2®, Lotus®, Rational®, Tivoli®, and WebSphere® products.
business process management developer resources
BPM how-to articles, downloads, tutorials, education, product info, and other resources to help you model, assemble, deploy, and manage business processes.
Join a conversation with developerWorks users and authors, and IBM editors and developers.
- developerWorks tech briefings
Free technical sessions by IBM experts to accelerate your learning curve and help you succeed in your most challenging software projects. Sessions range from one-hour virtual briefings to half-day and full-day live sessions in cities worldwide.
- developerWorks podcasts
Listen to interesting and offbeat interviews and discussions with software innovators.
- developerWorks on
Check out recent Twitter messages and URLs.
- IBM Education Assistant
A collection of multimedia educational modules that will help you better understand IBM software products and use them more effectively to meet your business requirements.
- Trial downloads for IBM software products