Level: Advanced David Peraza (dperaza@us.ibm.com), Software Engineer, IBM
06 Jul 2007 Learn how the WebSphere® message queuing (WMQ) features together with the IBM i5/OS clustering capabilities
can provide a highly available environment for WMQ persistent messages. See
how WMQ cluster and i5/OS cluster capabilities can be combined to provide a more
complete high availability solution.
Introduction
The WebSphere message queuing (WMQ) feature, when used with IBM
i5/OS® clustering capabilities, can provide a highly available environment for WMQ
persistent messages. This article describes how to use those technologies together in
an optimal, highly available (HA) environment. First you will read about the tangible risks
for WMQ solutions that do not have a WMQ HA strategy. Then you will learn the
cluster capabilities of WMQ, comparing and contrasting them with i5/OS cluster
capabilities. You can explore i5/OS cluster configurations, which have been
tested and proven effective for maintaining availability of the queue managers and
their persistent messages. Next you will see how a more complete HA solution can be
delivered with a combination of WMQ cluster and i5/OS cluster capabilities. Finally,
you can examine the results of performance measurements of the solution.
Understanding the risks of not having a WMQ HA strategy
As the value of information grows, so does the need to protect it and make it
available at all times. In the past, if a hardware system failed or needed to be
replaced, it might have been enough just to take the system off line, restart it,
and continue with business despite the small gap in service. In today's business
climate, information has to be available for consumption practically 100 percent of the
time. Consider the following example scenario.
An emergency room scenario
Consider a distributed system handling medical records. Assume there are two
hospitals in town and that each has its own specialization. Both hospitals can
handle general surgery, but hospital A specializes in neurosurgery while hospital B
specializes in cardiovascular surgery. The two hospitals have an agreement. If
hospital A admits a patient with a heart problem, the patient is transferred to
hospital B for surgery after being stabilized. Similarly, if a patient is admitted
to hospital B with a head injury, he is transferred to hospital A to receive
surgery from a specialist.
Obviously, the hospital medical record systems need to interact with each
other in order to be able to share data. For this purpose, a technical company has
been hired to host the data of the two hospitals and make information sharing
easier. The company uses WebSphere message queuing (WMQ) as the messaging provider
with one queue manager sitting at the data center. (Note: See the
Glossary if you need descriptions of
acronyms and terms used in this article.) Whenever patients are transferred, their
medical records (XML messages) are sent to the remote queue manager using a WMQ
client running at the source hospital. The XML messages are then retrieved by a
WMQ client running at the destination hospital, as shown in Figure 1.
Figure 1. Integration of two
medical record systems through WMQ
An unfortunate driver has an accident and needs to be taken urgently to a
hospital. The paramedics take the patient to hospital B, because this is the nearest
one. Upon arrival, the doctors determine that he has a head injury and needs
surgery as soon as possible. Because hospital A specializes in neurosurgery, the
patient needs to be transferred there.
Imagine what would happen if the queue manager at the data center went down due
to a hardware problem. It would certainly be very difficult to retrieve the
patient record from the data center in a timely manner. It is still possible if
the technical team acts promptly and retrieves the records manually by querying
the database. However, remember that a patient needs attention quickly, and
hospital A needs the information as soon as possible. That patient's life depends
on it. In this case, it can be said that the queue manager at the data center is a
single point of failure and that the data center is not highly available.
A natural disaster scenario
The previous scenario considers failure only to the system running the queue
manager. However, consider a less probable but more devastating scenario. Assume
the data center gets consumed by fire. It would be very hard to retrieve the
information for the patient that is being transferred. Now, neither hospital can
store or retrieve any medical records. Even if the data was backed up on tapes,
there is no way it can be restored because there is no physical system to restore
to. In this case, the data center is considered not disaster tolerant. In
other words, it cannot recover from a disaster.
Decreasing the risks by implementing a
WMQ HA solution
The data center in the scenarios needs to be rearchitected. It is the very center
of the distributed medical record system, but it is not highly available and
it is not disaster tolerant. This article offers possible solutions to the data
center's problems.
A solution for high availability
Start by addressing the first issue: that the system is not highly
available. Providing redundancy is one of many possible solutions, as shown in
Figure 2.
Figure 2. Redundancy of system
running queue manager
Figure 2 shows a new component, system B, which is an idle backup of system A. If
the queue manager fails, its replica on system B can take over the workload, and
the hospitals can continue to receive data when they need it.
A solution for disaster recovery
The solution shown in Figure 2 can handle a local failover of a queue manager.
However, it does not address the natural disaster scenario. If fire consumes the
data center, it does not matter how much local redundancy has been implemented,
the medical records become inaccessible, crippling the operation of the hospitals. Figure 3
illustrates a disaster-tolerant solution, although it is a more expensive one.
Figure 3. Redundancy of data
center
Figure 3 shows replication of the entire data center. With this configuration, if
data center A is destroyed by fire or any other natural disaster, data center B
can take over the workload. It is best that the data centers be geographically
separate to protect against natural disasters that
can affect broad areas, such as hurricanes and earthquakes. However, with greater distance comes higher network
latency that could introduce delay, especially if the data center is being
replicated synchronously. A compromise between higher protection of the
data center and lower network latency is usually necessary to maintain both within
a tolerable range.
Exploring a WMQ cluster: A partial HA solution
Having seen the importance of implementing a WMQ HA solution, here are the
different ways this could be attained with WMQ on the System i™ platform,
specifically running i5/OS. This section describes a feature of WMQ that can
be exploited to provide a certain level of HA. Although WMQ by itself does not
provide a complete HA solution, a WMQ cluster configuration can be used to
distribute the workload better and, at the same time, introduce a level of
redundancy.
Introduction to a WMQ cluster
A WMQ cluster is a collection of queue managers that are logically associated in
some way. The main objective is to allow messages to travel from queue manager to
queue manager with little or no effort from the WMQ administrator. The only
objects that need to be configured are the cluster sender and receiver channels.
After a queue manager has been set up, the local queues of a queue manager could be
shared and made available to every queue manager in the cluster. For instructions
on how to create a WMQ cluster and share its queues, see
Appendix G.
Queue managers in a cluster can be partial or full repositories. A full
repository stores information about every other queue manager in the cluster. This
information is updated dynamically without ending the queue managers. On the other
hand, a partial repository only knows about full repositories in the cluster and
gets all the information needed to connect to the rest of the queue managers by
querying a full repository.
From an application perspective, it does not make a difference whether a queue
manager is a full or partial repository. For the application, all queue managers
are simply nodes of the cluster. It does make a difference, however, for the administrator of the cluster who is
in charge of making the choice of which queue managers are full or partial
repositories. Even though a WMQ cluster can function with only one full
repository, it is not recommended to use only one full repository. Using two full
repositories avoids the problem of having a single point of failure. The WMQ
cluster can continue to function even if a full repository fails. However, the
cluster cannot be modified when the repository goes down. For example, a new
queue could not be shared, nor could a new queue manager be added to the cluster.
By using a second full repository, cluster information is constantly available to
all queue managers, even when one full repository queue manager fails.
Workload balancing
A WMQ cluster can be configured to distribute application workloads across
various servers. The same queue could be defined in one or more queue managers
that form a cluster and then shared, as shown in Figure 4.
Figure 4. WMQ cluster between a
hospital and the data center
The messaging of the distributed system handling medical records is now performed
through a WMQ cluster. A hospital can send a request message to the patient queue
through the local queue manager (MGR H in Figure 4). Because both queue managers at the
data center include the patient queue, the center uses a WMQ workload balancing
algorithm to choose the target queue manager. Then, an application running at the
data center processes the request message and sends a reply message back to the
reply queue on the hospital's queue manager.
To learn more about the WMQ workload balancing algorithm, refer to the WMQ
information center (see Resources).
Partial HA solution
Figure 4 not only shows the medical record workload being distributed across two
systems at the data center, it also shows a more highly available data center.
Imagine that MGR A fails due to a hardware failure on system A. MGR B can
take over any new workload, allowing uninterrupted message flow between a hospital
and the data center to be. The only drawback to this configuration is that any
unhandled message on MGR A at the time of failure becomes marooned and can only be
retrieved once the queue manager is brought back online. This restriction is why
a WMQ cluster is considered to be a partial WMQ HA solution.
Pros and cons
This solution has several advantages and disadvantages for you to evaluate
before implementing. The advantages are as follows:
- The recovery time is practically zero. New requests can be handled
immediately.
- No system stays idle without workload processing.
- It is easy to configure.
- There is no overhead created by the replication algorithm.
The negatives are as follows:
- Each hospital system must contain a queue manager.
- Messages that are not processed before the queue manager fails will stay on
the patient queue. These messages will be marooned until the queue manager is
brought back online.
- There is some overhead created by queue-manager-to-queue-manager
communication.
Discovering how i5/OS clustering
complements WMQ clustering
WMQ clustering by itself does not provide a complete HA solution. However,
running WMQ on i5/OS and exploiting i5/OS clustering capabilities can provide a
more comprehensive HA solution. This section describes the i5/OS cluster feature
and shows how WMQ can be configured to use it.
i5/OS cluster
An i5/OS cluster is a collection of i5/OS instances (System i computers or
partitions) that are logically linked together. The purpose of this grouping is to
allow for each i5/OS instance to be backed up, eliminating a single point of
failure and increasing application and data resiliency. With a cluster created,
the various cluster resource group (CRG) types can be configured to manage
applications, data, and devices in the cluster. For instructions on how to create
an i5/OS cluster, see Appendix B.
Device cluster resource groups
There are several types of cluster resource groups (CRGs). For more information
about the different types of
CRGs available, refer to the cluster resource group section within the System i
information center (see Resources).
(Note: This article concentrates on device CRGs.)
A device CRG describes and manages device resources such as independent auxiliary
storage pools (IASPs). (Note: See the Glossary at
the end of this article for descriptions of acronyms and terms used in this
article.) The device CRG also defines the recovery domain of the cluster nodes,
assigns a device (in this case an IASP), and assigns the exit program that will
handle cluster events. The recovery domain simply denotes which cluster node will
be considered the primary node. The rest of the nodes are considered to be
backups. The backup nodes are also ordered in the recovery domain, specifying
which node is the first backup, the second backup, and so on, depending on how
many nodes there are in the recovery domain. In the event of a primary node
failure, the exit program is run on all nodes in the recovery domain. The exit
program running on the first backup can then make the necessary initializations to
make this node the new primary node. For more information on how to create a
device CRG, see Appendix C.
Independent auxiliary storage pools
(IASPs)
IASPs are an important element of the device CRG. An IASP is a type of user ASP
that serves as an extension of single-level storage. It is a piece of storage
that, due to its independence from the system storage, can be easily manipulated
without having to IPL the system.
Some of the most desirable qualities of IASPs are its ability to be easily switched
to another i5/OS instance and to be replicated to a target IASP on another i5/OS
instance. Two methods can be used to switch an IASP between i5/OS
instances. The first method, which is the one used in the example configurations,
requires all the System i computers in the cluster and the switchable disk tower
containing the IASP to be connected using a High Speed Link (HSL) loop. The second
method requires the i5/OS instances to be partitions on the same System i computer
where IOPs could be switched between partitions. No special hardware is needed to
be able to replicate an IASP. The replication is performed using TCP/IP over the
network. For instructions on how to create an IASP, see
Appendix A.
Device CRG exit programs
The i5/OS cluster resource service calls a device CRG exit program when an
event occurs in one of the nodes the recovery domain defines. Out of many
events that the device CRG exit program can handle, this article focuses on two
events:
failover and switchover. A failover event occurs when the primary node of the
cluster fails and the CRGs are switched with all the resources they manage. On the
other hand, a switchover event occurs when a specific CRG is manually switched
from the primary node to the backup node. Either way, the exit program is in
charge of initializing and starting all the programs that were running on the
previous primary node, which converts the first backup node into the new primary node.
For example, with WMQ, the exit program should be in charge of starting the WMQ
subsystem (QMQM), starting the queue manager, and starting jobs, such as
the WMQ listener and trigger monitor. Note that with WMQ Version 6.0 and above,
jobs such as WMQ listeners and trigger monitors can be automatically started when
the queue manager starts by setting up service objects. For an example of a device
CRG exit program initializing and starting a queue manager, refer to
Appendix D.
Switchable IASP configuration
Returning to the sample distributed system handling medical records, WMQ can be set
up to take advantage of the clustering capabilities of i5/OS. First, an i5/OS
cluster must be created between the data center systems. Then, the queue manager
should be moved to an IASP. And finally, a CRG should be created defining the
recovery domain, the IASP, and the exit program. Refer to the appendixes for additional instructions on
how to create i5/OS clusters, IASPs, and CRGs. See Figure 5 for a sample WMQ HA
configuration using switchable IASPs.
Figure 5. i5/OS cluster with a
switchable IASP
As you can see, the queue manager library and IFS files are stored in a
switchable IASP. For instructions on how to move a queue manager to an IASP, see
Appendix E. In the event of a primary node failure, the
device CRG switches the IASP to the backup node, and the device CRG exit
program runs to start the MGR A queue manager there. The hospital WMQ client
then resumes working after a brief interruption due to the failover process.
All messages that were not processed when the primary node failed will be
processed after the queue manager is brought back online on the backup node.
The advantages of this solution are as follows:
- The hospital system does not need the full blown WMQ product installed; it
only needs a thin WMQ client program communicating with the server side
managers.
- No data replication is needed. There is only one copy of the resilient data in
the IASP.
- There is no performance hit due to replication algorithms.
- Messages that were not processed before the primary node went down will be
processed once the queue manager is brought back online on the backup node.
The disadvantages are as follows:
- The hardware becomes a single point of failure. If the IASP fails, access to
the WMQ libraries and IFS file is lost.
- There is a distance limit for the HSL loop. If the system uses fiber optics,
the limit is 250 meters. If the system uses copper, 15 meters is the limit. Therefore,
disaster recovery is limited.
- The backup node cannot process messages while the primary node is active. It
sits idle until the primary node fails or a switchover is performed.
- There is an expected outage of several minutes when the IASP switchover or
failover occurs. In a real-world application environment, the exception path of
the applications can be modified to surface useful status information to the end
user during the outage time. Clients are not able to access the queue
manager during this time.
XSM with geographical mirroring
An alternative to the switchable IASP configuration is the mirrored IASP
configuration. The only difference between these two configurations is that with
switchable IASP, the data is made available to the backup node using hardware. With
mirrored IASP, the data is replicated or mirrored to an IASP attached to the backup
node. See Figure 6 for a sample WMQ HA configuration using mirrored IASPs.
Figure 6. i5/OS cluster with a
mirrored IASP
The advantages of geographic mirroring are as follows:
- There is no hardware single point of failure. Both the application components
and the data are replicated to a backup IASP.
- There is no theoretical limit on the distance between the cluster nodes.
Mirroring could be done over the Internet, keeping in mind that the performance
is inversely proportional to the network latency.
- The client box does not need the full blown WMQ product installed; it only
needs a thin WMQ client program communicating with the server side managers.
- Messages not processed before the primary node goes down are processed after
the queue manager is brought back online on the backup node.
The disadvantages are as follows:
- Performance overhead is created due to the replication algorithm. The more the
mirroring latency the worse the performance will be.
- The backup system cannot process messages. It will sit idle until the primary
node fails or a switchover is performed.
- There might be an expected outage of several minutes or more when the IASP
switchover or failover occurs. In a real-world application environment, the
exception path of the applications can be modified to surface useful status
information to the end user during the outage time. Clients will not be able to
use WMQ during this time.
Performance measurements
Because the geographical mirroring algorithm's performance depends on network
latency, it is a good idea to collect performance information to help quantify performance
degradation as network latency increases. The main objective of the performance
testing is to establish a relationship between the latency of the network
connecting the System i nodes and the responsiveness of WMQ managers.
The example used an open source Linux® program, nistnet (see Resources),
to simulate network latency: setting up a Linux system as a router and building
and starting a nistnet kernel module. The nistnet interface delayed the delivery of
packets between the two System i computers, thus simulating a specific network
latency.
Three parameters were measured while a workload generator client JMS program ran:
- Network latency (L)
- Message size
- Messages per minutes (mpm)
The workload generator ran for an hour with the initial network latency. Every
five minutes the message size parameter automatically increased starting at 1KB up
to 4 MB. After the first hour, the workload generator paused, waiting for a
new value of network latency. After the new latency value was entered, the workload
generator resumed for another hour, changing the message size and recording mpm as
described. The whole process continued until the metrics for six network
latency values were captured. By the end of the test, the workload generator had
run for six hours plus the time that it took to modify the network latency. See
Figure 7 and Table 1 for the results of this test.
Figure 7: Graphical representation
of performance measurements
Table 1. Results of performance measurements
| msg size | mpm NM | mpm M (L=1 ms) | mpm M (L=10 ms) | mpm M (L=20 ms) | mpm M (L=40 ms) | mpm M (L=80 ms) | mpm M (L=160 ms) |
|---|
| 1024 | 5600 | 4750 | 4125 | 3375 | 2275 | 1595 | 850 |
|---|
| 2048 | 5525 | 4700 | 4050 | 3250 | 2225 | 1475 | 775 |
|---|
| 4096 | 5175 | 4275 | 3675 | 2925 | 2025 | 1375 | 725 |
|---|
| 8192 | 4450 | 3625 | 3075 | 2500 | 1735 | 1100 | 600 |
|---|
| 16384 | 3825 | 2875 | 2425 | 2025 | 1400 | 911 | 475 |
|---|
| 32768 | 3000 | 2025 | 1681 | 1400 | 991 | 648 | 324 |
|---|
| 65536 | 1975 | 1303 | 1107 | 908 | 611 | 386 | 198 |
|---|
| 131072 | 1144 | 777 | 663 | 520 | 339 | 207 | 121 |
|---|
| 262144 | 614 | 420 | 369 | 287 | 186 | 103 | 62 |
|---|
| 524288 | 334 | 228 | 165 | 163 | 85 | 62 | 30 |
|---|
| 1048576 | 162 | 120 | 97 | 76 | 50 | 25 | 14 |
|---|
| 2097152 | 90 | 62 | 46 | 39 | 28 | 13 | 4 |
|---|
| 4194304 | 53 | 30 | 25 | 19 | 12 | 6 | 5 |
|---|
Table 1 shows how the rate of message transmission, measured in messages per
minute (mpm), varies as the message size (msg size) and network latency (L)
increases. The first column shows the message size values, while subsequent
columns show the mpm for different network latency values. The second column does
not show a network latency value because these values were collected while the
workload was run with no mirroring (NM).
As expected, the WMQ message flow rate decreases as the network latency increases.
For example, with a message size of 32 KB, the message flow rate decreases by a
factor of 10 while the network latency changes from 1 ms to 160 ms.
Your choice of network transports also affects overall performance. Be sure
you have adequate network bandwidth to handle the specific needs of your messaging
application, especially during peak volume intervals. Using multiple
transports for the geographic mirroring replication processing between systems is
recommended. In the example testing, Ethernet transports were used.
Also, keep in mind that the example data was obtained with the latest available versions
of both WMQ and i5/OS and is subject to change as these products evolve. Also,
results listed here do not represent any particular customer environment. Actual
performance can vary significantly from what is provided here. The goal is to
provide an idea of how the performance of WMQ is expected to degrade as the
latency changes while a queue manager is being replicated using geographical
mirroring.
Combining WMQ and i5/OS clustering:
A complete HA solution
You have seen how WMQ clusters provide rapid recovery time and load balancing
capabilities, while i5/OS clusters offer message integrity. How can
these two technologies be combined to obtain a complete WMQ HA solution? You can
read about this kind of solution in this section.
WMQ and i5/OS cluster configuration
This time in the example, the distributed system handling medical records was set up using a
combination of WMQ clusters and i5/OS clusters. A WMQ cluster was
established between the manager at the hospital and the two managers at the data
center. And, an i5/OS cluster between the two systems at the data center was
established, allowing the queue managers running on them to be backed up, as
shown in Figure 8.
Figure 8. WMQ cluster and i5/OS
cluster
Initially, node A is the primary node for MGR A, and node B is the primary node
for MGR B. Node B is a backup of node A, and at the same time, node A is a backup of
node B. This configuration makes use of the WMQ workload balance capabilities
while using i5/OS clustering to ensure constant availability of messages. No
node stays idle. All processors are used. Notice that calculations are necessary
so that each node can handle the complete load in the event of a system failure.
If, for example, system A goes down due to a hardware failure, system B can
restart MGR A, and the hospital queue manager can retrieve marooned messages.
Here are the advantages of this approach:
- The recovery time is practically zero. New requests can be handled
immediately.
- No system stays idle without workload processing. Caution must be taken to
guarantee that each system runs at half capacity in case one of them should need
to handle the entire load after a switchover or failover.
- There is no single point of failure. Both the application components and the
data are replicated, and two queue managers share the load.
- No messages are marooned. Messages that are not processed before the primary
node goes down are processed after the queue manager is brought back online on
the backup node.
Here are the disadvantages:
- There is system overhead due to the replication algorithm.
- The hospital system must contain a queue manager.
Examining WMQ outage scenarios
This section describes different scenarios that can occur with a WMQ HA
configuration.
IASP manual switchover
This scenario is very common during an upgrade of WMQ while a queue manager is
stored in a switchable or mirrored IASP managed by a device CRG. See
Figure 5 and Figure 6 for graphical
representations of
this scenario. Consider the case where WMQ is to be upgraded from version 5.3 to
version 6.0.2.0. The backup system can easily be upgraded, because the queue manager
is not actively running. However, the queue manager running in the primary node
must be stopped before the WMQ version can be updated. In this case, to avoid a
WMQ outage during the migration process, the device CRG is instructed to switch
the IASP to the backup node making this node the new primary node. Call command
CHGCRGPRI [CLUSTER NAME] [CRG NAME] to instruct the CRG to switch the IASP. After
the IASP is successfully moved, the old primary becomes the backup and the upgrade
can take place. The IASP can then be switched back if desired.
IASP failover
This scenario occurs if the primary node of an i5/OS cluster fails. See
Figure 5 or Figure 6 for a graphical representation of
this scenario. If the primary node fails, the device CRG switches the IASP to the
backup node, and the CRG exit program is called there. The exit program then
starts the queue manager, and all the messages become accessible again. The only
difference between this scenario and the IASP switchover is that the exit program
receives a different action code on its command line argument. However, regardless
of the action code being switchover or failover, the exit program performs the
same initialization.
i5/OS cluster partition
Sometimes the primary node in an i5/OS fails in such a way that it cannot notify
other nodes that it has failed. This causes uncertainty, because the backup node
cannot determine whether the problem is caused by primary node failure or a
network problem. In this situation, the i5/OS cluster is said to have been
partitioned. A cluster partition event does not trigger any automatic action. Some
manual intervention is needed either to declare the primary node as failed or to
resolve the cluster partition, depending on what the reason was for the partition.
There are two main reasons for an i5/OS cluster to be partitioned: communication
failure or certain types of node failure. During a communication failure, the
backup node loses the connection with the primary node. In other words, the
primary node does not respond to the backup node heart beat. In this situation, no
immediate action needs to take place because the primary node is still active and
handling messages. The system administrator should try to determine what the
network problem is and try to resolve it. Notice that the primary node is still
running, but it is not highly available until the partition is resolved.
On the other hand, a partition can also occur if the primary node fails suddenly
without having time to notify the backup node that it has failed. This can occur
during a power outage, especially if the system does not have a backup battery.
This is known as a cluster partition caused by a node failure. To resolve the
partition, the system administrator should let the backup node know that the
primary node has failed. This is done by issuing the following command on the
backup node: CHGCLUNODE CLUSTER([CLUSTER NAME]) NODE([ID OF FAILING NODE])
OPTION(*CHGSTS). When the backup node is notified of the failure, it will behave
as if a failover event has occurred.
WMQ cluster node failure
This scenario occurs when one of the queue managers in a WMQ cluster fails. The
effects on the WMQ cluster vary, depending on whether a partial or full
repository queue manager fails. If a partial repository fails, the repercussions
are minimal. Other queue managers in the cluster can take over the workload. The
only problem is that unprocessed messages on the queues of the failing manager are
marooned and cannot be accessed until the manager is restarted.
On the other hand, with a single full repository configuration, if the full
repository fails, the consequences are bit more drastic. The cluster continues to
function, but it runs in a very unstable state. Changes to a queue manager in
the cluster will not be transmitted to other queue managers. For example, a new
queue cannot be shared, and a new queue manager cannot be added to the cluster.
This is why it is recommended to implement a WMQ cluster with two full
repositories that will serve as backup for each other.
Conclusion
This article illustrates a hypothetical scenario that exposes the risks due to the
lack of a WMQ HA strategy. Some configurations that can be
used for implementing a WMQ HA strategy are provided. The article also emphasizes that
high availability is an important aspect of your WMQ solutions. It is true that by
introducing an HA strategy, your information technology cost will likely increase.
However, this cost can be justified if you consider the consequences of failure to
the system hosting your queue manager.
Appendixes
Appendix A. Creating an IASP
- Identify the DASD tower with unconfigured disks.
- Make sure the tower is connected to your node via an HSL loop.
- Open iSeries Navigator.
- Expand the system that owns the tower.
- Select Configuration and Service > Hardware > Disk Units.
- Enter your SST username and password.
- If you want your IASP to be switchable, expand By Location, right-click
the tower, and select Make your tower switchable.
- If you want to mirror your IASP, expand By Location, right-click the
tower, and select Make your tower private.
- Right-click on Disk Pool, and select New Disk Pool.
- Select All parity sets, and click Next.
- Make the selections as shown in Figure A-1, and click OK.
Figure A-1. New Disk Pool setup
box
- Click Next.
- Click Add Parity-Protected Disk.
- Select the disk units you want to add, and click the Add button. Make sure the
units belong to the tower identified in Step 1.
- Click Next, and click Finish.
Appendix B. Creating an i5/OS cluster
- Make sure the internet daemon server is running using the command STRTCPSVR
SERVER(*INETD).
- Make sure the network attributes have been changed to allow additions to a
cluster with the command CHGNETA ALWADDCLU(*ANY).
- To create the cluster, call the command CRTCLU CLUSTER([CLUSTER NAME])
NODE((NODEA ('172.16.1.241')) (NODEB ('172.16.1.242') )). Note the following:
- NODEA and NODEB are sample node names, as are the IP addresses next to
them.
- Choose your own node names and IP addresses.
- To add a device domain entry for each node, call the command ADDDEVDMNE
CLUSTER([CLUSTER NAME]) DEVDMN([DEVICE DOMAIN NAME]) NODE([NODE NAME]). Note
the following:
- The device domain can be any text string as long as the string is not being
used already.
- Add the same device domain to each node in your cluster.
Appendix C. Creating a device CRG
- Identify the name of the cluster.
- Identify the CRG exit program name and library.
- Determine the name of the primary node and backup nodes to be defined by this
CRG.
- Identify the IASP to be managed by this CRG, and make sure it has been created
under the primary node.
- Create a device description in the backup nodes with the command CRTDEVASP
DEVD([IASP NAME]) RSRCNAME([IASP NAME]).
- Add the takeover IP address to all the nodes using the command ADDTCPIFC
INTNETADR(' [TAKEOVER IP]') LIND([LINE DESC]) SUBNETMASK('[SUBNET MASK]')
AUTOSTART(*NO).
- Start the takeover IP address only in the primary node with STRTCPIFC
INTNETADR('[TAKEOVER IP').
- If your IASP is switchable, call this command: CRTCRG CLUSTER([CLUSTER NAME])
CRG( [CRG NAME]) CRGTYPE(*DEV) EXITPGM([EXIT LIB]/[EXIT NAME]) USRPRF([EXIT
PROFILE]) RCYDMN(( [PRIMARY NODE] *PRIMARY) ([BACKUP NAME] *BACKUP))
EXITPGMFMT(EXTP0200) CFGOBJ(([IAPS NAME] *DEVD *ONLINE '[TAKEOVER IP]')).
- If your IASP is to be mirrored, call this command: CRTCRG CLUSTER([CLUSTER
NAME]) CRG([CRG NAME]) CRGTYPE(*DEV) EXITPGM([EXIT LIB]/[EXIT NAME])
USRPRF([EXIT PROFILE]) RCYDMN(([PRIMARY NODE] *PRIMARY *LAST [PRIMARY SITE NAME]
('[PRIMARY IP]')) ([BACKUP NAME] *BACKUP *LAST [BACKUP SITE NAME] ('[BACKUP
IP]'))) EXITPGMFMT(EXTP0200) CFGOBJ(([IAPS NAME] *DEVD *ONLINE '[TAKEOVER
IP]')). Note the following:
- [PRIMARY SITE NAME] and [BACKUP SITE NAME] could be any two distinct strings
of eight characters or fewer.
- [PRIMARY IP] and [BACKUP IP] are the IPs to be used for mirroring.
Appendix D. CRG exit program sample
Here is the CRG exit program used in this article's example.
Listing 1. Sample CRG exit program
0001.00 PGM PARM(&SUCCESS &ACTIONCODE &MGRNAME &INFO &FORMAT)
0002.00
0003.00 DCL &SUCCESS *INT 4
0004.00 DCL &ACTIONCODE *INT 4
0005.00 DCL &ACTIONDATA *INT 4
0006.00 DCL &MGRNAME *CHAR 256
0007.00 DCL &INFO *CHAR 500
0008.00 DCL &FORMAT *CHAR 8
0009.00 DCL &CURRNODEID *CHAR 8
0010.00 DCL &PRINODEID *CHAR 8
0011.00 DCL &PRVPRINODE *CHAR 8
0012.00 DCL &RECDOMOFF *DEC 4
0013.00 DCL &NODEIDPOS *DEC 4
0014.00 DCL &IASPOFF *DEC 4
0015.00 DCL &IASPPOS *DEC 4
0016.00 DCL &IASPNAM *CHAR 10
0017.00 DCL &PORT *INT 4
0018.00 DCL &RCVR *CHAR 100
0019.00 DCL &RCVRLEN *CHAR 4
0020.00 DCL &SUBSYSNAME *CHAR 20
0021.00 DCL &SUBSYSSTS *CHAR 10
//ASSIGNING THE LISTENER PORT
0022.00 IF (&MGRNAME *EQ 'MGR.I') DO
0023.00 CHGVAR &PORT 1515
0024.00 ENDDO
0025.00 ELSE DO
0026.00 CHGVAR &PORT 1616
0027.00 ENDDO
//HANDLING THE FAILOVER AND SWITCHOVER ACTION CODES
0028.00 IF (&ACTIONCODE *EQ 9 *OR &ACTIONCODE *EQ 10) DO
0029.00 CHGVAR &CURRNODEID %SST(&INFO 53 8)
0030.00 CHGVAR &RECDOMOFF %BIN(&INFO 113 4)
0031.00 CHGVAR &NODEIDPOS (&RECDOMOFF + 5)
0032.00 CHGVAR &PRINODEID %SST(&INFO &NODEIDPOS 8)
0033.00 CHGVAR &ACTIONDATA %BIN(&INFO 129 4)
0034.00 CHGVAR &RECDOMOFF %BIN(&INFO 133 4)
0035.00 CHGVAR &NODEIDPOS (&RECDOMOFF + 5)
0036.00 CHGVAR &PRVPRINODE %SST(&INFO &NODEIDPOS 8)
//MAKING SURE THIS EXIT PROGRAM IS ONLY EXECUTED IN THE FIRST BACKUP NODE.
0037.00 IF (&CURRNODEID *EQ &PRINODEID *AND *NOT (&ACTIONDATA *EQ 3) +
0038.00 *AND *NOT (&PRINODEID *EQ &PRVPRINODE)) DO
0039.00 CHGVAR &IASPOFF %BIN(&INFO 145 4)
0040.00 CHGVAR &IASPPOS (&IASPOFF + 1)
0041.00 CHGVAR &IASPNAM %SST(&INFO &IASPPOS 10)
//ATTACHING TO THE IASP
0042.00 SETASPGRP &IASPNAM
//DETERMINE WHETHER THE QMQM SUBSYSTEM IS ACTIVE
0043.00 CHGVAR %BIN(&RCVRLEN) 100
0044.00 CHGVAR &SUBSYSNAME 'QMQM QMQM '
0045.00 CALL PGM(QWDRSBSD) PARM(&RCVR &RCVRLEN 'SBSI0100' &SUBSYSNAME X'00000000')
0046.00 CHGVAR &SUBSYSSTS %SST(&RCVR 29 10)
//STARTING QMQM SUBSYSTEM, QUEUE MANAGER, LISTENER, AND TRIGGER MONITOR
0047.00 IF (&SUBSYSSTS *EQ *INACTIVE) DO
0048.00 STRSBS QMQM/QMQM
0049.00 ENDDO
0050.00 STRMQM &MGRNAME
0051.00 STRMQMTRM INIT &MGRNAME
0052.00 STRMQMLSR &PORT &MGRNAME
0053.00 ENDDO
0054.00 ENDDO
0055.00 CHGVAR &SUCCESS 0
0056.00 ENDPGM
|
Appendix E. Moving a queue manager to an
IASP
- Identify your primary node and your backup nodes.
- Do the following on your primary node:
- Make sure your queue manager is ended.
- Make sure your IASP is vary on using the command VRYCFG CFGOBJ([IASP NAME])
CFGTYPE(*DEV) STATUS(*ON).
- Create the qmgrs directory under the IASP. There will be a directory under
root with the name of your IASP, which is QSH CMD('mkdir -p
/[IASP_NAME]/QIBM/UserData/mqm/qmgrs/').
- Move the IFS objects of your manager to the qmgrs directory just created under
the IASP using QSH CMD('mv /QIBM/UserData/mqm/qmgrs/[MANAGER NAME] /[IASP
NAME]/QIBM/UserData/mqm/qmgrs').
- Create a temporary save file named MGRLIB with the command CRTSAVF
QGPL/MGRLIB.
- Save your queue manager library to the MGRLIB save file with the command
SAVLIB LIB([MANGER LIBRARY]) DEV(*SAVF) SAVF(QGPL/MGRLIB).
- Delete the queue manager library. Ignore all inquiry messages. Use the
command DLTLIB [MANAGER LIBRARY].
- Restore your queue manager library to the IASP with the following: RSTLIB
SAVLIB([MANAGER LIBRARY]) DEV(*SAVF) SAVF(QGPL/MGRLIB) RSTASPDEV([IASP NAME]).
- Delete temporary save file. Use the command DLTF FILE(QGPL/MGRLIB).
- Create a Symbolic Link to the queue manager IFS objects under the IASP with
the command ADDLNK OBJ('/[IASP NAME]/QIBM/UserData/mqm/qmgrs/[MANAGER NAME]')
NEWLNK('/QIBM/UserData/mqm/qmgrs/[MANAGER NAME]').
- Attach to the IASP by entering the command SETASPGRP [IASP NAME].
- Start your queue manager with the command STRMQM [MANAGER NAME].
- On your backup nodes do the following:
- Create temporary queue manager directory with the command QSH CMD('mkdir -p
/[IASP NAME]/QIBM/UserData/mqm/qmgrs/[MANAGER NAME]').
- Create a symbolic link to the queue manager temporary directory with the
command ADDLNK OBJ('/[IASP NAME]/QIBM/UserData/mqm/qmgrs/[MANAGER NAME]')
NEWLNK('/QIBM/UserData/mqm/qmgrs/[MANAGER NAME]').
- Delete the temporary directory with the command QSH CMD('rm -r /[IASP NAME]').
- Add the following at the end of the file /QIBM/UserData/mqm/mqs.ini:
QueueManager:
Name=[MANAGER NAME]
Prefix=/QIBM/UserData/mqm
Library=[MANAGER LIBRARY]
Directory=[MANAGER DIRECTORY]
|
Appendix F. Setting up geographical
mirroring
- Create the cluster between nodes that will participate on the
mirroring.
- Create an IASP under the primary node, as described in Appendix A.
- Create the device CRG that controls the IASP, as described in Appendix C.
- Make sure both your IASP and your device CRG are powered off.
- After the IASP is created, right-click on it and select Geographical
Mirroring > Configure Geographical Mirroring.
- Click Next, and click Next again.
- Type the name of the node that is to be your target.
- Enter your SST credentials for your target system.
- Click Add Disks.
- Select your target disks, and click the Add button.
- Click Next, and click Finish.
- Start the CRG with the command STRCRG [CLUSTER NAME] [CRG NAME].
- Vary on the IASP with the command VRYCFG CFGOBJ([IASP NAME]) CFGTYPE(*DEV)
STATUS(*ON).
Appendix G. Creating a WMQ cluster
- Create three queue managers, such as TESTMGRA, TESTMGRB, and TESTMGRC.
TESTMGRA and TESTMGRB are full repositories.
- Modify TESTMGRA and TESTMGRB to hold the repository (TESTCLUS) as follows:
- Call command CHGMQM MQMNAME('TESTMGRA') REPOS(TESTCLUS).
- Call command CHGMQM MQMNAME('TESTMGRB') REPOS(TESTCLUS).
- Create CLUSRCVR channels as follows:
- Call command CRTMQMCHL CHLNAME(TO.MGRA) CHLTYPE(*CLUSRCVR)
MQMNAME('TESTMGRA') CONNAME('HOSTNAME(PORTNO)'), where HOSTNAME is the name of
the computer where TESTMGRA resides and PORTNO is the port number where the
listener is listening.
- Call command CRTMQMCHL CHLNAME(TO.MGRB) CHLTYPE(*CLUSRCVR)
MQMNAME('TESTMGRB') CONNAME('HOSTNAME(PORTNO)'), where HOSTNAME is the name of
the computer where TESTMGRB resides and PORTNO is the port number where the
listener is listening.
- Call command CRTMQMCHL CHLNAME(TO.MGRC) CHLTYPE(*CLUSRCVR)
MQMNAME('TESTMGRC') CONNAME('HOSTNAME(PORTNO)'), where HOSTNAME is the name of
the computer where TESTMGRC resides and PORTNO is the port number where the
listener is listening.
- Create CLUSSDR channels, as follows:
- Call command CRTMQMCHL CHLNAME(TO.MGRA) CHLTYPE(*CLUSSDR)
MQMNAME('TESTMGRB') CONNAME('HOSTNAME(PORTNO)'), where HOSTNAME is the name of
the computer where TESTMGRA resides and PORTNO is the port number where the
listener is listening.
- Call command CRTMQMCHL CHLNAME(TO.MGRB) CHLTYPE(*CLUSSDR)
MQMNAME('TESTMGRA') CONNAME('HOSTNAME(PORTNO)'), where HOSTNAME is the name of
the computer where TESTMGRB resides and PORTNO is the port number where the
listener is listening.
- Point the TESTMGRC cluster sender channel to one of the repositories, such
as TESTMGRA. Call command CRTMQMCHL CHLNAME(TO.MGRA) CHLTYPE(*CLUSSDR)
MQMNAME('TESTMGRC') CONNAME('HOSTNAME(PORTNO)'), where HOSTNAME is the name of
the computer where TESTMGRA resides and PORTNO is the port number where the
listener is listening.
- Start listeners as follows:
- Call command STRMQMLSR PORT(PORTNO) MQMNAME(TESTMGRA).
- Call command STRMQMLSR PORT(PORTNO) MQMNAME(TESTMGRB).
- Define cluster-shared queue TESTQ under TESTMGRB with the command CRTMQMQ
QNAME(TESTQ) QTYPE(*LCL) MQMNAME('TESTMGRB') CLUSTER(TESTCLUS).
 |
Glossary
ASP: Auxiliary storage pool. A group of disk units defined from the
auxiliary storage devices.
Client box: Computer running any operating system, including i5/OS, where a
JMS client can be run.
CRG: Cluster resource group. A collection of related cluster resources that
define actions to be taken during a switchover or failover operation of the access
point of resilient resources. The group describes a recovery domain and supplies
the name of the cluster resource group exit program that manages the movement of
an access point.
CRG exit program: Program that is called by the i5/OS cluster resource
service when a CRG cluster-related event occurs. For example, when a CRG is
switched from primary node to backup node the CRG exit program is called.
CRG takeover IP: Dynamic IP address that is moved from primary node to
backup node when an application or device CRG is switched over.
Device CRG: Cluster resource group that describes and manages device
resources like IASPs.
Device domain: Identifies cluster nodes that are capable of assuming
control of the devices that are managed by the device CRG.
Failover: Occurs when the primary node of the cluster fails and the CRGs
are switched with all the resources they manage.
Geographic mirroring: A subfunction of cross-site mirroring (XSM) that
generates a mirror image of an independent disk pool on a system, which can be
geographically distant from the originating site for availability or
protection purposes.
HSL loop: High-speed link that interconnects all nodes and switchable disk
towers.
i5/OS: Pertaining to the IBM licensed program that can be used as the
operating system for the System i platform.
IASP: Independent Auxiliary Storage Pool. A type of user ASP that serves as
an extension of single level storage.
IFS: Integrated file system of i5/OS.
IOP: I/O processor, input/output processor. A processor dedicated to
controlling channels or communication links.
IPL: initial program load. The process that loads the system programs from
the system auxiliary storage, checks the system hardware, and prepares the system
for user operations.
JMS: Java™ messaging service. Standard Java API for sending and
receiving e-messages.
Marooned messages: WMQ messages that are not accessible at the moment
because of a problem with the queue manager, but can become accessible once the problem is
resolved.
Mirrored IASP: An IASP whose data is being mirrored to a target IASP using
geographical mirroring.
Resilient data: Application data that can withstand system or application
outage, without getting corrupted.
Single point of failure: A component that fails and makes the whole system
fail. Consider a system to be a collection of interdependent components.
Switchable IASP: IASP that can be switched between systems in a cluster.
This can be an expansion unit containing disk units in a multiple system
environment. This could also be an IOP containing disk units in an LPAR
environment.
Switchover: Occurs when a specific CRG is manually switched from the
primary node to the backup node.
System i cluster: Collection of System i computers or partitions that are
logically linked together.
Vary on: To make an independent disk pool available for its normal,
intended use. All of the primary and secondary disk pools in a disk pool group
will vary on together.
WMQ: WebSphere message queuing.
WMQ cluster: A collection of queue managers that are logically associated in
some way.
WMQ full repository: A complete set of information about every queue
manager in a cluster. This set of information is called the repository (or
sometimes the full repository) and is usually held by two of the queue managers in
the cluster.
WMQ partial repository: A partial set of information about queue managers
in a cluster. A partial repository is maintained by all cluster queue managers
that do not host a full repository.
Resources Learn
Get products and technologies
- Build your next
development project with IBM trial software
for download directly from developerWorks.
Discuss
About the author  | 
| | David Peraza is a software engineer with IBM Systems and Technology Group. He works with WebSphere MQ integration with the System i platform. He was a technical mentor for the PHP on System i 2006 summer intern project. You can reach David at dperaza@us.ibm.com. |
Rate this page
| 
