Creating a multi-instance queue manager for WebSphere MQ on Linux

This article shows you how to implement the new WebSphere MQ V7 multi-instance queue manager feature on Linux. It provides high availability without using an HA Coordinator, by automatically switching to a standby server if the active server fails. The active and standby servers are multiple instances of the same queue manager and share the same messages and data, which reside on networked storage accessed via a network file system (NFS) protocol, such as NFS V4.

Chaitra Sampige (csampige@in.ibm.com), Software Engineer, Websphere MQ Test Team, IBM

Chaitra Sampige is a Software Tester working on the Websphere MQ Test Team at the IBM Software Lab in Bangalore, India. She has four years of testing experience with the Websphere MQ Test Team and also handles WebSphere TxSeries and WebSphere MQ interoperability testing. She is a Certified Software Test Engineer (CSTE) and has a Bachelor of Engineering degree in Information Science from Viswesvaraya Technological University in India. You can contact Chaitra at csampige@in.ibm.com.



09 June 2010

Introduction

IBM® WebSphere® MQ V7 can increase messaging availability out-of-the-box without specialized skills or hardware. In the event of an unplanned outage, it provides automatic failover via multi-instance queue managers, and you can initiate planned outages such as to apply software maintenance via controlled switchovers.

Messages and data for multi-instance queue managers are held on networked storage accessed via a modern network file system protocol,such as Network File System (NFS) V4. You can define and start multiple instances of the queue manager on different machines, with one active and one standby instance. The active queue manager instance processes messages and accepts connections from applications and from other queue managers. It holds a lock on the queue manager data to ensure that there is only one active instance of the queue manager. The standby queue manager instance periodically checks whether the active queue manager instance is still running. If the active queue manager instance fails or is no longer connected, the standby instance acquires the lock on the queue manager data as soon as it is released, performs queue manager restart processing, and becomes the active queue manager instance.

Prerequisites

Install Websphere MQ V7.0.1.0

  1. Install Websphere MQ V7.0.1.0) on both machines.
  2. Run the File System Check tool (amqmfsck ) to verify that the file system is compliant with POSIX standards and capable of sharing queue manager data to support multi-instance queue managers.
  3. Before running the File System Check tool, create new directories on both machines and start the NFS server on Server 1:
    1. Create a directory HA under root.
    2. Create another directory mqdata under HA.
    3. Add the following user and group ownership to the HA directory recursively: chown -R mqm:mqm /HA .
    4. Add the following permissions to the HA directory recursively: chmod -R ug+rwx /HA.

Setting up NFS share

NFS server (Server 1)

  1. Use the following command to check whether NFS is on:
    chkconfig --list nfs
    For example: 
    [root@lins shared]# chkconfig --list nfs
    nfs    0:off   1:off   2:off   3:off   4:off   5:off   6:off
  2. If Fields 2, 3, 4, and 5 are off in the above output, then it means that NFS is turned off and you need to turn it on: chkconfig nfs on.
  3. To verify whether NFS is on:
    [root@lins shared]#  chkconfig --list nfs 
    nfs      0:off   1:off   2:on    3:on    4:on    5:on    6:off
  4. To export mount point (/HA) to another node:
    Add /HA *(rw,sync,no_wdelay,fsid=0) to /etc/exports
    Start the NFS daemon on Server 1.
    /etc/init.d/nfs start 
    
    For example: 
    root@lins shared]# /etc/init.d/nfs start
    Starting NFS services:                                [  OK  ]
    Starting NFS quotas:                                  [  OK  ]
    Starting NFS daemon:                                  [  OK  ]
    Starting NFS mountd:                                  [  OK  ]
  5. To verify the mount point that is being exported from this server to another server, use the following command:
    showmount -e
    For example: [root@lins shared]# showmount -e
    Export list for lins.in.ibm.com: /HA *

NFS client (Server 2)

  1. To mount the exported file system on Server 2, find the IP address of Server 1 by using the command ifconfig a on Server 1.
  2. Mount the file system as NFS4 from Server 1 to Server 2: mount -t nfs4 -o hard,intr 9.122.163.105:/ /HA.
  3. Run the command mount v to check whether the NFS mount was successful:
    For example: 
    [root@gtstress42 ~]# mount -v
    /dev/sdb2 on /testpool type ext3 (rw)
    /dev/sda5 on /usr type ext3 (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
    9.122.163.105:/ on /HA type nfs4 (rw,hard,intr,addr=9.122.163.105)

Running the WebSphere MQ File System Check tool

  1. Change the login to mqtest on both machines to run the amqmfsck tool:
    su - mqtest
    export PATH=/opt/mqm/bin:$PATH
    amqmfsck /HA/mqdata
    
    For example: 
    Server 1: [mqst@lins root]$ amqmfsck /HA/mqdata
    The tests on the directory completed successfully.
    Server 2: [mqst@gtstress42 /]$ amqmfsck /HA/mqdata
    The tests on the directory completed successfully.
    	
    amqmfsck -c /HA/mqdata
    
    For example: 
    Server 1: [mqst@lins root]$ amqmfsck -c /HA/mqdata
    Start a second copy of this program with the same parameters on another server.
    Writing to test file. This will normally complete within about 60 seconds...
    The tests on the directory completed successfully.
    
    For example: 
    Server 2: [mqst@gtstress42 /]$ amqmfsck -c /HA/mqdata
    Writing to test file. This will normally complete within about 60 seconds..
    The tests on the directly completed successfully
    
    amqmfsck -wv /HA/mqdata (Run this command at the same time on the machines)
                
    For example: 
    Server 1: [mqst@lins root]$ amqmfsck -wv /HA/mqdata
    System call: stat("/HA/mqdata",&statbuf)
    System call: fd = open("/HA/mqdata/amqmfsck.lkw",O_RDWR,0666)
    System call: fchmod(fd,0666)
    System call: fstat(fd,&statbuf)
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    Start a second copy of this program with the same parameters on another server.
    File lock acquired.
    Press Enter or terminate this process to release the lock.
    System call: close(fd)
    File lock released.
    The tests on the directory completed successfully.
    
    For example: 
    Server 2 : [mqst@gtstress42 /]$ amqmfsck -wv /HA/mqdata
    System call: stat("/HA/mqdata",&statbuf)
    System call: fd =open("/HA/mqdata/amqmfsck.lkw",O_RDWR,0666)
    System call: fchmod(fd,0666)
    System call: fstat(fd,&statbuf)
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    Waiting for the file lock.
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    Waiting for the file lock.
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    Waiting for the file lock.
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    Waiting for the file lock.
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    Waiting for the file lock.
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    Waiting for the file lock.
    System call: fcntl(fd,F_SETLK,F_WRLCK)
    File lock acquired.
    Press Enter or terminate this process to release the lock.
    System call: close(fd)
    File lock released.
    The tests on the directory completed successfully
  2. After verifying the file system for file locking, you can create a multi instance queue manager on the selected machine:
  3. On Server 1:
    cd  HA
    mkdir logs
    mkdir qmgrs
    chown -R mqm:mqm /HA
    chmod -R ug+rwx /HA
    crtmqm -ld /HA/logs -md /HA/qmgrs -q QM1
  4. Copy the queue manager details from Server 1 to Server 2:
    dspmqinf -o command QM1
    (Run this command on Server 1)
    For example: addmqinf -s QueueManager -v Name=QM1 -v Directory=QM1 -v 
    Prefix=/var/mqm -v DataPath=/HA/qmgrs/QM1
    Multi-instance queue manager setup
  5. Start the active instance of the queue manager on Server 1. Copy the output below and run it on Server 2:
    strmqm -x QM1
    		
    For example: 
    [mqst@lins HA]$ strmqm -x QM1
    WebSphere MQ queue manager 'QM1' starting.
    5 log records accessed on queue manager 'QM1' during the log replay phase.
    Log replay for queue manager 'QM1' complete.
    Transaction manager state recovered for queue manager 'QM1'.
    WebSphere MQ queue manager 'QM1' started.
    [mqst@lins HA]$ dspmq
    QMNAME(QM1)           STATUS(Running)
    [mqst@lins HA]$ dspmq -x -o standby
    QMNAME(QM1)           STANDBY(Permitted)
    INSTANCE(lins.in.ibm.com) MODE(Active)
    INSTANCE(gtstress42.in.ibm.com) MODE(Standby)
    
    Server 2:
    [mqst@gtstress42 /]$ strmqm -x QM1
    WebSphere MQ queue manager 'QM1' starting.
    A standby instance of queue manager 'QM1' has been started. 
    The active instance is running elsewhere.
    [mqst@gtstress42 /]$ dspmq
    QMNAME(QM1)           STATUS(Running as standby)
    [mqst@gtstress42 /]$ dspmq -x -o standby
    QMNAME(QM1)           STANDBY(Permitted)
    INSTANCE(lins.in.ibm.com) MODE(Active)
    INSTANCE(gtstress42.in.ibm.com) MODE(Standby)
  6. To completely stop a multi-instance queue manager, issue a normal endmqm command on the active instance on Server 1:
    endmqm -i QM1
    WebSphere MQ queue manager 'QM1’' ended.
    Both instances end
  7. To stop just the standby instance on Server 2, use this command:
    endmqm -x QM1
    WebSphere MQ standby queue manager instance 'QM1’' ended.
    Failing Queue Manager A
  8. To switch servers from active to standby, on machine A, enter: endmqm -is QM1 WebSphere MQ queue manager 'QM1' ended, permitting switchover to a standby instance. Once the active instance has ended, the standby instance will try to become the active instance:
    Server 1:
    [mqst@lins HA]$ endmqm -is QM1
    WebSphere MQ queue manager 'QM1' ending.
    WebSphere MQ queue manager 'QM1' ended, permitting switchover to a standby instance.
    [mqst@lins HA]$ dspmq
    QMNAME(QM1)
    STATUS(Running elsewhere)
    
    Server 2:
    [mqst@gtstress42 /]$ dspmq
    QMNAME(QM1)           
    STATUS(Running)
    Clients connecting through standby instance

Checking and creating users and groups

  1. The UID and GID of mqm and mqtest must be the same on both machines before installing WebSphere MQ. If they are different on the two nodes, change them to be the same and reboot the machine to reflect the changes. Here are the steps to change the UID and GID of mqm and mqmtestr:
    1. For example, if UID and GID of mqm are 501 on Server 1, then you need to set the UID and GID of mqm on Server 2 to 501:
      groupmod -g 501 mqm
      usermod -u 501 -g 501 mqm
    2. Reboot Server 2 to reflect the changes.
    3. The same step applies to mqtest, and mqtest needs to be a part of mqm group.
    4. To create new mqm and mqtest ids on a system, use the commands below:
      groupadd -g <GID> mqm
      useradd -u  <UID> mqm -g mqm
      useradd -u  <UID> mqtest -g mqm
  2. To check that the ids have been created correctly on both machines, use the commands below:
    id mqm
    For example: 
    Server 1: uid=501(mqm) gid=501(mqm) groups=501(mqm)
    Server 2: uid=501(mqm) gid=501(mqm) groups=501(mqm)
    
    id mqtest
    For example: Server 1: uid=550(mqtest) gid=501(mqm) groups=501(mqm)
    Server 2: uid=550(mqtest) gid=501(mqm) groups=501(mqm)

Acknowledgments

The author would like to thank Ummahesh Ponnuswamy of IBM Software Group and Swetha Vb Prabhakar of IBM Global Business Services for their help with this article.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=495144
ArticleTitle=Creating a multi-instance queue manager for WebSphere MQ on Linux
publish-date=06092010