Example: Deploying a simple Native HA configuration on Linux

This example installs a queue manager using a simple Native HA configuration across three Linux hosts.

Before you begin

To complete this example, you must first have completed the following prerequisites:

Follow the steps in Checking requirements on Linux.
Follow the steps in Planning to install IBM MQ on Linux
For each of the three Linux hosts, follow the steps in Preparing the system on Linux.
Download the 10.0.0 or above IBM® MQ base install images for your Linux distribution (see IBM MQ downloads).

About this task

The Native HA configuration defines a group of instances, which together form a highly available queue manager. Each instance is installed on a different host, so if one host fails the others can take over.

This example defines a single Native HA group consisting of three IBM MQ instances installed on their own Linux hosts. The host names of the instances are alpha, beta, and gamma. Throughout the example, these names should be replaced with the host names of your systems.

This example uses the default installation location (/opt/mqm) and data directory (/var/mqm). If you are using non-default locations, be sure to replace these paths accordingly throughout the example.

See Native HA for more details on Native HA configurations. Native HA also supports Cross Region Replication (CRR), where an additional Native HA group acts as a recovery group, see Native HA CRR for more information.

This example uses a single certificate to secure the queue manager instance traffic.

This example walks through defining a Native HA group on Linux hosts to provide high availability, setting up monitoring and restarts, and forcing a failover from the active instance to a replica.

Procedure

Install IBM MQ on all three Linux hosts.

Follow the instructions in Installing and uninstalling IBM MQ on Linux to accept the IBM MQ license and install the IBM MQ server on all three of your hosts.

For this example, it is easiest to use the default locations for installation (/opt/mqm) and data (/var/mqm). If you change the install or data location on one system, ensure that you mirror the same change across all other hosts.
Set up environments ready to install the Native HA instances:
1. On each Linux host, log in as the mqm user and set up your environment by using the following command:
```
. /opt/mqm/bin/setmqenv -s 
```
2. Check that the environment is set up correctly by entering the following command:
```
dspmqver
```
  If the command completes successfully, and the expected version number and installation name are returned, the environment is set up correctly.
Create the Native HA queue manager instances. On hosts alpha, beta, and gamma, create a queue manager with log replication. (You might want to use the AutoConfig stanza to ensure objects are always defined in a uniform way, the -ic option can be added to crtmqm to supply a mqsc file for the queue manager.)
1. Create the first instance on the node named alpha:
```
crtmqm -lr alpha -lf 8192 -lp 10 -ls 10 -p 1414 MYQMGR
```
  The queue manager is created:
```
IBM MQ queue manager 'MYQMGR' created.
Directory '/var/mqm/qmgrs/MYQMGR' created.
The queue manager is associated with installation 'Installation1'.
Creating or replacing default objects for queue manager 'MYQMGR'.
Default objects statistics : 83 created. 0 replaced. 0 failed.
Completing setup.
Setup completed.
```
2. Create the second instance on the node named beta:
```
crtmqm -lr beta -lf 8192 -lp 10 -ls 10 -p 1414 MYQMGR
```
3. Create the third instance on the node named gamma:
```
crtmqm -lr gamma -lf 8192 -lp 10 -ls 10 -p 1414 MYQMGR
```
Create a keystore for securing the instance and group traffic:
1. To create this keystore, log on to the alpha host and run the following commands:
```
runmqakm -keydb -create -db /var/mqm/qmgrs/MYQMGR/ssl/keystore.kdb -pw PASSWORD -stash 
runmqakm -cert -create -db /var/mqm/qmgrs/MYQMGR/ssl/keystore.kdb -pw PASSWORD -label nha-qm-replication -dn CN=MYQMGR-REPLICATION -size 2048 
```
  Where PASSWORD is a password of your own choosing.
2. Copy the four keystore files to the same location (/var/mqm/qmgrs/MYQMGR/ssl/) on the beta and gamma hosts, for example, by using sftp.
3. On each host (alpha, beta, and gamma), ensure that the files have mqm group ownership and have read access for the mqm group:
```
chown :mqm /var/mqm/qmgrs/MYQMGR/ssl/keystore.* 
chmod g+r /var/mqm/qmgrs/MYQMGR/ssl/keystore.*
```
  Important: The examples described here are not suitable for a production environment, and are solely intended as examples to get going quickly. Certificate management is a complex subject for advanced users. For production, you must consider things like rotation, revocation, key length, disaster recovery and much more.
Create the Native HA group. The group is created by editing the /var/mqm/qmgrs/MYQMGR/qm.ini files for each queue manager instance.
1. Update the NativeHALocalInstance stanza for each of the alpha, beta, and gamma instances to define the group and the certificate to use to secure the instance traffic:
```
NativeHALocalInstance:
   Name=<instance-name>
   CipherSpec=ANY_TLS12
   CertificateLabel=nha-qm-replication
   KeyRepository=/var/mqm/qmgrs/MYQMGR/ssl/keystore
   
```
  Where:
  - <instance-name> is the name of the instance that you are configuring (that is, alpha, beta, or gamma).
  - CertificateLabel specifies the certificate for traffic between instances.
2. For each of the alpha, beta, and gamma instances add the local group members:
```
NativeHAInstance:
   Name=alpha
   ReplicationAddress=alpha.example.com(9414)
NativeHAInstance:
   Name=beta
   ReplicationAddress=beta.example.com(9414)
NativeHAInstance:
   Name=gamma
   ReplicationAddress=gamma.example.com(9414)
```
  (Replace host names and replication addresses with the names and replication addresses of your own Linux nodes.)
3. Start the instances in the group. You can now start the queue manager instances on alpha, beta, and gamma hosts:
  strmqm MYQMGR
4. Verify the group has deployed correctly. Issue the dspmq -o nativeha -x command on the alpha, beta, and gamma hosts, to display the group status. On the active instance, the status is:
```
QMNAME(MYQMGR)                                            ROLE(Active) INSTANCE(<instance-name>) INSYNC(yes) QUORUM(3/3) GRPLSN(<n:n:n:n>) GRPNAME() GRPROLE(Live)
 INSTANCE(alpha) ROLE(Active) REPLADDR(<alpha-fqdn>) CONNACTV(yes) INSYNC(yes) BACKLOG(0) CONNINST(yes) ACKLSN(<n:n:n:n>) HASTATUS(Normal) SYNCTIME(<timestamp>) ALTDATE(<date>) ALTTIME(<time>)
 INSTANCE(beta) ROLE(Replica) REPLADDR(<beta-fqdn>) CONNACTV(yes) INSYNC(yes) BACKLOG(0) CONNINST(yes) ACKLSN(<n:n:n:n>) HASTATUS(Normal) SYNCTIME(<timestamp>) ALTDATE(<date>) ALTTIME(<time>)
 INSTANCE(gamma) ROLE(Replica) REPLADDR(<gamma-fqdn>) CONNACTV(yes) INSYNC(yes) BACKLOG(0) CONNINST(yes) ACKLSN(<n:n:n:n>) HASTATUS(Normal) SYNCTIME(<timestamp>) ALTDATE(<date>) ALTTIME(<time>)
```
  The key parts of the status are:
  - ROLE(Active)
  - QUORUM(3/3) - it has full quorum, note this can take a few seconds
  Note which host in the group has the Active role. The other hosts will have a similar status, but the ROLE is Replica.
Add monitoring and restarting of the queue manager.

If an instance unexpectedly fails, it is important to detect the failure and restart the instance as quickly as possible. In this example, a sample demonstrates the implementation of a 'liveness' check. The sample uses a python script and systemd daemon. When you use this method, you must use systemd to start and end the queue manager instead of the IBM MQ commands.

Follow the instructions in Monitoring, restarting, and ending queue manager instances to add queue manager monitoring and restarting to the alpha, beta, and gamma hosts.
Define a queue. On the active instance, start MQSC by entering the following command:
```
runmqsc MYQMGR
```
A message indicates when MQSC starts. MQSC has no command prompt. Define a local persistent queue:
```
DEFINE QLOCAL('QUEUE1') DEFPSIST(YES)
```
A message indicates when the queue is created. Stop MQSC by entering the following command:
```
end
```
Messages are shown, followed by the command prompt. The QUEUE1 queue is replicated to the other instances in the group.
Add persistent messages to the queue. The DEFPSIST(YES) attribute on the QUEUE1 queue causes messages put to the queue with the amqsputc sample to be persistent. They are replicated to the replica instances.
Change directory into /opt/mqm/samp/bin, which contains the sample programs. Put messages on the queue by entering the following commands:
```
./amqsput QUEUE1 MYQMGR
```
If connection to the queue manager is successful, the following response is output:
```
target queue is QUEUE1
```
Put several messages to the queue, by entering some text then pressing Enter each time. Enter a blank line to end the message input. The following message is shown:
```
Sample AMQSPUT0 end
```
Your messages are now on the queue and the command prompt is shown.
Restart the active to mimic a host failure. Stop the active instance, wait a second and the start the active using the following command:
```
sudo systemctl stop mqmonitor@MYQMGR
sudo systemctl start mqmonitor@MYQMGR
```
Locate the new active instance. Issue the dspmq -o nativeha -x command on alpha, beta, and gamma hosts, to display the group status and identify which host is now the active instance.
Retrieve messages sent before the active instance was ended. On the current active instance, change directory into /opt/mqm/samp/bin, which contains the sample programs. Get the messages from the queue by entering the following command:
```
./amqsget QUEUE1 MYQMGR
```
The messages you added in step 8 are available, and are output. After a few seconds, the command exits.

What to do next

You could also try adding a recovery group to this group to provide cross region replication using the instructions at Example: Deploying a simple Native HA CRR configuration on Linux.