Example: Configuring a Native HA queue manager

This example shows how you deploy a queue manager using the native high availability feature into the Red Hat® OpenShift® Container Platform (OCP) using the IBM® MQ Operator.

Before you begin

To complete this example, you must first have completed the following prerequisites:

Install the IBM MQ client, and add the installed samp/bin and bin directories to your PATH. The client provides the runmqakm, amqsputc and amqsgetc applications required by this example. Install the IBM MQ client as follows:
- For Windows and Linux®: Install the IBM MQ redistributable client for your operating system from https://ibm.biz/mq92redistclients
- For Mac: Download and set up the IBM MQ MacOS Toolkit. See https://ibm.biz/mqdevmacclient.
Install the OpenSSL tool for your operating system. You need this to generate a self-signed certificate for the queue manager, if you do not already have a private key and certificate.
Create a Red Hat OpenShift Container Platform (OCP) project/namespace for this example and follow the steps in the task Preparing your Red Hat OpenShift project for IBM MQ
On the command line, log in to the OCP cluster, and switch to the namespace that you just created.
Ensure the IBM MQ Operator is installed and available in the namespace.
Configure a default storage class in the OCP, to be used by your queue manager. If you want to complete this tutorial without setting a default storage class, see Note 2: Using a non-default storage class.

About this task

Native HA queue managers involve an active and two replica Kubernetes Pods. These run as part of a Kubernetes Stateful Set with exactly three replicas and a set of Kubernetes Persistent Volumes. For more information about native HA queue managers, see High availability for IBM MQ in containers.

The example provides a custom resource YAML that defines a native HA queue manager that uses persistent storage and is configured with TLS. After deploying the queue manager into the OCP, you simulate failure of the active queue manager pod. You see automatic recovery occur, and prove that it has succeeded by putting and getting messages after the failure.

Example

Create a TLS private key and certificates for MQ server

You can create a self-signed certificate for the queue manager, and add the certificate to a key database to act as the truststore for the client. If you already have a private key and certificate, you can use those instead. Note that you should only use self-signed certificates for development purposes.

To create a self-signed private key and a public certificate in the current directory, run the following command:

openssl req -newkey rsa:2048 -nodes -keyout tls.key -subj "/CN=localhost" -x509 -days 3650 -out tls.crt

Create a TLS private key and certificates for internal use by Native HA

The three pods in a Native HA queue manager replicate data over the network. You can create a self-signed certificate for use when replicating internally. Note that you should only use self-signed certificates for development purposes.

To create a self-signed private key and a public certificate in the current directory, run the following command:

openssl req -newkey rsa:2048 -nodes -keyout nativeha.key -subj "/CN=localhost" -x509 -days 3650 -out nativeha.crt

Add the queue manager public key to a client key database

A client key database is used as the truststore for the client application.

Create the client key database:

runmqakm -keydb -create -db clientkey.kdb -pw password -type cms -stash

Add the previously generated public key to the client key database:

runmqakm -cert -add -db clientkey.kdb -label mqservercert -file tls.crt -format ascii -stashed

Create a secret containing TLS certificates for queue manager deployment

So that your queue manager can reference and apply the key and certificate, create a Kubernetes TLS secret, referencing the files created above. When you do this, ensure you are in the namespace you created before you began this task.

oc create secret tls example-ha-secret --key="tls.key" --cert="tls.crt"

Create a secret containing the internal native HA TLS certificate and key

oc create secret tls example-ha-secret-internal --key="nativeha.key" --cert="nativeha.crt"

Create a config map containing MQSC commands

Create a Kubernetes config map containing the MQSC commands to create a new queue and a SVRCONN Channel, and to add a channel authentication record that allows access to the channel by blocking only those users called nobody.

Note that this approach should be used only for development purposes.

Ensure you are in the namespace you created earlier (see Before you begin), then enter the following YAML in the OCP UI, or using the command line:

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-mi-configmap
data:
  tls.mqsc: |
    DEFINE QLOCAL('EXAMPLE.QUEUE') DEFPSIST(YES) REPLACE 
    DEFINE CHANNEL(HAQMCHL) CHLTYPE(SVRCONN) TRPTYPE(TCP) SSLCAUTH(OPTIONAL) SSLCIPH('ANY_TLS12_OR_HIGHER')
    SET CHLAUTH(HAQMCHL) TYPE(BLOCKUSER) USERLIST('nobody') ACTION(ADD)

Configure routing

If you are using an IBM MQ client or toolkit at IBM MQ 9.2.1 or later, you can configure routing to the queue manager by using a queue manager configuration file (an INI file). Within the file, you set the OutboundSNI variable to route based on hostname rather than channel name.

Create a file in the directory you are running commands in, named mqclient.ini, containing exactly the following text:

SSL:
   OutboundSNI=HOSTNAME

Do not change any values in this INI file. For example, the string HOSTNAME must not be changed.

For further details, see SSL stanza of the client configuration file.

If you are using an IBM MQ client or toolkit earlier than IBM MQ 9.2.1, you need to create an OCP route instead of the previous configuration file. Follow the steps in Note 1: Creating a route.

Deploy the queue manager

Important: In this example we use the MQSNOAUT variable to disable authorization on the queue manager, which allows us to focus on the steps required to connect a client using TLS. This is not recommended in a production deployment of IBM MQ, because it causes any applications connecting to have full administrative powers, with no mechanism to lower the permissions for individual applications.

Copy and update the following YAML.

Ensure the correct license is specified. See Licensing reference for mq.ibm.com/v1beta1. In IBM Cloud Pak® for Integration 2021.1.1, the license must the evaluation license L-RJON-BYRMYW
Accept the license by changing false to true.

Queue manager custom resource YAML:

apiVersion: mq.ibm.com/v1beta1
kind: QueueManager
metadata:
  name: nativeha-example
spec:
  license:
    accept: false
    license: L-RJON-C7QG3S
    use: Production
  queueManager:
    name: HAEXAMPLE
    availability:
      type: NativeHA
      tls:
        secretName: example-ha-secret-internal
    mqsc:
    - configMap:
        name: example-mi-configmap
        items:
        - tls.mqsc
  template:
    pod:
      containers:
        - env:
            - name: MQSNOAUT
              value: 'yes'
          name: qmgr
  version: 9.2.5.0-r3
  pki:
    keys:
      - name: example
        secret:
          secretName: example-ha-secret
          items: 
          - tls.key
          - tls.crt

Ensuring that you are in the namespace created earlier, deploy the updated YAML, using the Red Hat OpenShift Container Platform web console, the command line, or using the IBM Cloud Pak for Integration Platform Navigator.

There is a brief delay while the system configures the Native HA queue manager, after which the queue manager should be available for use.

Validation

In this section we validate that the queue manager behaves as expected.

Confirm that the queue manager is running

The queue manager is now being deployed. Confirm it is in Running state before proceeding. For example:

oc get qmgr nativeha-example

Test the connection to the queue manager

To confirm the queue manager is configured for one-way TLS communication, use the amqsputc and amqsgetc sample applications:

Find the queue manager hostname

To find the queue manager hostname for route nativeha-example-ibm-mq-qm, run the following command. The hostname is returned in the HOST field.

oc get routes nativeha-example-ibm-mq-qm

Specify the queue manager details

Create a file CCDT.JSON that specifies the queue manager details. Replace the host value with the hostname returned by the previous step.

{
    "channel":
    [
        {
            "name": "HAQMCHL",
            "clientConnection":
            {
                "connection":
                [
                    {
                        "host": "<host from previous step>",
                        "port": 443
                    }
                ],
                "queueManager": "HAEXAMPLE"
            },
            "transmissionSecurity":
            {
              "cipherSpecification": "ECDHE_RSA_AES_128_CBC_SHA256"
            },
            "type": "clientConnection"
        }
   ]
}

Export environment variables

Export the following environment variables, in the manner appropriate for your operating system. These variables will be read by amqsputc and amqsgetc.

Update the path to the files on your system:

export MQCCDTURL='<full_path_to_file>/CCDT.JSON'
export MQSSLKEYR='<full_path_to_file>/clientkey'

Put messages to the queue

Run the following command:

amqsputc EXAMPLE.QUEUE HAEXAMPLE

If connection to the queue manager is successful, the following response is output:

target queue is EXAMPLE.QUEUE

Put several messages to the queue, by entering some text then pressing Enter each time.

To finish, press Enter twice.

Retrieve the messages from the queue

Run the following command:

amqsgetc EXAMPLE.QUEUE HAEXAMPLE

The messages you added in the previous step have been consumed, and are output.

After a few seconds, the command exits.

Force the active pod to fail

To validate the automatic recovery of the queue manager, simulate a pod failure:

View the active and standby pods

Run the following command:

oc get pods --selector app.kubernetes.io/instance=nativeha-example

Note that, in the READY field, the active pod returns the value 1/1, whereas the replica pods return the value 0/1.

Delete the active pod

Run the following command, specifying the full name of the active pod:

oc delete pod nativeha-example-ibm-mq-<value>

View the pod status again

Run the following command:

oc get pods --selector app.kubernetes.io/instance=nativeha-example

View the queue manager status

Run the following command, specifying the full name of one of the other pods:

oc exec -t Pod -- dspmq -o nativeha -x -m HAEXAMPLE

You should see the status shows that the active instance has changed, for example:

QMNAME(HAEXAMPLE) ROLE(Active) INSTANCE(inst1) INSYNC(Yes) QUORUM(3/3)
INSTANCE(inst1) ROLE(Active) REPLADDR(9.20.123.45) CONNACTV(Yes) INSYNC(Yes) BACKLOG(0) CONNINST(Yes) ALTDATE(2022-01-12) ALTTIME(12.03.44)
INSTANCE(inst2) ROLE(Replica) REPLADDR(9.20.123.46) CONNACTV(Yes) INSYNC(Yes) BACKLOG(0) CONNINST(Yes) ALTDATE(2022-01-12) ALTTIME(12.03.44)
INSTANCE(inst3) ROLE(Replica) REPLADDR(9.20.123.47) CONNACTV(Yes) INSYNC(Yes) BACKLOG(0) CONNINST(Yes) ALTDATE(2022-01-12) ALTTIME(12.03.44)

Put and get messages again

After the standby pod becomes the active pod (that is, after the READY field value becomes 1/1), use the following commands again, as previously described, to put messages to the queue manager then retrieve messages from the queue manager:

amqsputc EXAMPLE.QUEUE HAEXAMPLE

amqsgetc EXAMPLE.QUEUE HAEXAMPLE

Congratulations, you have successfully deployed a native HA queue manager, and shown that it can automatically recover from a pod failure.

Additional information

Note 1: Creating a route

If you are using an IBM MQ client or toolkit earlier than IBM MQ 9.2.1, you need to create an Route.

To create the route, ensure you are in the namespace you created earlier (see Before you begin), then enter the following YAML in the Red Hat OpenShift Container Platform web console, or using the command line:


apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: example-mi-route
spec:
  host: hamqchl.chl.mq.ibm.com
  to:
    kind: Service
    name: nativeha-example-ibm-mq
  port:
    targetPort: 1414
  tls:
    termination: passthrough

Note that the Red Hat OpenShift Container Platform Router uses SNI for routing requests to the IBM MQ queue manager. If you change the channel name specified in the config map containing MQSC commands, you must also change the host field here, and in the CCDT.JSON file that specifies the queue manager details. For more information, see Configuring a Route to connect to a queue manager from outside a Red Hat OpenShift cluster.

Note 2: Using a non-default storage class

This example expects a default storage class to have been configured in Red Hat OpenShift Container Platform, therefore no storage information is required in the queue manager custom resource YAML. If you do not have a storage class configured as default, or you want to use a different storage class, add

defaultClass:
<storage_class_name>

under spec.queueManager.storage.

The storage class name must exactly match the name of a storage class that already exists. That is, it must match the name returned by the command

oc get
storageclass

. It must also support ReadWriteMany. For more information, see Storage considerations for IBM MQ Operator.