[OpenShift Container Platform][Kubernetes][IBM Cloud Pak for Integration]

High availability for IBM MQ in containers

There are three choices for high availability with IBM® MQ Operator: Native HA queue manager (which has an active replica and two standby replicas), Multi-instance queue manager (which is an active-standby pair, using a shared, networked file system), or Single resilient queue manager (which offers a simple approach for HA using networked storage). The latter two rely on the file system to ensure the availability of the recoverable data, however Native HA does not. Therefore, when not using Native HA, the availability of the file system is critical to queue manager availability. Where data recovery is important the file system should ensure redundancy through replication.

You should consider separately message and service availability. With IBM MQ for Multiplatforms, a message is stored on exactly one queue manager. So if that queue manager becomes unavailable, you temporarily lose access to the messages it holds. To achieve high message availability, you need to be able to recover a queue manager as quickly as possible. You can achieve service availability by having multiple instances of queues for client applications to use, for example by using an IBM MQ uniform cluster.

A queue manager can be thought of in two parts: the data stored on disk, and the running processes that allow access to the data. Any queue manager can be moved to a different Kubernetes Node, as long as it keeps the same data (provided by Kubernetes Persistent Volumes) and is still addressable across the network by client applications. In Kubernetes, a Service is used to provide a consistent network identity.

IBM MQ relies on the availability of the data on the persistent volumes. Therefore, the availability of the storage providing the persistent volumes is critical to queue manager availability, because IBM MQ cannot be more available than the storage it is using. If you want to tolerate an outage of an entire availability zone, you need to use a volume provider that replicates disk writes to another zone.

[MQ 9.2.3 Jul 2021][IBM Cloud Pak for Integration]

Native HA queue manager

Native HA queue managers are available from IBM Cloud Pak® for Integration 2021.2.1, using IBM MQ Operator 1.6 or higher, with IBM MQ 9.2.3 or higher.

Native HA queue managers involve an active and two replica Kubernetes Pods, which run as part of a Kubernetes StatefulSet with exactly three replicas each with their own set of Kubernetes Persistent Volumes. The IBM MQ requirements for shared file systems also apply when using a native HA queue manager (except for lease-based locking), but you do not need to use a shared file system. You can use block storage, with a suitable file system on top. For example, xfs or ext4. The recovery times for a native HA queue manager are controlled by the following factors:
  1. How long the replica instances take to detect that the active instance has failed. This is configurable.
  2. How long it takes for the Kubernetes Pod readiness probe to detect that the ready container has changed and redirect network traffic. This is configurable.
  3. How long it takes IBM MQ clients to reconnect.

For more information, see Native HA

[UNIX, Linux, Windows, IBM i]

Multi-instance queue manager

Multi-instance queue managers involve an active and a standby Kubernetes Pod, which run as part of a Kubernetes Stateful Set with exactly two replicas and a set of Kubernetes Persistent Volumes. The queue manager transaction logs and data are held on two persistent volumes, using a shared file system.

Multi-instance queue managers require both the active and the standby Pods to have concurrent access to the persistent volume. To configure this, you use Kubernetes Persistent Volumes with access mode set to ReadWriteMany. The volumes must also meet the IBM MQ requirements for shared file systems, because IBM MQ relies on the automatic release of file locks to instigate a queue manager failover. IBM MQ produces a list of tested file systems.

The recovery times for a multi-instance queue manager are controlled by the following factors:

  1. How long it takes after a failure occurs for the shared file system to release the locks originally taken by the active instance.
  2. How long it takes for the standby instance to acquire the locks and then start.
  3. How long it takes for the Kubernetes Pod readiness probe to detect that the ready container has changed and redirect network traffic. This is configurable.
  4. How long it takes for IBM MQ clients to reconnect.
[UNIX, Linux, Windows, IBM i]

Single resilient queue manager

A single resilient queue manager is a single instance of a queue manager running in a single Kubernetes Pod, where Kubernetes monitors the queue manager and replaces the Pod as necessary.

The IBM MQ requirements for shared file systems also apply when using a single resilient queue manager (except for lease-based locking), but you do not need to use a shared file system. You can use block storage, with a suitable file system on top. For example, xfs or ext4.

The recovery times for a single resilient queue manager are controlled by the following factors:

  1. How long it takes for the liveness probe to run, and how many failures it tolerates. This is configurable.
  2. How long the Kubernetes Scheduler takes to re-schedule the failed Pod to a new Node.
  3. How long it takes to download the container image to the new Node. If you use an imagePullPolicy value of IfNotPresent, then the image might already be available on that Node.
  4. How long it takes for the new queue manager instance to start.
  5. How long it takes for the Kubernetes Pod readiness probe to detect that the container is ready. This is configurable.
  6. How long it takes for IBM MQ clients to reconnect.
Important:

Although the single resilient queue manager pattern offers some benefits, you need to understand whether you can reach your availability goals with the limitations around Node failures.

In Kubernetes, a failing Pod is typically recovered quickly; but the failure of an entire Node is handled differently. When using a stateful workload like IBM MQ with a Kubernetes StatefulSet, if a Kubernetes Master Node loses contact with a worker node, it cannot determine if the node has failed or if it has simply lost network connectivity. Therefore Kubernetes takes no action in this case until one of the following events occurs:
  1. The node recovers to a state where the Kubernetes Master Node can communicate with it.
  2. An administrative action is taken to explicitly delete the Pod on the Kubernetes Master Node. This does not necessarily stop the Pod from running, but just deletes it from the Kubernetes store. This administrative action must therefore be taken very carefully.