Viewing RDQM and HA group status

You can view the status of the HA group and of individual replicated data queue managers (RDQMs).

About this task

You use the rdqmstatus command to view the status of individual RDQMs and of the HA group as a whole.

[MQ 9.2.3 Jul 2021] The summary status for a node also displays information about the DRBD kernel module that RDQM relies upon. When you upgrade RDQM, it is important to ensure that the correct version of the DRBD kernel module is installed for the version of RHEL kernel running on the system. The status displays the version of the OS kernel, the kernel version that the DRBD module was built for, the DRBD version, and the DRBD kernel module loaded status.

You must be a user in the mqm and haclient groups to run the rdqmstatus command. You can run the command on any of the three nodes.

Procedure

To view the summary status of a node and the RDQMs that are part of the HA configuration:

rdqmstatus

The identity of the node that you ran the command on, the kernel and DRBD details for that node, and the status of the RDQMs in the HA configuration is displayed, for example:

Node:                                   mqhavm07.exampleco.com
OS kernel version:                      3.10.0-1160.15.2
DRBD OS kernel version:                 3.10.0-1160
DRBD version:                           9.1.1
DRBD kernel module status:              Loaded
 
Queue manager name:                     RDQM8
Queue manager status:                   Running elsewhere
HA current location:                    mqhavm08.exampleco.com
HA preferred location:                  mqhavm08.exampleco.com
HA blocked location:                    None
 
Queue manager name:                     RDQM9
Queue manager status:                   Running elsewhere
HA current location:                    mqhavm09.exampleco.com
HA preferred location:                  mqhavm09.exampleco.com
HA blocked location:                    None
 
Queue manager name:                     RDQM7
Queue manager status:                   Running
HA current location:                    This node
HA preferred location:                  This node
HA blocked location:                    None

The DRBD kernel module status is one of the following values:

Loaded: Indicates that the DRBD module has been loaded.
Partially loaded: Can occur when the DRBD module has been loaded, but does not function correctly due to a mismatch.
Not loaded: The DRBD module is not loaded. This can be displayed on a newly installed configuration, when no RDQM queue managers have yet been created.
Not installed: Indicates that either the DRBD module is not installed. or that IBM® MQ was unable to determine the OS kernel version of the DRBD module.
Previously installed version still loaded: This status can arise if a new DRBD module is installed while the existing DRBD module is running (that is, an RDQM queue manager is running). The newly installed module is reported in the status, but is not the module that is actually running.

To view the status of the three nodes in the HA group, enter the following command:
```
rdqmstatus -n
```
The online or offline status of each node is reported. For example:
```
Node mqha04(mqhavm04.example.com) is online
Node mqha05(mqhavm05.example.com) is offline
Node mqha06(mqhavm06.example.com) is online
```
To view the status of a particular queue manager on all the nodes in the HA group, enter the following command:
```
rdqmstatus -m qmname
```
where qmname is the name of the RDQM you want to view the status for. The status of the RDQM on the current node is displayed, followed by a summary of the status of the other two nodes from the perspective of the current node.
To view the status of a particular queue manager on all the nodes in the HA group, including details of any failed resource actions, enter the following command:
```
rdqmstatus -m qmname -a
```
where qmname is the name of the RDQM you want to view the status for. The status of the RDQM on the current node is displayed, followed by a summary of the status of the other two nodes from the perspective of the current node. This is followed by details about any failed resource actions associated with the RDQM.

The following table summarizes the information about the current node that can be returned by the rdqmstatus -m qmname command for an RDQM.

Table 1. Current node status
Status attribute	Possible values	When displayed
Node name	`nodename`	Always displayed
Queue manager status	Running Running elsewhere Ended Unavailable	Always displayed
CPU	`n.nn`%	Only shown when current node has primary role (that is, the RDQM is running on this node)
Memory	`nnn`MB used, `y.y`GB allocated	Only shown when current node has primary role (that is, the RDQM is running on this node)
Queue manager file system	`nnn`MB used, `y`.`y`GB allocated [`z`%]	Only shown when current node has primary role (that is, the RDQM is running on this node)
HA role	Primary Secondary Unknown	Always displayed
HA status	All nodes in standby This node in standby Remote nodes in standby Mixed `status of remote nodes`	All nodes in standby Current node in standby Both remote nodes in standby Different status for each remote node (see next table for individual status) Same status for both remote nodes (see next table for all values)
HA control	Enabled Disabled Unknown	Always displayed. Shows whether RDQM is under Pacemaker control
HA preferred location	None This node Unknown `nodename`	Always displayed
HA blocked location	None - The queue manager is not blocked from running on any nodes This node - The queue manager is blocked from running on the current node due to one or more failed resource actions `nodename` - The queue manager is blocked from running on `nodename` due to one or more failed resource actions `nodename1`, `nodename2` - The queue manager is blocked from running on `nodename1` and `nodename2` due to one or more failed resource actions All nodes - The queue manager is blocked from running on all nodes due to one or more failed resource actions	Always displayed
HA floating IP interface	`Interface_name`	Always displayed
HA floating IP address	`IPV4_address`	Always displayed

The following table summarizes the information that is returned by the

rdqmstatus -m
qmname

command for the other nodes in the HA group.

Table 2. Other node status
Status attribute	Possible values	When displayed
Node name	`nodename`	Always displayed
HA status	Normal Synchronization in progress Remote unavailable Inconsistent Paused Remote node in standby Unknown	Nodes are in sync with each other Synchronizing with remote node Unable to communicate with remote node Out of sync with remote node, and not synchronizing Replication paused Remote node in standby
HA synchronization in progress	`n`.`n`%	Displayed when synchronization in progress, and command run as `root`
HA estimated synchronization time	`yyyy-mm-dd hh:mm:ss.nnn`	Displayed when synchronization in progress
HA out of sync data	`n`KB	Displayed when remote node unavailable or inconsistent
HA last in sync	`yyyy-mm-dd hh:mm:ss.nnn`	Displayed when HA data is out of sync (after initial synchronization). Gives the time and date when the data was last in sync.

Example

Example of normal status on primary node:

Node:                                   mqhavm07.exampleco.com
Queue manager status:                    Running
CPU:                                     0.00
Memory:                                  123MB
Queue manager file system:               606MB used, 1.0GB allocated [60%]
HA role:                                 Primary
HA status:                               Normal
HA control:                              Enabled
HA current location:                     This node
HA preferred location:                   This node
HA preferred location:                   This node
HA blocked location:                     None
HA floating IP interface:                eth4
HA floating IP address:                  192.0.2.4


Node:                                    mqhavm08.exampleco.com
HA status:                               Normal


Node:                                    mqhavm09.exampleco.com
HA status:                               Normal

Example of normal status on a secondary node:

Node:                                    mqhavm08.exampleco.com
Queue manager status:                    Running elsewhere
HA role:                                 Secondary
HA status:                               Normal
HA control:                              Enabled
HA current location:                     mqhavm07.exampleco.com
HA preferred location:                   mqhavm07.exampleco.com
HA blocked location:                     None
HA floating IP interface:                eth4
HA floating IP address:                  192.0.2.4
 
Node:                                    mqhavm07.exampleco.com
HA status:                               Normal
 
Node:                                    mqhavm09.exampleco.com
HA status:                               Normal

Example of status on primary node when synchronization is in progress:

Node:                                    mqhavm07.exampleco.com
Queue manager status:                    Running
CPU:                                     0.53
Memory:                                  124MB
Queue manager file system:               51MB used, 1.0GB allocated [5%]
HA role:                                 Primary
HA status:                               Synchronization in progress
HA control:                              Enabled
HA current location:                     This node
HA preferred location:                   This node
HA blocked location:                     None
HA floating IP interface:                eth4
HA floating IP address:                  192.0.2.4
 
Node:                                    mqhavm08.exampleco.com
HA status:                               Synchronization in progress
HA synchronization progress:             11.0%
HA estimated time to completion:         2017-09-06 14:55:05
 
Node:                                    mqhavm09.exampleco.com
HA status:                               Synchronization in progress
HA synchronization progress:             11.0%
HA estimated time to completion:         2017-09-06 14:55:06

Example of status on primary node when synchronization has been lost:

Node:                                    mqhavm07.exampleco.com
Queue manager status:                    Running
CPU:                                     0.53
Memory:                                  124MB
Queue manager file system:               51MB used, 1.0GB allocated [5%]
HA role:                                 Primary
HA status:                               Mixed
HA control:                              Enabled
HA current location:                     This node
HA preferred location:                   This node
HA blocked location:                     None
HA floating IP interface:                eth4
HA floating IP address:                  192.0.2.4
 
Node:                                    mqhavm08.exampleco.com
HA status:                               Normal
 
Node:                                    mqhavm09.exampleco.com
HA status:                               Inconsistent
HA out of sync data:                     15932KB
HA last in sync:                         2017-09-06 14:55:06

Example of a primary node showing multiple states:

Node:                                    mqhavm07.exampleco.com
Queue manager status:                    Running
CPU:                                     0.02
Memory:                                  124MB
Queue manager file system:               51MB used, 1.0GB allocated [5%]
HA role:                                 Primary
HA status:                               Mixed
HA control:                              Enabled
HA current location:                     This node
HA preferred location:                   This node
HA blocked location:                     None
HA floating IP interface:                eth4
HA floating IP address:                  192.0.2.4
 
Node:                                    mqhavm08.exampleco.com
HA status:                               Normal
 
Node:                                    mqhavm09.exampleco.com
HA status:                               Inconsistent

Example of a primary node showing failed resource actions:

Node:                                   mqhavm07.exampleco.com
Queue manager status:                   Running
CPU:                                    0.00%
Memory:                                 123MB
Queue manager file system:              606MB used, 1.0GB allocated [60%]
HA role:                                Primary
HA status:                              Normal
HA control:                             Enabled
HA current location:                    This node
HA preferred location:                  mqhavm08.exampleco.com
HA blocked location:                    mqhavm08.exampleco.com
HA floating IP interface:               eth4
HA floating IP address:                 192.0.2.4

Node:                                   mqhavm08.exampleco.com
HA status:                              Normal

Node:                                   mqhavm09.exampleco.com
HA status:                              Normal

Failed resource action:                 Start
Resource type:                          Filesystem
Failure node:                           mqhavm08.exampleco.com
Failure time:                           2017-09-06 12:00:00
Failure reason:                         Couldn't find directory  [/var/mqm/vols/qmname] to use as a mount point
Blocked location:                       mqhavm08.exampleco.com

This status shows that Pacemaker was unable to start the filesystem on node mqhavm08.exampleco.com at 12:00:00. This failed resource action means that the queue manager is blocked from running on mqhavm08.exampleco.com. After the underlying issue that caused the failed resource action is resolved, run the rdqmclean command to clear the failed action so that Pacemaker can retry the action (if required).

Example of a summary status showing a mismatch between the OS kernel version (RHEL 7.9) and the DRBD kernel module (targeted for RHEL 7.8). Even though the status reports that DRBD kernel module is loaded and queue manager is running, you should update DRBD kernel module with the version targeted for the running OS kernel in this situation.

Node:                                   mqhavm07.exampleco.com
OS kernel version:                      3.10.0-1160.15.2
DRBD OS kernel version:                 3.10.0-1127
DRBD version:                           9.1.1
DRBD kernel module status:              Loaded
 
Queue manager name:                     RDQM7
Queue manager status:                   Running
HA current location:                    This node
HA preferred location:                  This node
HA blocked location:                    None

Example of a summary status showing a mismatch between the OS kernel version (RHEL 7.9) and the DRBD kernel module (targeted for RHEL 7.6). In this example the version mismatch is more severe and DRBD kernel module fails to load successfully. As the result the queue manager fails to start on its preferred node and its HA status in Unknown. To resolve this failure the DRBD kernel module must be updated with the version target for the running OS kernel.

Node:                                   mqhavm57.exampleco.com
OS kernel version:                      3.10.0-1160.15.2
DRBD OS kernel version:                 3.10.0-957
DRBD version:                           9.1.2+ptf.3
DRBD kernel module status:              Partially loaded

Queue manager name:                     QM2
Queue manager status:                   Running elsewhere
HA status:                              Unknown
HA current location:                    mqhavm58.exampleco.com
HA preferred location:                  This node
HA blocked location:                    All nodes