Viewing DR/HA RDQM and HA group status

You can view the HA status and DR role of DR/HA replicated data queue managers (RDQMs).

About this task

You use the rdqmstatus command to view the status of individual RDQMs or get an overview of the status of all the RDQMs known to the HA group.

[MQ 9.2.3 Jul 2021] The summary status for a node also displays information about the DRBD kernel module that RDQM relies upon. When you upgrade RDQM, it is important to ensure that the correct version of the DRBD kernel module is installed for the version of RHEL kernel running on the system. The status displays the version of the OS kernel, the kernel version that the DRBD module was built for, the DRBD version, and the DRBD kernel module loaded status.

You must be a user in the mqm and haclient groups to run the rdqmstatus command. You can run the command on any of the nodes in either of the HA groups.

Procedure

To view the summary status of a node and the RDQMs that are part of the HA configuration:
```
rdqmstatus
```
The identity of the node that you ran the command on and the status of the RDQMs in the HA configuration is displayed, plus their current DR role, for example:
```
Node:                                   main-alice
OS kernel version:                      3.10.0-1160.15.2
DRBD OS kernel version:                 3.10.0-1160
DRBD version:                           9.1.1
DRBD kernel module status:              Loaded
 
Queue manager name:                     RDQM1
Queue manager status:                   Running elsewhere
HA current location:                    main-charlie
HA preferred location:                  main-charlie
HA blocked location:                    None 
 
Queue manager name:                     RDQM9
Queue manager status:                   Running elsewhere
HA current location:                    main-bob
HA preferred location:                  main-bob
HA blocked location:                    None
DR role:                                Primary
 
Queue manager name:                     RDQM7
Queue manager status:                   Running
HA current location:                    This node
HA preferred location:                  This node
HA blocked location:                    None
DR role:                                Primary
```
In this example, RDQM7 and RDQM8 are both DR/HA RDQMs, while RDQM1 is an HA RDQM, which is not configured to be able to switch over to a disaster recovery site.

The DRBD kernel module status is one of the following values:

Loaded

Indicates that the DRBD module has been loaded.

Partially loaded

Can occur when the DRBD module has been loaded, but does not function correctly due to a mismatch.

Not loaded

The DRBD module is not loaded. This can be displayed on a newly installed configuration, when no RDQM queue managers have yet been created.

Not installed

Indicates that either the DRBD module is not installed. or that IBM® MQ was unable to determine the OS kernel version of the DRBD module.

Previously installed version still loaded

This status can arise if a new DRBD module is installed while the existing DRBD module is running (that is, an RDQM queue manager is running). The newly installed module is reported in the status, but is not the module that is actually running.
To view the status of a particular queue manager on all the nodes in the HA group, enter the following command:
```
rdqmstatus -m qmname
```
where qmname is the name of the RDQM you want to view the status for. The status of the RDQM on the current node is displayed, followed by a summary of the status of the other two nodes from the perspective of the current node.
To view the status of a particular queue manager on all the nodes in the HA group, including details of any failed resource actions, enter the following command:
```
rdqmstatus -m qmname -a
```
where qmname is the name of the RDQM you want to view the status for. The status of the RDQM on the current node is displayed, followed by a summary of the status of the other two nodes from the perspective of the current node. This is followed by details about any failed resource actions associated with the RDQM.

The following table summarizes the information about the current node that can be returned by the rdqmstatus -m qmname command for an RDQM.

Table 1. Current node status
Status attribute	Possible values	When displayed
Node name	`nodename`	Always displayed
Queue manager status	state of queue manager (one of the states that are valid for dspmq command)	Always displayed
CPU	`n.nn`%	Only shown when RDQM is running on this node
Memory	`nnn`MB used	Only shown when RDQM is running on this node
Queue manager file system	`nnn`MB used, `y`.`y`GB allocated [`z`%]	Only shown when RDQM is running on this node
HA role	Primary Secondary Unknown	Always displayed
HA status	All nodes in standby This node in standby Remote nodes in standby Mixed	All nodes in standby Current node in standby Both remote nodes in standby Different status for each remote node
HA control	Enabled Disabled Unknown	Always displayed. Shows whether RDQM is under Pacemaker control
HA preferred location	None This node Unknown `nodename`	Always displayed
HA blocked location	None - The queue manager is not blocked from running on any nodes This node - The queue manager is blocked from running on the current node due to one or more failed resource actions `nodename` - The queue manager is blocked from running on `nodename` due to one or more failed resource actions `nodename1`, `nodename2` - The queue manager is blocked from running on `nodename1` and `nodename2` due to one or more failed resource actions All nodes - The queue manager is blocked from running on all nodes due to one or more failed resource actions	Always displayed
HA floating IP interface	`Interface_name`	Always displayed
HA floating IP address	`IPV4_address`	Always displayed
DR role	Primary Secondary Secondary pending Unknown	Always displayed
DR status	Normal Synchronization in progress Partitioned Remote system unavailable Inconsistent Reverting to snapshot Remote system not configured Negotiation failed	Everything is OK. Synchronization is in progress. The user has started the queue manager on each node while the DR replication network was unavailable. The connection to the other node has been lost. A synchronization was in progress but was interrupted. The user has chosen to revert to the snapshot that was taken when the queue manager entered the Inconsistent state. The primary has been configured but the secondary has not. The initial negotiation between the primary and secondary nodes failed. This can be caused by incompatible replication types or if the secondary node is configured with a smaller filesystem size.
DR status (on HA secondary node)	See `HA_Primary_Node`	Displayed on the HA secondary nodes as the DR status is only known on the HA primary node.
DR port	The TCP/IP port used to replicate the data for this queue manager.	Always displayed.
DR local IP address	The local IP address this queue manager will use for DR replication	Always displayed.
DR remote IP address list	The remote IP addresses this queue manager will use for DR replication. A comma-separated list of three IP addresses.	Always displayed.
DR current remote IP address	The current remote IP this queue manager is connected to for DR replication.	For an HA primary with an active DR connection.
DR current remote IP address (on HA secondary node)	See `HA_Primary_Node`	Displayed on an HA Secondary node as the DR connection is only on the HA Primary node
DR out of sync data	`x`KB	Displayed when remote node unavailable or inconsistent.
DR synchronization progress	`y`%	Displayed when a synchronization is in progress.
DR estimated time to completion	`YYYY-MM-DD HH:MM:SS`	Displayed when a synchronization is in progress.
Snapshot reversion progress	`y`%	Displayed when DR status is "Reverting to snapshot"
DR last in sync	`YYYY-MM-DD HH:MM:SS`	Displayed when DR data is out of sync (after initial synchronization). Gives the time and date when the data was last in sync.

Example

These examples illustrate the command rdqmstatus -m qm1 run on various nodes of the following DR/HA configuration:

Example of normal status on a node that is the DR primary and HA primary:

Node:                                   main-alice
Queue manager status:                   Running
CPU:                                    0.00%
Memory:                                 123MB
Queue manager file system:              51MB used, 1.0GB allocated [5%]
HA role:                                Primary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    This node
HA preferred location:                  This node
HA blocked location:                    None 
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Primary
DR status:                              Normal
DR port:                                3000
DR local IP address:                    192.168.1.1
DR remote IP address list:              192.168.2.1,192.168.2.2,192.168.2.3
DR current remote IP address:           192.168.2.1

Node:                                   main-bob 
HA status:                              Normal

Node:                                   main-charlie 
HA status:                              Normal

Example of normal status on a node that is the DR primary and an HA secondary:

Node:                                   main-bob
Queue manager status:                   Running elsewhere
HA role:                                Secondary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    main-alice
HA preferred location:                  main-alice
HA blocked location:                    None
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Primary
DR status:                              See main-alice
DR port:                                3000
DR local IP address:                    192.168.1.2
DR remote IP address list:              192.168.2.1,192.168.2.2,192.168.2.3
DR current remote IP address:           See main-alice

Node:                                   main-alice
HA status:                              Normal

Node:                                   main-charlie  
HA status:                              Normal

Example of normal status on a node that is the DR secondary and an HA primary:

Node:                                   dr-alice
Queue manager status:                   Ended immediately
HA role:                                Primary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    This node
HA preferred location:                  This node 
HA blocked location:                    None
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Secondary
DR status:                              Normal
DR port:                                3000
DR local IP address:                    192.168.2.1
DR remote IP address list:              192.168.1.1,192.168.1.2,192.168.1.3
DR current remote IP address:           192.168.1.1

Node:                                   dr-bob 
HA status:                              Normal

Node:                                   dr-charlie 
HA status:                              Normal

Example of normal status on a node that is the DR secondary and an HA secondary:

Node:                                   dr-bob
Queue manager status:                   Ended immediately
HA role:                                Secondary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    dr-alice
HA preferred location:                  dr-alice
HA blocked location:                    None
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Secondary
DR status:                              See dr-alice
DR port:                                3000
DR local IP address:                    192.168.2.2 
DR remote IP address list:              192.168.1.1,192.168.1.2,192.168.1.3
DR current remote IP address:           See dr-alice

Node:                                   dr-alice
HA status:                              Normal

Node:                                   dr-charlie 
HA status:                              Normal

Example of DR synchronization in progress on a node that is a DR primary and HA primary:

Node:                                   main-alice
Queue manager status:                   Running
CPU:                                    0.00%
Memory:                                 123MB
Queue manager file system:              51MB used, 1.0GB allocated [5%]
HA role:                                Primary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    This node
HA preferred location:                  This node 
HA blocked location:                    None
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Primary
DR status:                              Normal
DR port:                                3000
DR local IP address:                    192.168.1.1
DR remote IP address list:              192.168.2.1,192.168.2.2,192.168.2.3
DR current remote IP address:           192.168.2.1
DR synchronization progress:            11.0%
DR estimated time to completion:        2018-09-06 14:55:05

Node:                                   main-bob 
HA status:                              Normal

Node:                                   main-charlie 
HA status:                              Normal

Example of DR partitioned on a node that is a DR primary and HA primary:

Node:                                   main-alice
Queue manager status:                   Running
CPU:                                    0.00%
Memory:                                 123MB
Queue manager file system:              51MB used, 1.0GB allocated [5%]
HA role:                                Primary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    This node
HA preferred location:                  This node 
HA blocked location:                    None
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Primary
DR status:                              Partitioned
DR port:                                3000
DR local IP address:                    192.168.1.1
DR remote IP address list:              192.168.2.1,192.168.2.2,192.168.2.3
DR current remote IP address:           192.168.2.1
DR out of sync data:                    372KB

Node:                                   main-bob 
HA status:                              Normal

Node:                                   main-charlie 
HA status:                              Normal

Example of DR out of sync on a node that is a DR primary and HA primary:

Node:                                   main-alice
Queue manager status:                   Running
CPU:                                    0.00%
Memory:                                 123MB
Queue manager file system:              51MB used, 1.0GB allocated [5%]
HA role:                                Primary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    This node
HA preferred location:                  This node 
HA blocked location:                    None
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Primary
DR status:                              Remote unavailable
DR port:                                3000
DR local IP address:                    192.168.1.1
DR remote IP address list:              192.168.2.1,192.168.2.2,192.168.2.3
DR current remote IP address:           Unknown
DR out of sync data:                    372KB
DR last in sync:                        2020-02-02 20:22:02

Node:                                   main-bob 
HA status:                              Normal

Node:                                   main-charlie 
HA status:                              Normal

Example of HA out of sync on a node that is a DR primary and HA primary:

Node:                                   main-alice
Queue manager status:                   Running
CPU:                                    0.00%
Memory:                                 123MB
Queue manager file system:              51MB used, 1.0GB allocated [5%]
HA role:                                Primary
HA status:                              Normal 
HA control:                             Enabled 
HA current location:                    This node
HA preferred location:                  This node 
HA blocked location:                    None
HA floating IP interface:               None
HA floating IP address:                 None
DR role:                                Primary
DR status:                              Normal
DR port:                                3000
DR local IP address:                    192.168.1.1
DR remote IP address list:              192.168.2.1,192.168.2.2,192.168.2.3
DR current remote IP address:           192.168.2.1

Node:                                   main-bob 
HA status:                              Inconsistent
HA out of sync data:                    15932KB
HA last in sync:                        2020-02-02 20:22:02

Node:                                   main-charlie 
HA status:                              Normal

Example of a summary status showing a mismatch between the OS kernel version (RHEL 7.9) and the DRBD kernel module (targeted for RHEL 7.8). Even though the status reports that DRBD kernel module is loaded and queue manager is running, you should update DRBD kernel module with the version targeted for the running OS kernel in this situation.

Node:                                   main-alice
OS kernel version:                      3.10.0-1160.15.2
DRBD OS kernel version:                 3.10.0-1127
DRBD version:                           9.1.1
DRBD kernel module status:              Loaded
 
Queue manager name:                     QM1
Queue manager status:                   Running
HA current location:                    This node
HA preferred location:                  This node
HA blocked location:                    None
DR role:                                Primary

Example of a summary status showing a mismatch between the OS kernel version (RHEL 7.9) and the DRBD kernel module (targeted for RHEL 7.6). In this example the version mismatch is more severe and DRBD kernel module fails to load successfully. QM1 is an HA/DR queue manager and moves to another node, its HA status is unknown and its DR status is unknown. To resolve this failure the DRBD kernel module must be updated with the version target for the running OS kernel.

Node:                                   main-alice
OS kernel version:                      3.10.0-1160.15.2
DRBD OS kernel version:                 3.10.0-957
DRBD version:                           9.1.2+ptf.3
DRBD kernel module status:              Partially loaded

Queue manager name:                     QM1
Queue manager status:                   Running elsewhere
HA status:                              Unknown
HA current location:                    main-bob
HA preferred location:                  This node
HA blocked location:                    None
DR role:                                Primary
DR status:                              Unknown