Operational test 2: Checking the status of the management domain and the peer domain

Use the following procedure to check the status of the management domain and peer domain.

On any node, execute the rmcdomainstatus command.
```
# /opt/rsct/bin/rmcdomainstatus -s ctrmc
```
If there is no output, the node is not a member of a peer domain or a management domain. If the node is a member of a peer domain, a list similar to the following should be displayed.
```
Peer Domain Status
   I A  0x09898b3065189db6  0002  c174tr6.ppd.pok.ibm.com
   S S  0x07e7287425d0becd  0001  c174tr5.ppd.pok.ibm.com
```
If the node is an MCP, a list similar to the following should be displayed.
```
Management Domain Status: Managed Nodes
   I a  0xbf1fb04e5b7d0b06  0001  C174tr4 !/+
   I a  0x3a75dd6c235c428e  0002  C174tr3  masMMtest/+ (1)
   I A  0x07e7287425d0becd  0003  C174tr5  masfive/+ (2)
   I A  0x09898b3065189db6  0004  C174tr6  masfive/+(2)
```
If the node is a Managed Node, a list similar to the following should be displayed.
```
Management Domain Status: Management Control Points
   I A  0xef889c809d9617c7 0001  9.57.24.139
```
A node may be a member of a peer domain, the MCP of a management domain, and a managed node in a different management domain simultaneously. In this case, the output of the rmcdomainstatus would contain one or more of the preceding lists.

Each line of output represents the status of a cluster node, relative to the node upon which the command is executed. The first token of the node status line is either S, I, i, O, X, or Z.
S

Indicates the line is the status of a peer node itself.

I

Indicates, in a management domain, that the node is Up as determined by the RMC heartbeat mechanism. In a peer domain, it indicates that the RMC daemon on the specified node is a member of the rmc_peers Group Services group and the node is online in the peer domain.

i

Indicates, in a management domain, the node is Pending Up. Communication has been established, but the initial handshake between two RMC daemons has not been completed. If this indicator is present upon successive executions of the rmcdomainstatus command, then message authentication is most likely failing. See Troubleshooting the cluster security services to validate proper security services configuration between the specified node and the node upon which the command is executed.

O

Indicates, in a management domain, that the node is Down, as determined by the RMC heartbeat mechanism. In a peer domain, it indicates the RMC daemon on the specified node is no longer a member of the rmc_peers Group Services group. The most likely cause is that the node is down, as determined by the Topology Services component of RSCT. It may also indicate that the RMC daemon on the specified node is not functioning.

X
Indicates, in a management domain, that a communication problem has been discovered, and the RMC daemon has suspended communications with the RMC daemon that is on the specified node. This is typically the result of a configuration problem in the network, such that small heartbeat packets can be exchanged between the RMC daemon and the RMC daemon that is on the specified node, but larger data packets cannot. This is usually the result of a difference in MTU sizes in the network adapters of the nodes. To recover, run the following command after correcting any communication problems.
```
refresh -s ctrmc
```
If the rmcdomainstatus output still indicates X, contact the IBM® Support Center.
Z

Indicates that the RMC daemon has suspended communication with the RMC daemon that is on the specified node because the up/down state of the node is changing too rapidly. This is typically the result of more than one node having the same node ID.
If the rmcdomainstatus output still indicates Z, contact the IBM Support Center.
The second token of the node status line is either S, A, R, a, or r.

S

Indicates the line is the status of a peer node itself.

A

Indicates that there are no messages queued to the specified node.

R

Indicates that messages are queued to the specified node. If this indication persists upon repeated executions of the rmcdomainstatus command over several minutes, and your network is not operating under a heavy load, contact the IBM Support Center.

a

Has the same meaning as A, but the specified node is executing a version of the RMC daemon that is at a lower code level than the local RMC daemon.

r

Has the same meaning as R, but the specified node is executing a version of the RMC daemon that is at a lower code level than the local RMC daemon.

The third token of the status line is the ID of the specified node. The node ID is a 64-bit number that is created when RSCT is installed. It is derived using a True Random Number Generator and is used to uniquely identify a node to the RMC subsystem. The node ID is maintained in the /var/ct/cfg/ct_node_id file. A backup copy is maintained in the /etc/ct_node_id file. If this value is not unique among all systems where RSCT is installed, contact the IBM Support Center.

The fourth token of the status line is an internal node number that is used by the RMC daemon.

If the list is a list of Peer Nodes or Managed Nodes, the fifth token is the name of the node as known to the RMC subsystem. If the list is a list of MCPs, the fifth token is the first configured IP address of the specified MCP.
If the list is a list of Managed Nodes, the sixth token has the form:
```
<PD_name>/<PD_status> (n)
```
where PD_name is the name of the Peer Domain of which the Managed Node is an online member. Otherwise it is the ! character and (n) is not present. PD_status is the + character if Peer Domain status has been received from the Managed Node. Otherwise it is the - character. n is the number of online nodes in the peer domain of which the specified Managed Node is a member.
If there is no status line for an expected node, contact the IBM Support Center.

If the following command is executed on a MCP or Managed Node,

# /opt/rsct/bin/rmcdomainstatus -s ctrmc -a ip

the fifth token in both the Managed Node list and the MCP list is the first configured IP address of the specified node. There is a subsequent line for each additional configured IP address for the specified node, consisting of only the IP address:

Management Domain Status: Managed Nodes
  I A  0x6dfa8e3206ff26c7  0003  9.114.113.233
  I A  0xe0870ff61109de87  0005  9.114.113.179
  I A  0xd7d2795c2516ecf8  0004  9.114.113.201
  I A  0x794697e35a3ab4c3  0006  9.114.113.200
  I A  0x7fb34d5799e489ad  0002  9.114.113.189
                                 192.160.2.1
  I A  0xd9b9a059686b4979  0001  9.114.113.188
                                 192.160.2.6
  Management Domain Status: Management Control Points
  I A  0xb2ae35236d8585bb  0001  192.160.2.8
                                 9.114.113.70
  I A  0x718336df238c7968  0002  192.160.2.10
                                 9.114.113.67