mmnetverify command

Verifies network configuration and operation in a cluster.

Synopsis

mmnetverify [Operation[ Operation...]] [-N {Node[,Node...] | all}]
            [--target-nodes {Node[,Node...] | all}]
            [--configuration-file File] [--log-file File]
            [--verbose] [--min-bandwidth Number]

Availability

Available on all IBM Spectrum Scale™ editions.

Description

With the mmnetverify command, you can verify the network configuration and operation of a group of nodes before you organize them into an IBM Spectrum Scale cluster. You can also run the command to analyze network problems after you create a cluster.

If you have not created an IBM Spectrum Scale cluster yet, you must run the command with a configuration file. See the --configuration-file option in the Parameters section.

The command uses the concepts of local nodes and target nodes. A local node is a node from which a network test is run. The command can be started from one node and run from several separate local nodes. A target node is a node against which a test is run.

The command has the following requirements:

IBM Spectrum Scale must be installed on all the nodes that are involved in the test, including both local nodes and target nodes.
Each node must be able to issue remote shell commands to all nodes, including itself, without a password. (This requirement is tested in the shell check.)

The following table lists the types of output messages and where they are sent. Messages are not added to a log file unless you specify one on the command line:

Table 1. Information and error messages
Message type	Printed to console	Added to log file, if one is specified
Information messages	Yes (`stdout`)	Yes
Verbose information messages	Yes (`stdout`) If --verbose is specified on the command line	Yes
Error messages	Yes (`stderr`)	Yes

Parameters

Operation[ Operation...]

Specifies one or more operations, separated by blanks, to be verified against the target nodes. The operations are described in Table 3. Shortcut terms are described in Table 2.

Start of change If you do not specify any operations, the command does all the operations except flood-node, flood-cluster, and data-large against the target nodes. End of change

-N {Node[,Node...] | all}

Specifies a list of nodes on which to run the command. If you specify more than one node, the command is run on each specified node in turn, starting with the node on which you run the command. Each instance of the command tests all the target nodes. Start of change

If you do not specify this parameter, the command runs the operations from the node where you enter the command). End of change

Node[,Node...]: Specifies a list of nodes in the local cluster.
all: Specifies all the nodes in the local cluster.

--target-nodes {Node[,Node...] | all}]

Specifies a list of nodes that are the targets of the testing. Start of change

If you do not specify this parameter, the command runs the operations against all the nodes in the cluster. End of change

Node[,Node...]: Specifies a list of nodes in the local cluster.
all: Specifies all the nodes in the local cluster.

[--configuration-file File]

Specifies the path of a configuration file. You must use a configuration file if you have not created an IBM Spectrum Scale cluster yet. You can also specify a configuration file if you have created a cluster but you do not want the command to run with the IBM Spectrum Scale cluster configuration values. Only the node parameter is required. The other parameters revert to their default values if they are not specified. The format of the file is as follows:

node Node [AdminName]
rshPath Path
rcpPath Path
tscTcpPort Port
mmsdrservPortPort Port
tscCmdPortRange Min-Max

where:

node Node [AdminName]: Specifies a node name, followed optionally by the node's admin name. If you do not specify an admin name, the command uses the node name as the admin name.
You can have multiple node parameters. Add a node parameter for each node that you want to be included in the testing, either as a local node or as a target node. You must include the node from which you are running the command.
rshPath Path: Optional. Specifies the path of the remote shell command to be used. The default value is /usr/bin/ssh. Specify this parameter only if you want to use a different remote shell command.
rcpPath Path: Optional. Specifies the path of the remote file copy command to be used. The default value is /usr/bin/scp. Specify this parameter only if you want to use a different remote copy command.
tscTcpPort Port: Optional. Specifies the TCP port number to be used by the local GPFS™ daemon when it contacts a remote cluster. The default value is 1191. Specify this value only if you want to use a different port.
mmsdrservPortPort Port: Optional. Specifies the TCP port number to be used by the mmsdrserv service to provide access to configuration data to the rest of the nodes in the cluster. The default value is the value that is stored in mmfsdPort.
tscCmdPortRange Min-Max: Optional. Specifies the range of port numbers to be used for extra TCP/IP sockets that are needed by some commands. If this parameter is not specified, then additional sockets are dynamically assigned by the operating system. For more information, see GPFS port usageGPFS port usage in IBM Spectrum Scale: Administration Guide.

[--log-file File]

Specifies the path of a file to contain the output messages from the network checks. If you do not specify this parameter, messages are displayed only on the console. See Table 1.

--verbose

Causes the command to generate verbose output messages. See Table 1.

[--min-bandwidth Number]

Specifies the minimum acceptable bandwidth for the data bandwidth check.

The network checks

The following table lists the shortcut terms that you can specify for the network checks that are listed in Table 3:

Note:

These network checks might cause entries to the mmfs log that say that connections are being killed. These entries are expected and do not indicate problems in your system. End of change

Table 2. Shortcut terms for network checks
Shortcut	Checks performed
local	interface
connectivity	resolution, ping, shell, and copy
port	daemon-port, sdserv-port, and tsccmd-port
data	data-small, data-medium, data-large, and data-bandwidth
bandwidth	bandwidth-node and bandwidth-cluster
flood	flood-node and flood-cluster
all	All checks except flood-node and flood-cluster

The following table lists the parameters that you can specify for network checks. Separate these parameters with a blank on the command line. For example, the following command runs the interface and copy checks on the local node against all the nodes in the cluster:

mmnetverify interface copy

Table 3. Network checks
Command-line option	Test description	Test items
interface	Network interface configuration	The local node's daemon and admin network addresses are enabled.
resolution	Hostname resolution	Checks: The target node's name and daemon node name can be resolved on the local node and they resolve to the same address. The target node's IP address resolves to either the node name or the daemon node name.
ping	Network connectivity via ping	The local node can ping the target node with its name, daemon node name, rel_hostname, admin_shortname, and IP address entries in the mmsdrfs file.
shell	Remote shell command	Checks: The local node can issue remote shell commands to the target node's admin interface without requiring a password. The target node's daemon and admin names refer to the same node.
copy	Remote copy	The local node can issue a remote copy command to the target node's admin interface without requiring a password.
time	Date and time	The time and date on the local node and target node do not differ by a wide margin.
daemon-port	GPFS daemon connectivity	The target node can establish a TCP connection to the local node on the mmfsd daemon port of the local node: The target node uses the port that is specified in the cluster configuration property tscTcpPort. The default value is 1191. If mmfsd and mmsdrserv are not running on the local node, the command starts an echo server on the daemon port of the local node for this test.
sdrserv-port	GetObject daemon connectivity	The target node can establish a TCP connection directed to the local node on the port that is specified in the cluster configuration property mmsdrservPort: The default value of this port is the value that is specified in the cluster configuration property tscTcpPort. If mmfsd and mmsdrserv are not running on the local node, the command starts an echo server on the daemon port of the local node for this test.
tsccmd-port	TS-command connectivity	The target node can establish a TCP connection directed to the local node on a port in the range that is specified in the tscCmdPortRange property: The command starts an echo server on the local node for this test. If the tscCmdPortRange property is set, then the echo server listens on a port in the specified range. If not, then the echo server listens on an ephemeral port that is provided by the operating system.
data-small	Small data exchange	The target node can establish a TCP connection to the local node and exchange a series of small-sized data messages without network errors: The command starts an echo server on the local node for this test. If the tscCmdPortRange property is set, then the echo server listens on a port in the specified range. If not, then the echo server listens on an ephemeral port that is provided by the operating system.
data-medium	Medium data exchange	The target node can establish a TCP connection to the local node and exchange a series of medium-sized data messages without network errors: The command starts an echo server on the local node for this test. If the tscCmdPortRange property is set, then the echo server listens on a port in the specified range. If not, then the echo server listens on an ephemeral port that is provided by the operating system.
data-large	Large data exchange	The target node can establish a TCP connection to the local node and exchange a series of large-sized data messages without network errors: The command starts an echo server on the local node for this test. If the tscCmdPortRange property is set, then the echo server listens on a port in the specified range. If not, then the echo server listens on an ephemeral port that is provided by the operating system.
bandwidth-node	Network bandwidth one-to-one	The target node can establish a TCP connection to the local node and send a large amount of data with adequate bandwidth: Bandwidth is measured on the target node. If the min-bandwidth parameter was specified on the command line, the command verifies that the actual bandwidth exceeds the specified minimum bandwidth.
bandwidth-cluster	Network bandwidth many-to-one	All target nodes can establish a TCP connection to the local node and send a large amount of data in parallel: The bandwidth is measured on each target node. If the min-bandwidth parameter was specified on the command line, the command verifies that the actual bandwidth exceeds the specified minimum bandwidth. None of the target nodes has a significantly lower bandwidth than the other target nodes.
flood-node	Flood one-to-one	When the local node is flooded with datagrams, the target node can successfully send datagrams to the local node: The target node tries to flood the local node with datagrams. The command records packet loss. The command verifies that some of the datagrams were received.
flood-cluster	Flood many-to-one	When the local node is flooded with datagrams from all the target nodes in parallel, each target node can successfully send datagrams to the local node: The command records packet loss for each target node. Check that each target node received some datagrams. Check that none of the target nodes has a packet loss significantly higher than the other nodes.

Exit status

0: The command completed successfully and all tests completed successfully.

1: The command encountered problems with options or with running tests.

2: The command completed successfully, but one or more tests was unsuccessful.

Security

You must have root authority to run the mmnetverify command.

The node on which you enter the command must be able to execute remote shell commands on any other administration node in the cluster. It must be able to do so without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.

Examples

The following command runs all checks, except flood-node, flood-cluster, and data-large, from the node where you enter the command against all the nodes in the cluster:
```
mmnetverify
```
The following command runs connectivity checks from the node where you enter the command against all the nodes in the cluster:
```
mmnetverify connectivity
```
The following command runs connectivity checks from nodes c49f04n11 and c49f04n12 against all the nodes in the cluster:
```
mmnetverify connectivity -N c49f04n11,c49f04n12
```
The following command runs all checks, except flood-node and flood-cluster, from the node where you enter the command against nodes c49f04n11 and c49f04n12:
```
mmnetverify all --target-nodes c49f04n11,c49f04n12
```
The following command runs network port checks on nodes c49f04n07 and c49f04n08, each checking against nodes c49f04n11 and c49f04n12:
```
mmnetverify port -N c49f04n07,c49f04n08 --target-nodes c49f04n11,c49f04n12
```

Location

/usr/lpp/mmfs/bin