Setting up the netmon.cf file on a RoCE network (Linux)
On a remote direct memory access (RDMA) over Converged
Ethernet (RoCE) network, one or more pingable IP addresses must be manually set up in the
netmon.cf configuration file. The
netmon.cf file is required by Reliable Scalable Cluster Technology (RSCT) to
monitor the network and ensure that the interfaces are pingable or not.
The
procedures documented in this page are no longer required as adapter port liveliness test has been
enhanced and automated. Some restrictions apply. Refer to technote#0733765 for
restrictions.
Determine which IP address should be entered into the members'
netmon.cf configuration file.
On the member host, to check the
communication adapter ports and the associated destination IP subnet, run the
route command.
/sbin/route | grep -v link-local
For
example, based on the figure at the end of this topic:
Member 0
[root@host3]# route | grep -v link-local
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
192.168.2.0 * 255.255.255.0 U 0 0 0 eth1
9.26.92.0 * 255.255.254.0 U 0 0 0 eth2
default 9.26.92.1 0.0.0.0 UG 0 0 0 eth2
Member 2
[root@host5]# route | grep -v link-local
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
192.168.2.0 * 255.255.255.0 U 0 0 0 eth1
9.26.92.0 * 255.255.254.0 U 0 0 0 eth2
default 9.26.92.1 0.0.0.0 UG 0 0 0 eth2
The
last column (with column name "Iface") lists the adapters on the current host. Choose the adapter
that corresponds to the target communication adapter port. In this example, "eth0" and "eth1" are the target RoCE adapters. The corresponding IP addresses
in the first column shows the target IP subnet to be used in the next step. In this case, the IP
subnets are "192.168.1.0" and "192.168.2.0".
With
the IP subnet, use the IP interfaces created on the switch 1 and switch 2 that the current host
connects to with the same IP subnet. (The IP interface should already be created as part of the RoCE network configuration steps, for details see Setting up the IP interfaces on the switch on a RoCE network (Linux).)
In this example, assuming the IP interfaces on switch 1 have IP addresses of 192.168.1.2 and
192.168.2.2, and switch 2 have IP addresses of 192.168.1.5 and 192.168.2.5, these entries are added
to the members configuration file/var/ct/cfg/netmon.cf.
token2 - eth0 and eth1 are the RoCE adapter interface names on
the local host
token3 - 192.168.1.2, 192.168.2.5, 192.168.1.5, and 192.168.2.2
are the external pingable IP addresses assigned to the interface created on the switches
The following is an example of what the full configuration file
/var/ct/cfg/netmon.cf looks like for
members:
Determine which IP address should be entered into the cluster caching facilities (CFs)
netmon.cf configuration file.
To check the communication adapter port and the associated destination IP subnet,
enter:
/sbin/route | grep -v link-local
For example:
Host1> $ /sbin/route | grep -v link-local
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.4.0 * 255.255.255.0 U 0 0 0 eth3
192.168.3.0 * 255.255.255.0 U 0 0 0 eth1
192.168.2.0 * 255.255.255.0 U 0 0 0 eth2
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
9.26.92.0 * 255.255.252.0 U 0 0 0 eth2
default rsb-v94-hsrp.to 0.0.0.0 UG 0 0 0 eth2
The
last column (Iface) indicates the adapter interface name. In this case, eth0, eth1, eth2, and eth3
are the only communication adapter port interface on this host.
Four IP subnets are relevant to this host.
All four IP addresses created on the switch (which
covers all four IP subnets) must be entered into this host's netmon.cf
configuration file. For
example:
