Using remote access with public and private IP addresses

GPFS™ permits the use of both public and private IP address. Private IP addresses are typically used to communicate on private networks.

Private IP addresses are on one of these subnets:

10.0.0.0
172.16.0.0
192.168.0.0

See RFC 1597 - Address Allocation for Private Internets for more information.

Use the mmchconfig command, subnets attribute, to specify the private IP addresses to be accessed by GPFS.

Figure 1 describes an AIX® cluster named CL1 with nodes named CL1N1, CL1N2, and so forth, a Linux cluster named CL2 with nodes named CL2N1, CL2N2, and another Linux cluster named CL3 with a node named CL3N1. Both Linux clusters have public Ethernet connectivity, and a Gigabit Ethernet configured with private IP addresses (10.200.0.1 through 10.200.0.24), not connected to the public Ethernet. The InfiniBand Switch on the AIX cluster CL1 is configured using public IP addresses on the 7.2.24/13 subnet and is accessible from the outside.

With the use of both public and private IP addresses for some of the nodes, the setup works as follows:

All clusters must be created using host names or IP addresses that correspond to the public network.
Using the mmchconfig command for the CL1 cluster, add the attribute: subnets=7.2.24.0.
This allows all CL1 nodes to communicate using the InfiniBand Switch. Remote mounts between CL2 and CL1 will use the public Ethernet for TCP/IP communication, since the CL2 nodes are not on the 7.2.24.0 subnet.

GPFS assumes subnet specifications for private networks are independent between clusters (private networks are assumed not physically connected between clusters). The remaining steps show how to indicate that a private network is shared between clusters.
Using the mmchconfig command for the CL2 cluster, add the subnets='10.200.0.0/CL2.kgn.ibm.com;CL3.kgn.ibm.com' attribute. Alternatively, regular expressions are allowed here, such as subnets='10.200.0.0/CL[23].kgn.ibm.com'. See note 2 for the syntax allowed for the regular expressions.
This attribute indicates that the private 10.200.0.0 network extends to all nodes in clusters CL2 or CL3. This way, any two nodes in the CL2 and CL3 clusters can communicate through the Gigabit Ethernet.

This setting allows all CL2 nodes to communicate over their Gigabit Ethernet. Matching CL3.kgn.ibm.com with the cluster list for 10.200.0.0 allows remote mounts between clusters CL2 and CL3 to communicate over their Gigabit Ethernet.
Using the mmchconfig command for the CL3 cluster, add the subnets='10.200.0.0/CL3.kgn.ibm.com;CL2.kgn.ibm.com' attribute, alternatively subnets='10.200.0.0/CL[32].kgn.ibm.com'.
This attribute indicates that the private 10.200.0.0 network extends to all nodes in clusters CL2 or CL3. This way, any two nodes in the CL2 and CL3 clusters can communicate through the Gigabit Ethernet.

Matching of CL3.kgn.ibm.com with the cluster list for 10.200.0.0 allows all CL3 nodes to communicate over their Gigabit Ethernet, and matching CL2.kgn.ibm.com with that list allows remote mounts between clusters CL3 and CL2 to communicate over their Gigabit Ethernet.

Use the subnets attribute of the mmchconfig command when you wish the GPFS cluster to leverage additional, higher performance network connections that are available to the nodes in the cluster, or between clusters.

Notes:

Use of the subnets attribute does not ensure a highly available system. If the GPFS daemon is using the IP address specified by the subnets attribute, and that interface goes down, GPFS does not switch to the other network. You can use mmdiag --network to verify that the subnet is in fact being used.
Each subnet can be listed at most once in each cluster. For example, specifying:
```
subnets='10.200.0.0/CL2.kgn.ibm.com 10.200.0.0/CL3.kgn.ibm.com'
```
where the 10.200.0.0 subnet is listed twice, is not allowed. Therefore, subnets that span multiple clusters have to be assigned a cluster name pattern or a semicolon-separated cluster name list. It is possible to combine these, for example, items in semicolon-separated cluster lists can be plain names or regular expressions, as in the following:
```
subnets='1.0.0.1/CL[23].kgn.ibm.com;OC.xyz.ibm.com'
```
The following shows examples of patterns that are accepted:
```
[af3] matches letters 'a' and 'f', and number 3
[0-7] matches numbers 0, 1, ... 7
[a-p0-7] matches letter a, b, ... p and numbers from 0 to 7 inclusive
* matches any sequence of characters
? matches any (one) character
```
If the subnets attribute lists multiple subnets, and there are multiple subnets in common between the local cluster and a given remote cluster, then the first subnet in common in the list is used for communications between the local and remote clusters. As an example, suppose that the subnets attribute is set as follows, on cluster CL2.kgn.ibm.com:
```
subnets='10.200.0.0/CL[23].kgn.ibm.com 10.201.0.0/CL[23].kgn.ibm.com'
```
If node CL2N1 on cluster CL2.kgn.ibm.com has network interfaces with IP addresses 10.200.0.1 and 10.201.0.1, and node CLN31 on cluster CL3.kgn.ibm.com has network interfaces with IP addresses 10.200.0.5 and 10.201.0.5, then the communication between these two nodes will flow over the 10.200.0.0 subnet, with CL2N1 using the interface with IP address 10.200.0.1, and CLN31 using the interface with IP address 10.200.0.5.
Specifying a cluster name or a cluster name pattern for each subnet is only needed when a private network is shared across clusters. If the use of a private network is confined within the local cluster, then no cluster name is required in the subnet specification.

Figure 1. Use of public and private IP addresses in three GPFS clusters

This figure describes an AIX cluster named CL1 with nodes named CL1N1, CL1N2, and so forth, a Linux cluster named CL2 with nodes named CL2N1, CL2N2, and another Linux cluster named CL3 with a node named CL3N1. Both Linux clusters have public Ethernet connectivity to the outside, and a Gigabit Ethernet configured with private IP addresses (10.200.0.1 through 10.200.0.24), not connected to the public Ethernet. The InfiniBand Switch on the AIX cluster, CL1 is configured using public IP addresses on the 7.2.24/13 subnet, and is accessible from the outside.