CES IP aliasing to network adapters on protocol nodes

Cluster Export Services (CES) is a functionality in IBM Storage Scale that enables NFS, SMB, and Object protocols. Irrespective of which protocols you choose, all are accessible through a floating pool of IP addresses called CES IP addresses. This pool of CES IP addresses is considered floating because each IP can move independently among all protocol nodes. During a protocol node failure, accessibility to all protocols is maintained as the CES IP addresses automatically move from the failed protocol node to a healthy protocol node. Use this information to understand how CES IP addresses are assigned and are aliased to adapters with or without VLAN tagging.

Virtual LANs (VLANs) are often associated with secure networks because they provide a means of separating network devices into independent networks. Although the physical network infrastructure is shared, unicast, multicast, and broadcast traffic from a network device in a VLAN is restricted to other devices within that same VLAN.

How are CES IP addresses assigned

CES IP addresses are automatically assigned and aliased to existing network adapters on protocol nodes during startup. The following example shows aliased CES IP addresses in a flat network environment or a single VLAN environment. The switch ports in these environments are set to Access mode and thus do not need VLAN tagging.

Example of aliased CES IP addresses by using the ip addr command

eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.11.1.122/24 brd 10.11.1.255 scope global eth1
   valid_lft forever preferred_lft forever
inet 10.11.1.5/24 brd 10.11.1.255 scope global secondary eth1
   valid_lft forever preferred_lft forever
inet 10.11.1.7/24 brd 10.11.1.255 scope global secondary eth1
   valid_lft forever preferred_lft forever

Example of preexisting routes

Kernel IP routing table
Destination     Gateway        Genmask         Flags Metric Ref  Use  Iface
default          gateway        0.0.0.0         UG   100    0   0     eth1
10.11.1.0        0.0.0.0        255.255.255.0   U    100    0   0     eth1
172.31.128.0    0.0.0.0         255.255.128.0   U    300    0   0     data0

In the preceding example, eth1 preexists with an established route and IP: 10.11.1.122. This IP is manually assigned and must be accessible before any CES configuration. When the CES services are active, CES IP addresses are then automatically aliased to this base adapter, thus creating eth1. The floating CES IP addresses assigned to the aliases are 10.11.1.5 and 10.11.1.7. Both CES IP addresses are allowed to move to other nodes if there is a failure. This automatic movement combined with the ability to manually move CES IP addresses, might cause a variance in the number of aliases and CES IP addresses among protocol nodes. The data0 interface illustrates how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter that is used for CES IP addresses.

Example distribution of CES IP addresses among two protocol nodes after enablement of protocols

mmces address list

Address       Node                        Group      Attribute
-------------------------------------------------------------------------
10.11.1.5     protocol-node-1             none       none
10.11.1.6     protocol-node-2             none       object_database_node,object_singleton_node
10.11.1.7     protocol-node-1             none       none
10.11.1.8     protocol-node-2             none       none

CES IP addresses and VLAN tags

A network switch port can be considered a trunk port if it gives access to multiple VLANs. When it occurs, it is necessary for a VLAN tag to be added to each frame. This VLAN tag is an identification that allows switches to contain traffic within specific networks. If multiple networks must access data from IBM Storage Scale protocol nodes, then one possible option is to configure trunk ports on the switch that is directly connected to the IBM Storage Scale protocol nodes. After a trunk port is configured, VLAN tags are necessary on the connected network adapters. The CES IP addresses are automatically assigned and aliased to existing network adapters on protocol nodes during startup. To enable this process, the available VLAN tags require a preexisting network adapter with an established route and IP so that the CES IP addresses can alias to it.

Example of aliased CES IP addresses by using the ip addr command (with VLAN tag)

eth1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
    valid_lft forever preferred_lft forever
 
eth1.3016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.16.122/24 brd 10.30.16.255 scope global eth1.3016
   valid_lft forever preferred_lft forever
inet 10.30.16.5/24 brd 10.30.16.255 scope global secondary eth1.3016
   valid_lft forever preferred_lft forever
inet 10.30.16.7/24 brd 10.30.16.255 scope global secondary eth1.3016
   valid_lft forever preferred_lft forever

Example of pre-existing routes (with VLAN tag)

Kernel IP routing table
Destination      Gateway     Genmask         Flags Metric Ref Use   Iface
default          gateway     0.0.0.0         UG    100    0   0    eth1.3016
10.30.16.0       0.0.0.0     255.255.255.0   U     100    0   0    eth1.3016
172.31.128.0     0.0.0.0     255.255.128.0   U     300    0   0    data0

As in the no VLAN tag example, an existing network adapter must be present so that CES\ IP addresses can alias to it. No IP addresses are assigned to the non-VLAN base adapter eth1. In this example, the preexisting network adapter with an established route and IP is eth1.3016. The IP for eth1.3016 is 10.30.16.122 and the VLAN tag is 3016. This preexisting IP can be used for network verification, before configuration of CES IP, by pinging it from external to the cluster or pinging it from other protocol nodes. It is a good practice to make sure that all protocol node base adapter IP addresses are accessible before the protocols are enabled. The data0 interface shows how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter that is used for CES IP addresses.

Example distribution of CES IP addresses among two protocol nodes after enablement of protocols (with VLAN tag)

mmces address list

Address        Node                         Group      Attribute
-------------------------------------------------------------------------
10.30.16.5     protocol-node-1              none       none
10.30.16.6     protocol-node-2              none       object_database_node,object_singleton_node
10.30.16.7     protocol-node-1              none       none
10.30.16.8     protocol-node-2              none       none

CES IP addresses and multiple VLAN tags

The following diagram shows a node with two network adapters that are devoted to CES protocols: eth1 and eth2. Two VLANs are associated with the eth1 interface: 3016 and 3017. One VLAN is associated with the eth2 interface: 80.

Example of aliased CES IP addresses by using the ip addr command (with multiple VLAN tags)

eth1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
    valid_lft forever preferred_lft forever
 
eth1.3016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.16.122/24 brd 10.30.16.255 scope global eth1.3016
   valid_lft forever preferred_lft forever
inet 10.30.16.5/24 brd 10.30.16.255 scope global secondary eth1.3016
   valid_lft forever preferred_lft forever
inet 10.30.16.7/24 brd 10.30.16.255 scope global secondary eth1.3016
   valid_lft forever preferred_lft forever
 
eth1.3017: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.17.50/24 brd 10.30.17.255 scope global eth1.3017
   valid_lft forever preferred_lft forever
inet 10.30.17.100/24 brd 10.30.16.255 scope global secondary eth1.3017
   valid_lft forever preferred_lft forever
inet 10.30.17.103/24 brd 10.30.16.255 scope global secondary eth1.3017
   valid_lft forever preferred_lft forever
 
eth2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
    valid_lft forever preferred_lft forever
 
eth2.80: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
  link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.11.80.50/24 brd 10.11.80.255 scope global eth1.80
   valid_lft forever preferred_lft forever
inet 10.11.80.55/24 brd 10.11.80.255 scope global secondary eth1.80
   valid_lft forever preferred_lft forever

Example of preexisting routes (with multiple VLAN tag)

Kernel IP routing table
Destination    Gateway     Genmask         Flags Metric Ref  Use  Iface
default        gateway     0.0.0.0         UG    100    0     0   eth1.3016
10.30.16.0     0.0.0.0     255.255.255.0   U     100    0     0   eth1.3016
10.30.17.0     0.0.0.0     255.255.255.0   U     400    0     0   eth1.3017
10.11.80.0     0.0.0.0     255.255.255.0   U     200    0     0   eth2.80
172.31.128.0   0.0.0.0     255.255.128.0   U     300    0     0   data0

Example distribution of CES IP addresses from multiple VLANs among two protocol nodes after enablement of protocols

mmces address list
Address         Node                           Group      Attribute
-------------------------------------------------------------------------
10.11.80.54     protocol-node-2                none       none
10.11.80.55     protocol-node-1                none       none
10.30.16.5      protocol-node-1                none       none
10.30.16.6      protocol-node-2                none       none
10.30.16.7      protocol-node-1                none       none
10.30.16.8      protocol-node-2                none       none
10.30.17.100    protocol-node-1                none       none
10.30.17.101    protocol-node-2                none       none
10.30.17.102    protocol-node-2                none       object_database_node,object_singleton_node
10.30.17.103    protocol-node-1                none       none