CES IP aliasing to network adapters on protocol nodes
Cluster Export Services (CES) is a functionality in IBM Spectrum Scale that enables NFS, SMB, and Object protocols. Irrespective of which protocols you choose, all are accessible through a floating pool of IP addresses called CES IPs. This pool of CES IPs is considered floating because each IP can move independently among all protocol nodes. In the event of a protocol node failure, accessibility to all protocols is maintained as the CES IPs automatically move from the failed protocol node to a healthy protocol node. Use this information to understand how CES IPs are assigned and how they are aliased to adapters with or without VLAN tagging.
Virtual LANs (VLANs) are often associated with secure networks because they provide a means of separating network devices into independent networks. Although the physical network infrastructure is shared, unicast, multicast, and broadcast traffic from a network device in a VLAN is restricted to other devices within that same VLAN.
How are CES IPs assigned
CES IPs are automatically assigned and aliased to existing network adapters on protocol nodes during startup. The following example shows aliased CES IPs in a flat network environment or a single VLAN environment in which the switch ports are set to Access mode and thus, do not need VLAN tagging.

Example of aliased CES IPs using the ip addr command
eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.11.1.122/24 brd 10.11.1.255 scope global eth1
valid_lft forever preferred_lft forever
inet 10.11.1.5/24 brd 10.11.1.255 scope global secondary eth1:0
valid_lft forever preferred_lft forever
inet 10.11.1.7/24 brd 10.11.1.255 scope global secondary eth1:1
valid_lft forever preferred_lft forever
Example of preexisting routes
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 eth1
10.11.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eth1
172.31.128.0 0.0.0.0 255.255.128.0 U 300 0 0 data0
In the preceding example, eth1
preexists with an established route and IP:
10.11.1.122
. This is manually assigned and must be accessible prior to any CES
configuration. Once CES services are active, CES IPs are then automatically aliased to this base
adapter, thus creating eth1:0
and eth1:1
. The floating CES IPs
assigned to the aliases are 10.11.1.5
and 10.11.1.7
. Both CES IPs
are allowed to move to other nodes in case of a failure. This automatic movement combined with the
ability to manually move CES IPs, might cause a variance in the number of aliases and CES IPs among
protocol nodes. The data0
interface illustrates how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter used for CES IPs.
Example distribution of CES IPs among two protocol nodes after enablement of protocols
mmces address list
Address Node Group Attribute
-------------------------------------------------------------------------
10.11.1.5 protocol-node-1 none none
10.11.1.6 protocol-node-2 none object_database_node,object_singleton_node
10.11.1.7 protocol-node-1 none none
10.11.1.8 protocol-node-2 none none
CES IPs and VLAN tags
A network switch port can be considered a trunk port if it gives access to multiple VLANs. When this occurs, it is necessary for a VLAN tag to be added to each frame. This VLAN tag is an identification allowing switches to contain traffic within specific networks. If multiple networks must access data from IBM Spectrum Scale protocol nodes, then one possible option is to configure trunk ports on the switch directly connected to the IBM Spectrum Scale protocol nodes. Once a trunk port exists, VLAN tags are necessary on the connected network adapters. Note that CES IPs are automatically assigned and aliased to existing network adapters on protocol nodes during startup. Due to this, the existence of VLAN tags requires a preexisting network adapter with an established route and IP so that CES IPs can alias to it.

Example of aliased CES IPs using the ip addr command (with VLAN tag)
eth1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
valid_lft forever preferred_lft forever
eth1.3016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.16.122/24 brd 10.30.16.255 scope global eth1.3016
valid_lft forever preferred_lft forever
inet 10.30.16.5/24 brd 10.30.16.255 scope global secondary eth1.3016:0
valid_lft forever preferred_lft forever
inet 10.30.16.7/24 brd 10.30.16.255 scope global secondary eth1.3016:1
valid_lft forever preferred_lft forever
Example of pre-existing routes (with VLAN tag)
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 eth1.3016
10.30.16.0 0.0.0.0 255.255.255.0 U 100 0 0 eth1.3016
172.31.128.0 0.0.0.0 255.255.128.0 U 300 0 0 data0
As in the no VLAN tag example, an existing network adapter must be present so that CES\ IPs can
alias to it. Note the non-VLAN base adapter eth1, has no IPs assigned. In this example, the
preexisting network adapter with an established route and IP is eth1.3016
. The IP
for eth1.3016
is 10.30.16.122
and the VLAN tag is
3016
. This preexisting IP can be used for network verification prior to CES IP
configuration by pinging it from external to the cluster or pinging it from other protocol nodes. It
is a good practice to make sure that all protocol node base adapter IPs are accessible before
enabling protocols. The data0
interface shows how a network used for GPFS intra-cluster connectivity between nodes can be separate from the adapter used for CES IPs.
Example distribution of CES IPs among two protocol nodes after enablement of protocols (with VLAN tag)
mmces address list
Address Node Group Attribute
-------------------------------------------------------------------------
10.30.16.5 protocol-node-1 none none
10.30.16.6 protocol-node-2 none object_database_node,object_singleton_node
10.30.16.7 protocol-node-1 none none
10.30.16.8 protocol-node-2 none none
CES IPs and multiple VLAN tags
The following diagram shows a node with two network adapters devoted to CES protocols:
eth1
and eth2
. The eth1
interface has two VLANs
associated: 3016
and 3017
. The eth2
interface has
one VLAN associated: 80
.

Example of aliased CES IPs using the ip addr command (with multiple VLAN tag)
eth1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
valid_lft forever preferred_lft forever
eth1.3016: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.16.122/24 brd 10.30.16.255 scope global eth1.3016
valid_lft forever preferred_lft forever
inet 10.30.16.5/24 brd 10.30.16.255 scope global secondary eth1.3016:0
valid_lft forever preferred_lft forever
inet 10.30.16.7/24 brd 10.30.16.255 scope global secondary eth1.3016:1
valid_lft forever preferred_lft forever
eth1.3017: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.30.17.50/24 brd 10.30.17.255 scope global eth1.3017
valid_lft forever preferred_lft forever
inet 10.30.17.100/24 brd 10.30.16.255 scope global secondary eth1.3017:0
valid_lft forever preferred_lft forever
inet 10.30.17.103/24 brd 10.30.16.255 scope global secondary eth1.3017:1
valid_lft forever preferred_lft forever
eth2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
valid_lft forever preferred_lft forever
eth2.80: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000
link/ether 00:50:56:83:16:e5 brd ff:ff:ff:ff:ff:ff
inet 10.11.80.50/24 brd 10.11.80.255 scope global eth1.80
valid_lft forever preferred_lft forever
inet 10.11.80.55/24 brd 10.11.80.255 scope global secondary eth1.80:0
valid_lft forever preferred_lft forever
Example of preexisting routes (with multiple VLAN tag)
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 eth1.3016
10.30.16.0 0.0.0.0 255.255.255.0 U 100 0 0 eth1.3016
10.30.17.0 0.0.0.0 255.255.255.0 U 400 0 0 eth1.3017
10.11.80.0 0.0.0.0 255.255.255.0 U 200 0 0 eth2.80
172.31.128.0 0.0.0.0 255.255.128.0 U 300 0 0 data0
Example distribution of CES IPs from multiple VLANs among two protocol nodes after enablement of protocols
mmces address list
Address Node Group Attribute
-------------------------------------------------------------------------
10.11.80.54 protocol-node-2 none none
10.11.80.55 protocol-node-1 none none
10.30.16.5 protocol-node-1 none none
10.30.16.6 protocol-node-2 none none
10.30.16.7 protocol-node-1 none none
10.30.16.8 protocol-node-2 none none
10.30.17.100 protocol-node-1 none none
10.30.17.101 protocol-node-2 none none
10.30.17.102 protocol-node-2 none object_database_node,object_singleton_node
10.30.17.103 protocol-node-1 none none