Part one of a three-part series investigating a multi-zonal workload utilizing multiple network interfaces and discussing reachability among interfaces.
In IBM Cloud, more than one network interface can be attached to a virtual server instance. Having multiple network interfaces allows for the isolation of network traffic for different purposes and the application of different security rules to each interface. Moreover, a higher aggregated network throughput can be achieved by using multiple network interfaces to overcome the throughput cap of a single interface.
There is one important thing you must be aware of when utilizing multiple network interfaces: routing. In most cases, multiple network interfaces work without an additional routing configuration. For example, if each network interface is attached to a different subnet and is utilized for communication within the subnet, the default routing rules set by the guest operating system work as intended. However, there are cases in which routing rules need to be configured in the guest operating system to enable network communication.
In this and the following two parts of this three-part blog series, we will investigate a multi-zonal workload utilizing multiple network interfaces and discuss reachability among interfaces. Specifically, it is desirable to achieve communication via a specific set of interfaces so that one can isolate traffic (as well as control bandwidth).
- Multiple Virtual Network Interfaces Connectivity and Usage in IBM Cloud VPC: Part 1
- Multiple Virtual Network Interfaces Connectivity and Usage in IBM Cloud VPC: Part 2
- Multiple Virtual Network Interfaces Connectivity and Usage in IBM Cloud VPC: Part 3
We will describe the following three different ways to achieve the desired configuration via various routing control mechanisms:
- Custom routes
- Source interface-based routing tables
- Network namespaces
We assume Ubuntu Linux as our guest operating system and some basic understanding on the reader’s part. In this blog series, we will not cover how to provision multiple network interfaces themselves. Readers are referred to our online manual.
Motivating example
Let’s suppose we want to design an environment that has two main types of subnets in each availability zone (see the figure below). The first is the control network. From the control network in Zone1 we want to be able to drive (control) the workload to Zone1 and Zone2 via “Control Subnet1” and “Control Subnet2,” respectively. Each worker node has two attached interfaces. To be clear, only control messages are communicated on this network, and the workload execution/data are not transmitted on this subnet. The primary interface, in this case, is attached to the Control Subnets. Additionally, the “Data Subnet1” or “Data Subnet2” is attached via a secondary network interface. By setting up the network interfaces this way, we hope to isolate data traffic from the control communication:
Why the default routing rules do not work
By default, Ubuntu Linux on Worker1 will set up routing rules as follows:
Rules C and E define routes to its local subnets — Control Subnet1 and Data Subnet1, respectively. Rules A and B define a default route to utilize Interface A (ens3), connected to Control Subnet1, for all of the packets whose destination is not Control Subnet1 or Data Subnet1. Worker2 has similar routing rules, but for Control Subnet2 and Data Subnet2.
With these routing rules, Interfaces A and C on Worker1 and Worker2, respectively, can communicate with each other, but Interfaces B and D on Worker1 and Worker2 cannot.
This could be explained as follows — suppose an application on Worker1 needs to communicate with Worker2 using the Data subnets. It sends a request packet to Interface D on Worker2 and receives a response. Because on Worker1 there is no specific rule that matches Interface D or Data Subnet2, the request packet will go out from Interface A through Control Subnet1, using the default route. This is not desirable because we want the request packet to use Data Subnet1. In addition, depending on the security rules, the request packet might not even reach Interface D from Interface A.
This issue can be avoided by explicitly specifying that we use Interface B for the request packet. However, a more serious issue arises for response packets. On Worker2, the response packet will attempt to go out from Interface C through Control Subnet2 for a similar reason as the request packet used Interface A. Unlike the request packet, the response packet is not allowed to go out using the default route because of the anti-spoofing mechanism. Note that the response packet has the IP address of Interface D in its source IP field because the request packet came in from Interface D. However, the anti-spoofing mechanism in IBM Cloud prevents an egress packet whose source IP field does not match the IP address of the interface being utilized to send it out. Disabling the anti-spoofing mechanism could be one option, but it would not be what you want to try first, owing to security concerns.
Summary
Having multiple network interfaces effectively isolates network traffic and achieves higher aggregated throughput, but there are cases where you must configure additional routing rules to establish reachability among virtual server instances. In this post, we have described why the default routing rules do not work in a multi-zone controller/worker scenario.
In the next post in this three-post series, we will present a simple approach called Custom Routes to configure the reachability.
- Multiple Virtual Network Interfaces Connectivity and Usage in IBM Cloud VPC: Part 1
- Multiple Virtual Network Interfaces Connectivity and Usage in IBM Cloud VPC: Part 2
- Multiple Virtual Network Interfaces Connectivity and Usage in IBM Cloud VPC: Part 3
To learn more, check out this tutorial, which walks you through creating your own IBM Cloud Virtual Private Cloud (VPC) with multiple subnets and a virtual server instance (VSI) in each subnet.
Was this article helpful?
YesNo