The OpenStack network
Get started with iptables, tables, rules, and chains
This article describes how OpenStack handles networking with iptables, chains, and rules, much like other systems. But first, let's look at the structure of iptables as a refresher of the technology used in this article.
An iptable is a user-space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall; iptable specifically refers to IPv4.
To set up a Linux firewall, rules are used, each of which specifies what to match within a packet and what to do with such a packet. Chains are a list of rules.
The previous incarnation of iptables, ipchains, added the concept of chains of rules; iptables extended this further into tables. So the structure of iptables is: iptables > tables > chains > rules.
The iptable has four built-in tables:
- Filter table: The default table with the following chains:
- INPUT for packets coming to the local server.
- OUTPUT for packets generated locally and going out of the local server.
- FORWARD for packets routed through the local server.
- NAT table (network address translation):
- PREROUTING: Used for destination NAT, it alters packets IP addresses before routing.
- POSTROUTING: Used for source NAT, it alters packets IP addresses after routing.
- OUTPUT: The NAT for locally generated packets on the firewall.
- Mangle table: For specialized packet alteration:
- Raw table: For configuration exemptions:
The iptable within OpenStack
Within OpenStack, you'll find iptable chains and rules predominantly in the Cloud Compute-Nova module, a cloud computing fabric controller (the main part of an IaaS system) written in Python that uses many external libraries. This article details the nova-network FlatDHCPManager component, as well as other OpenStack components, necessary for networking tasks.
When it starts, OpenStack defines some of the OpenStack chains. These chains form a chain fabric with Linux's built-in chains. Another task at startup is to define some rules for fixed network range and the metadata service. Once a network is created and used, nova-network sets up some rules. When one instance (also known as a server and VMs) is created, nova-compute creates one instance-specific chain and set up rules under this chain to assure the instance's connectivity. Regarding floating IPs, OpenStack also uses some rules to make it work. In addition, OpenStack's security group and its rules are embodied by iptables rules.
OpenStack, a global collaboration of developers and cloud computing technologists producing the open standard cloud operating system for both public and private clouds, is free open source software released under the terms of the Apache license. Cloud service providers, enterprises, and government organizations can take advantage of the freely available, Apache-licensed software to build massively scalable cloud environments.
OpenStack currently consists of six core software projects:
- Cloud Compute-Nova
- Cloud Storage-Swift
- Image Service-Glance (delivery and registration)
- Identity Service-Keystone
- Network Connectivity-Quantum
These projects, along with a vibrant ecosystem of technology providers and future projects, deliver a pluggable framework and operating system for public and private clouds.
There are more than 10 binaries in the Nova project, among which 3 are related to VMs connectivity:
- nova-api provides the metadata service for VMs.
- nova-compute sets up the network environment for VMs.
- nova-network sets up the network environment for the entire cloud ecosystem, tasks such as IP allocation, DHCP settings and so on.
The Nova module primarily consists of a set of Python daemons, though it requires and integrates with a number of native system components for databases, messaging, and virtualization capabilities. It uses a special metadata service to enable virtual machine instances to retrieve instance-specific data. Instances access the metadata service at http://169.254.169.254.
The metadata includes public SSH keys (identified by key/pair name when a user requests a new instance), user data (passed as the
user_data parameter in the API call or by the
--user_data flag in the Nova boot command). The binary nova-api implements the metadata service.
OpenStack is complex set of components. To learn more about the system and how one of the other components works, follow the extensive OpenStack resources in the Related topics section.
Before you move onto rules and chains in Nova, let's take a look at IP addressing modes.
IP addressing in Nova
Every VM is automatically assigned a private IP address from each available nova-network. These IP addresses are called fixed IPs. You can optionally assign public IP addresses to instances. OpenStack uses the term floating IP to refer to an IP address (typically public) that can be dynamically added to a running virtual instance.
There are multiple strategies available for implementing fixed IPs:
- Flat mode
- Flat DHCP mode
- VLAN DHCP mode
- Nova-network with quantum mode
Flat mode is the simplest networking mode. Each instance receives a fixed IP from the pool. All instances are attached to the same bridge (br100) by default. The bridge must be configured manually. The networking configuration is injected into the instance before it is booted. And there is no floating IP feature in this mode.
Flat DHCP mode
This is similar to the flat mode in that all instances are attached to the same bridge. In this mode Nova does a bit more configuration; it will attempt to bridge into an Ethernet device (eth0 by default). It will also run
dnsmasq as a dhcpserver listening on this bridge. Instances receive their fixed IPs by doing a
dhcpdiscover. Moreover, floating IP feature is provided.
VLAN DHCP mode
This is the default networking mode and supports the most features. For multiple machine installation, it requires a switch that supports host-managed VLAN tagging. In this mode, Nova will create a VLAN and bridge for each project (same as a tenant). The project gets a range of private IPs that are only accessible from inside the VLAN. For a user to access the instances in his project, a special VPN instance (code-named "cloudpipe") needs to be created. Nova generates a certificate and key for the user to access the VPN and starts the VPN automatically.
Nova-network with quantum mode
In this mode, the DHCP server can also be launched automatically, but floating IP is not supported. There is a quantum agent running on each compute host to connect virtual instances to quantum network. The network topology can be much sophisticated.
In Nova, a security group is a named collection of network access rules, like firewall policies. These access rules specify which incoming network traffic should be delivered to all VM instances in the group; all other incoming traffic being discarded. Users can modify rules for a group at any time. The new rules are automatically enforced for all running instances and instances launched from then on. A security group can be thought of as a security profile or a security role, such as "webappserver."
Rules and chains of iptables
As described earlier, an iptable is a user space application program that allows a system administrator to configure the tables provided by the Linux kernel firewall and the chains and rules it stores.
Also noted previously, several different tables can be defined; among those used in OpenStack are filter and NAT. Each table contains a number of built-in chains and can also contain user-defined chains. Each chain is a list of rules that can match a set of packets. Each rule specifies what to do with a packet that matches — this is called a target, which might be a jump to a user-defined chain in the same table.
From here on, I'll discuss the various key chains and rules.
Chains created when nova-network starts
Figure 1. NetworkManager class connects other classes
As shown in Figure 1,
NetworkManager is a big class that
connects with many other classes or modules. Among these classes or modules, one of them is
linux_net contains an
IptablesManager object, the
__init__() method of which initializes the chains defined in OpenStack system.
Figure 2. Initializing the chains
Tables of IPv4 packet filter rules in the Linux kernel have some built-in chains as mentioned earlier: PREROUTING, INPUT, FORWARD, OUTPUT and POSTROUTING. In general, a packet arriving at a Linux host go to PREROUTING chain. After passing it, the kernel makes a routing decision. If the packet is for the Linux host machine, it goes to INPUT chain, then if accepted, goes to the target process. If the packet is not for the Linux host machine, it goes to FORWARD chain, then to POSTROUTING chain, and then leave the host machine. The packets generated by local processes go to OUTPUT chain first, then if accepted, go to POSTROUTING chain.
Besides these built-in chains, OpenStack system creates some others chains that are hooked into the chain system. OpenStack chains consist of two types — unwrapped and wrapped chains.
Examples of unwrapped chains include nova-filter-top and nova-postrouting-bottom. The nova-filter-top is added at the top of the FORWARD and OUTPUT chains. Its name is not wrapped, so it's shared between the various Nova workers. It's intended for rules that need to live at the top of the FORWARD and OUTPUT chains. It's in both the IPv4 and IPv6 set of tables.
Examples of wrapped chains include the green colored chains. The name of these chains has the process name as a suffix. For example, nova-network creates the nova-network-PREROUTING chain.
For IPv4 and IPv6, the built-in INPUT, OUTPUT, and FORWARD filter chains are wrapped, meaning that the "real" INPUT chain has a rule that jumps to the wrapped INPUT chain, etc. Additionally, there's a wrapped chain named local which is jumped to from nova-filter-top.
For IPv4, the built-in PREROUTING, OUTPUT, and POSTROUTING NAT chains are wrapped in the same way as the built-in filter chains. Additionally, there's an sNAT chain and a float-sNAT chain that are applied after the POSTROUTING chain.
Rules when nova-network starts
Figure 3 shows part of the startup process of FlatDHCPManager. The
__init__() method that constructs an
IptableManager object to create many OpenStack chains
was previously described. So let's focus on
init_host() method calls the
initialize() method of LinuxNet3 which calls methods of
linux_net to set up some iptables rules in a NAT table.
Figure 3. Startup process of FlatDHCPManager
Let's look at these rules.
Rule for metadata host
Among rules created by
linux_net, one rule is to allow IPs under
FLAGS.fixed_range to access metadata_host (it is 192.168.1.90 in this case).
-A nova-network-POSTROUTING -s 10.0.0.0/8 -d 192.168.1.90/32 -j ACCEPT
ensure_metadata_ip() adds 169.254.169.254/32 to
lo device with the following command:
# ip addr add 169.254.169.254/32 scope link dev lo
metadata_forward() adds a dNAT rule to route packets from 169.254.169.254/32 to metadata_host:
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.90:8775
If the nova-network and nova-api are not running on the same host, you must define the metadata_host on the nova-network host to point to the nova-api host.
Rule to access dmz
FLAGS.dmz_cidr defines a list of dmz (perimeter networking)
CIDRs (classless inter-domain routing). By default, it is an empty list. In this case
it is 10.128.0.0/24. So, the rule is:
-A nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.128.0.0/24 -j ACCEPT
Rule for VMs to connect each other
Another rule is to ensure VMs with two fixed IPs can talk with each other:
-A nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -m conntrack ! --ctstate DNAT -j ACCEPT
Rule for access outside of fixed subnet
add_snat_rule() method adds an sNAT rule into wrapped chain
sNAT of the NAT table. You can see
FLAGS.routing_source_ip in Figure 3. The
ip_range's value is defined by
FLAGS.fixed_range. In this case, it is 10.0.0.0/8.
default, which is defaulted by function
flags._get_my_ip(). In this case,
FLAGS.my_ip is 192.168.1.90. After that, you get a rule such as:
-A nova-network-snat -s 10.0.0.0/8 -j SNAT --to-source 192.168.1.90
For fixed IPs to access outside, you must create a network that is a subnet of
FLAGS.fixed_range. That way, with the default gateway pointed to one IP of br100 on the nova-network machine, the VMs with this subnet's IPs are able to access an outside network.
Rules for each network
To set up a network, nova-network creates some rules in the filter table. To trigger the nova-network's operation on a given network, run following commands:
- Create a network and set the host:
# ./bin/nova-manage network create mynet 10.10.10.0/24
- Boot a server:
nova boot --image a3fb743d-42df-49ba-b9c4-8042ebbd344e --flavor 1 myserver
After executing these commands, you have these rules:
- Allow a forwarded traffic pass bridge so the IP on br100 can work as a gateway:
-A nova-network-FORWARD -i br100 -j ACCEPT -A nova-network-FORWARD -o br100 -j ACCEPT
- Allow DHCP and DNS traffic to come into the local
-A nova-network-INPUT -i br100 -p udp -m udp --dport 67 -j ACCEPT -A nova-network-INPUT -i br100 -p tcp -m tcp --dport 67 -j ACCEPT -A nova-network-INPUT -i br100 -p udp -m udp --dport 53 -j ACCEPT -A nova-network-INPUT -i br100 -p tcp -m tcp --dport 53 -j ACCEPT
Note: 67 is DHCP port and 53 is DNS port.
Rules on nova-compute host
The nova-compute module also creates wrapped chains. On these chains it creates some rules in the filter table. Let's look at those.
Rules to allow forwarded traffic pass bridge
These rules on the nova-compute host allow VMs to connect with nova-network host and VMs on other compute hosts.
-A nova-compute-FORWARD -i br100 -j ACCEPT -A nova-compute-FORWARD -o br100 -j ACCEPT
Chain and rules for each instance
For each instance, it creates one chain and some rules in the filter table (Figure 4).
Figure 4. Chain and rules for each instance
You can see:
- nova-compute creates one chain for each instance. In Figure 4, the chain's name is nova-compute-inst-1.
- All traffic targeted to this instance, no matter whether it is forwarded or generated from local process, go into this instance's specific chain.
- All traffic from IPs of the subnet that the instance resides in is allowed.
- All DHCP traffic from the specified DHCP server is allowed.
- All other traffic is dropped.
On starting, nova-api creates a rule in the filter table to allow others to access the nova-api service.
-A nova-api-INPUT -d 192.168.1.90/32 -p tcp -m tcp --dport 8775 -j ACCEPT
To see how floating IPs are implemented, first associate a floating IP to the instance's fixed IP. The fixed IP of the instance previously created is 10.10.10.2.
Create a floating IP in the default pool
To create a floating IPs in the default pool:
# nova-manage floating create --ip_range=192.168.1.232/30
To allocate a floating IP from this pool:
# Nova floating-ip-create
You've got an IP of 192.168.1.233. Now assign it to the instance that has an ID of 8f773639-c04f-4885-9349-ac7d6a799843:
# nova add-floating-ip 8f773639-c04f-4885-9349-ac7d6a799843 192.168.1.233
Bind floating IP to public interface
FLAGS.public_interface is used to bind floating IPs. After you
nova add-floating-ip command, you can see you have the following floating IP under the
# ip addr list dev wlan0 3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 08:11:96:75:91:54 brd ff:ff:ff:ff:ff:ff inet 192.168.1.90/16 brd 192.168.255.255 scope global wlan0 inet 192.168.1.233/32 scope global wlan0 inet6 fe80::a11:96ff:fe75:9154/64 scope link valid_lft forever preferred_lft forever
Rules in NAT table for floating IPs
After the instance gets a floating IP on the nova-network host, these rules apply:
-A nova-network-OUTPUT -d 192.168.1.233/32 -j DNAT --to-destination 10.10.10.2 -A nova-network-PREROUTING -d 192.168.1.233/32 -j DNAT --to-destination 10.10.10.2 -A nova-network-float-snat -s 10.10.10.2/32 -j SNAT --to-source 192.168.1.233
You can see that dNAT rule is used to translate the floating IP into the instance's
fixed IP. If a packet arrives at the nova-network host with the floating IP as target
IP, the target IP is translated. And then you have a sNAT rule that translates traffic
from the instance's fixed IP into the floating IP. Since all the traffic from VMs to
outside of the fixed network is pointed to gateway, which is set by the nova-network's
dnsmasq process, with this sNAT rule the traffic out from
VMs can be masked as from the floating IP successfully. In addition, you have a dNAT
rule in a wrapped OUTPUT chain that allows the local process on nova-network to access VMs with floating IPs.
Ping VMs with floating IP
To ping VMs with floating IPs, you need some more rules. Remember that on the nova-compute host, you have a specific chain for each instance; the rules in it just allow the traffic from IPs in the fixed subnet. If you ping a floating IP, the traffic will be dropped by those rules since the source IP of the ping packet is not within the fixed subnet. Obviously, you need add a rule to allow icmp traffic.
To add a rule allowing ping, use OpenStack's security group rule concept:
# nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
After that, you can see you have one more rule created under that instance's specific chain:
-A nova-compute-inst-1 -p icmp -j ACCEPT
In the same way, you can enable SSH to the VMs with floating IP.
Rules from iptables are used extensively by OpenStack network. Its security group and floating IPs concepts are just the start of using the iptables rules. Dive into the resources that follow to learn more about using OpenStack's IaaS environment.
Explore the OpenStack system with resources from the developer community:
- Cloud Compute-Nova module:
- Network Connectivity-Quantum at OpenStack
- Image Services-Glance
- Identity Services-Keystone
- Cloud Storage-Swift at OpenStack
- Dashboard-Horizon at OpenStack
- OpenStack manuals:
- Answers to common questions