June 19, 2015 | Written by: Iqbal Mohomed
Share this post:
As of May 23rd IBM Bluemix Container Service now provides a native Kubernetes operations experience while removing the burden of maintaining master nodes. Kubernetes itself is based on the Docker engine for managing software images and instantiating containers. Get the details.
I’m a big fan of Docker containers because I truly believe they are nothing short of a game changer in how application backends are developed. That said, one tricky part has been setting up networking in a multi-host environment.
When you’re on a single Docker host, your container gets a private IP address, which is wired to the host’s docker0 bridge. In order to expose a container, you simply make use of Network Address Translation (NAT) to map a host port to a container port. This happy situation gets quite a bit more complicated when you have containers spanning multiple hosts. The network design we’ve managed to achieve in the IBM Containers service is one of its coolest features, in my opinion. We make use of an overlay network on top of our hosts, allowing each tenant to receive a private subnet. Containers created by a given tenant on any host get IP addresses in that tenant’s private subnet. Moreover, there is no need to deal with port mappings and NAT!
Since the time we started designing and building our service, Docker has evolved at an impressive speed. One of the features I’m most excited about is Docker Swarm. It allows you to set up a single tenant multi-host environment pretty much anywhere with great ease. You can use bare metal machines, or VMs in a public cloud like IBM’s Bluemix VM service, Softlayer or AWS. Swarm’s already got support for a variety of discovery backends such as etcd and Consul, and Docker picked some clever defaults for how clusters get setup. As I played with Swarm clusters, the one feature I wished for was a more advanced networking experience, similar to what we created in the IBM Container Service.
I decided to finally scratch that particular itch and I’ll detail my solution in the rest of this post. First, I made use of Neutron, the powerful Openstack networking piece. I created a small piece of code called Clampify that bridges the world’s of Neutron and Docker. This agent runs on every Swarm minion (what I like to think of as compute nodes) and listens to events from the local Docker Daemon. In the usual case, whenever a new container is created on the host, the clampify agent will create a VIF (virtual interface), move it inside the container’s network namespace and wire it to the local neutron OVS bridge (br-int). From this point on, the regular Neutron machinery kicks in. The Neutron L2 agent, which is also running on each minion/compute node, detects a new configured port on br-int and sets up the rest of the details for getting the overlay happy. Voila! You now have isolated private networks across your swarm minions. Here’s a picture of how all the pieces fit together:
When I first started this project a few months ago, I was worried that I wouldn’t be able to separate Neutron from Nova. I’m happy to report that I was able to achieve this without making any code changes to either the Openstack components or Docker. I have to credit the neutron-debug agent for helping me figure out the incantations to get the overlay networking pieces happy.
An interesting setup allowed by the mechanism we’ve got is a hybrid scenario where you take an existing Openstack installation that works with VMs, and run the clampify agent (and other Swarm components) on a subset of the compute nodes. On this subset, there are no nova components – just Docker and swarm. It is interesting to compare and contrast this with Magnum (then Openstack container project). With this approach, you get containers on baremetal and also hooked directly into Neutron. Thus, you can have communication between containers on swarm and Virtual Machines.
Clampify is pretty early code and is barebones. However, it works and it scratches the itch I had when it came to networking on Swarm. I hope others will find it useful as well. I welcome feedback, ideas and code contributions. Openstack Neutron is very powerful technology and can be complex. That said, there is enough information about it to make the task of setting it up the way you want tractable.