Configuring flannel overlay network with VXLAN for Docker on IBM Power Systems servers

Flannel provides a configurable virtual overlay network for use with containers. For a multihost container cluster, this provides an easy way to have a routable IP address inside the cluster. Flannel uses etcd distributed key-value store to save the network configurations and make it available to different hosts in the cluster. In this article, we'll see how to set up flannel with Virtual Extensible LAN (VXLAN) back end and use it with Docker on OpenPOWER servers. More technical details on flannel is available on its GitHub page.

Where to get etcd and flannel binaries?

Etcd can be built from source. Alternatively, binary packages for Ubuntu (16.04 +) and Fedora (24 +) are available from distribution repositories. Binary packages for RHEL 7.X LE are available at the Unicamp repository or IBM yum repository.

Similarly, flannel can also be built from source. Binary packages for Fedora is available from distribution repository. Ubuntu packages are not yet available. Binary packages for RHEL 7 LE is available at Unicamp repository or IBM yum repository.

The steps below assume that etcd and flannel have been installed either from source or via binary packages on your Linux instance.

Steps to set up flannel overlay network

This section explains the steps required to set up flannel overlay network.

Step 1. Set up etcd

If you are installing etcd through RPM or DEB packages, then you can find the etcd configuration under /etc/etcd/etcd.conf for Red Hat based systems and under /etc/default/etcd.conf for Ubuntu. Ensure that you at least set up ETCD_LISTEN_CLIENT_URLS and ETCD_ADVERTISE_CLIENT_URLS correctly so that the remote clients can reach the etcd server.

My test setup includes the following values:

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:4001"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:4001, http://192.168.122.173:4001"

Step 2. Start etcd

Start the etcd daemon using the following command:

# systemctl restart etcd

Step 3. Set up flannel network configuration in etcd

Perform the following steps to set up flannel network configuration:

1. Create a network configuration JSON file for the overlay network. Refer to the following example configuration:

   # cat flannel-config.json
   {
     "Network": "10.20.0.0/16",
     "SubnetLen": 24,
     "Backend": {
       "Type": "vxlan",
       "VNI": 1
     }
   }

2. The Type attribute specifies the VXLAN back end. This requires the host kernel to have VXLAN support. As a quick way to validate VXLAN support in the host kernel, run the following command:

   # cat /boot/config-`uname -r` | grep CONFIG_VXLAN
   CONFIG_VXLAN=m

3. Add the network configuration to etcd.

   # etcdctl set kubernetes-cluster/network/config < flannel-config.json

4. Use an etcd key prefix of your choice instead of kubernetes-cluster/network.
   You can validate the etcd entry by running the following command:

   # etcdctl get kubernetes-cluster/network/config
   
   {
     "Network": "10.20.0.0/16",
     "SubnetLen": 24,
     "Backend": {
       "Type": "vxlan",
       "VNI": 1
     }
   }

Step 4. Start flannel

At a minimum, the location of the etcd server and the etcd key prefix need to be provided to the flannel daemon. If running flannel through system, then the values needs to be provided through the appropriate configuration file. For Red Hat based systems, the configuration file is /etc/sysconfig/flanneld. Ensure that the values for FLANNEL_ETCD and FLANNEL_ETCD_KEY are correctly specified.
Here are the values from my test setup:

FLANNEL_ETCD="http://192.168.122.173:4001"
FLANNEL_ETCD_KEY="/kubernetes-cluster/network"

Start the flannel daemon using the following command:

# systemctl restart flanneld

Following is an example of running flannel daemon directly:

# flanneld -etcd-endpoints=http://192.168.122.173:4001 -etcd-prefix=kubernetes-cluster/network

If you are planning to set up flannel in Ubuntu, then you might need to use the following content and create flanneld.service under /lib/systemd/system/flanneld.service.

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
EnvironmentFile=-/etc/default/flanneld
ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

The /etc/default/flanneld configuration file looks like the following:

# Flanneld configuration options
# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://192.168.122.173:4001"
# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/kubernetes-cluster/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

Step 5. Configure Docker to use flannel

Flannel daemon writes host specific flannel network configuration under /run/flannel/subnet.env. Additionally, it also creates a file, /run/flannel/docker, with the environment variables required for the Docker daemon.

For example, refer to the following output from two systems in my test setup:

# hostname
pkb-rhel71-1.kube.com
# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.20.0.0/16
FLANNEL_SUBNET=10.20.31.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

# cat /run/flannel/docker
DOCKER_OPT_BIP="--bip=10.20.31.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.20.31.1/24 --ip-masq=true --mtu=1450 "


# hostname
pkb-ubuntu1604-1.kube.com
# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.20.0.0/16
FLANNEL_SUBNET=10.20.48.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

# cat /run/flannel/docker
DOCKER_OPT_BIP="--bip=10.20.48.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.20.48.1/24 --ip-masq=true --mtu=1450 "

On each host, values for FLANNEL_SUBNET and FLANNEL_MTU needs to be used with --bip and --mtu Docker daemon options respectively.

When using systemd, the recommended way to specify the above options for the Docker daemon is to use a systemd drop-in file.

Refer to the following drop-in file example for the Docker daemon service.

# cat /etc/systemd/system/docker.service.d/docker.conf
[Service]
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
EnvironmentFile=-/run/flannel/docker
ExecStart=
ExecStart=/usr/bin/dockerd $OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY

Conclusion

Hope the configuration steps explained in this article help you to get started with flannel overlay network on Power servers.

Downloadable resources

• PDF of this content