Configuring flannel overlay network with VXLAN for Docker on IBM Power Systems servers
Flannel provides a configurable virtual overlay network for use with containers. For a multihost container cluster, this provides an easy way to have a routable IP address inside the cluster. Flannel uses etcd distributed key-value store to save the network configurations and make it available to different hosts in the cluster. In this article, we'll see how to set up flannel with Virtual Extensible LAN (VXLAN) back end and use it with Docker on OpenPOWER servers. More technical details on flannel is available on its GitHub page.
Where to get etcd and flannel binaries?
Etcd can be built from source. Alternatively, binary packages for Ubuntu (16.04 +) and Fedora (24 +) are available from distribution repositories. Binary packages for RHEL 7.X LE are available at the Unicamp repository or IBM yum repository.
Similarly, flannel can also be built from source. Binary packages for Fedora is available from distribution repository. Ubuntu packages are not yet available. Binary packages for RHEL 7 LE is available at Unicamp repository or IBM yum repository.
The steps below assume that etcd and flannel have been installed either from source or via binary packages on your Linux instance.
Steps to set up flannel overlay network
This section explains the steps required to set up flannel overlay network.
Step 1. Set up etcd
If you are installing etcd through RPM or DEB packages, then you can find the etcd
configuration under /etc/etcd/etcd.conf for Red Hat based systems and under
/etc/default/etcd.conf for Ubuntu. Ensure that you at least set up
ETCD_LISTEN_CLIENT_URLS
and ETCD_ADVERTISE_CLIENT_URLS
correctly so that the remote clients
can reach the etcd server.
My test setup includes the following values:
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:4001" ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:4001, http://192.168.122.173:4001"
Step 2. Start etcd
Start the etcd daemon using the following command:
# systemctl restart etcd
Step 3. Set up flannel network configuration in etcd
Perform the following steps to set up flannel network configuration:
- Create a network configuration JSON file for the overlay network. Refer to the following
example configuration:
# cat flannel-config.json { "Network": "10.20.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }
- The
Type
attribute specifies the VXLAN back end. This requires the host kernel to have VXLAN support. As a quick way to validate VXLAN support in the host kernel, run the following command:# cat /boot/config-`uname -r` | grep CONFIG_VXLAN CONFIG_VXLAN=m
- Add the network configuration to etcd.
# etcdctl set kubernetes-cluster/network/config < flannel-config.json
- Use an etcd key prefix of your choice instead of kubernetes-cluster/network.
You can validate the etcd entry by running the following command:# etcdctl get kubernetes-cluster/network/config { "Network": "10.20.0.0/16", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }
Step 4. Start flannel
At a minimum, the location of the etcd server and the etcd key prefix need to be provided
to the flannel daemon. If running flannel through system, then the values needs to be
provided through the appropriate configuration file. For Red Hat based systems, the
configuration file is /etc/sysconfig/flanneld. Ensure that the values for FLANNEL_ETCD
and
FLANNEL_ETCD_KEY
are correctly specified.
Here are the values from my test setup:
FLANNEL_ETCD="http://192.168.122.173:4001" FLANNEL_ETCD_KEY="/kubernetes-cluster/network"
Start the flannel daemon using the following command:
# systemctl restart flanneld
Following is an example of running flannel daemon directly:
# flanneld -etcd-endpoints=http://192.168.122.173:4001 -etcd-prefix=kubernetes-cluster/network
If you are planning to set up flannel in Ubuntu, then you might need to use the following
content and create flanneld.service
under /lib/systemd/system/flanneld.service.
[Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service [Service] Type=notify EnvironmentFile=-/etc/default/flanneld ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS Restart=on-failure [Install] WantedBy=multi-user.target RequiredBy=docker.service
The /etc/default/flanneld configuration file looks like the following:
# Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD="http://192.168.122.173:4001" # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY="/kubernetes-cluster/network" # Any additional options that you want to pass #FLANNEL_OPTIONS=""
Step 5. Configure Docker to use flannel
Flannel daemon writes host specific flannel network configuration under /run/flannel/subnet.env. Additionally, it also creates a file, /run/flannel/docker, with the environment variables required for the Docker daemon.
For example, refer to the following output from two systems in my test setup:
# hostname pkb-rhel71-1.kube.com # cat /run/flannel/subnet.env FLANNEL_NETWORK=10.20.0.0/16 FLANNEL_SUBNET=10.20.31.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false # cat /run/flannel/docker DOCKER_OPT_BIP="--bip=10.20.31.1/24" DOCKER_OPT_IPMASQ="--ip-masq=true" DOCKER_OPT_MTU="--mtu=1450" DOCKER_NETWORK_OPTIONS=" --bip=10.20.31.1/24 --ip-masq=true --mtu=1450 " # hostname pkb-ubuntu1604-1.kube.com # cat /run/flannel/subnet.env FLANNEL_NETWORK=10.20.0.0/16 FLANNEL_SUBNET=10.20.48.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false # cat /run/flannel/docker DOCKER_OPT_BIP="--bip=10.20.48.1/24" DOCKER_OPT_IPMASQ="--ip-masq=true" DOCKER_OPT_MTU="--mtu=1450" DOCKER_NETWORK_OPTIONS=" --bip=10.20.48.1/24 --ip-masq=true --mtu=1450 "
On each host, values for FLANNEL_SUBNET
and FLANNEL_MTU
needs to be used with --bip
and
--mtu
Docker daemon options respectively.
When using systemd
, the recommended way to specify the above options for the Docker daemon
is to use a systemd
drop-in file.
Refer to the following drop-in file example for the Docker daemon service.
# cat /etc/systemd/system/docker.service.d/docker.conf [Service] EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage EnvironmentFile=-/etc/sysconfig/docker-network EnvironmentFile=-/run/flannel/docker ExecStart= ExecStart=/usr/bin/dockerd $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $BLOCK_REGISTRY \ $INSECURE_REGISTRY
Conclusion
Hope the configuration steps explained in this article help you to get started with flannel overlay network on Power servers.