Using GRE with PowerKVM 2.1.1 – A Openstack Neutron Lab Simulation.

Body

Using GRE with PowerKVM 2.1.1 – A Openstack Neutron Lab Simulation.

Before we begin there are few things to note:

1. OpenVswitch can be configured both as a FLOW as well as a NORMAL switch.

2. By default every switch created using “add-br” is a NORMAL vswitch.

3. This is not a GRE tutorial but a hands-on lab to create a host to host GRE tunnel and prove that it works via Virtual Machine connectivity and requires the primary knowledge of GRE.

Typical Openstack installations create 3 bridges – br-int/br-tun and br-ex.

Each bridge has it's own functionality:

1. br-tun – This is the tunnel bridge. It is used to create the point to point Tunnels across hypervisors. This bridge operates in the FLOW mode.

2. br-int – This is the integration bridge. It is used to host all the Virtual Machine veth pairs. The bridge typically operates using VLANs and in NORMAL mode.

3. br-ex – This is the external bridge. This bridge is used to communicate with the internet and can be used to host ports of a virtual router etc.

We would create a very similar setup using the following commands:

ovs-vsctl add-br br-int
ovs-vsctl add-br br-ex
ovs-vsctl add-br br-tun
ovs-vsctl add-port br-ex eth0
ifconfig br-tun 0
ifconfig br-ex 10.10.1.9 netmask 255.255.255.0
ovs-vsctl add-port br-int int-br-ex
ovs-vsctl set interface int-br-ex type=patch
ovs-vsctl add-port br-int patch-tun
ovs-vsctl set interface patch-tun type=patch
ovs-vsctl add-port default phy-br-ex
ovs-vsctl set interface phy-br-ex type=patch
ovs-vsctl add-port br-tun patch-int
ovs-vsctl set interface patch-int type=patch
ovs-vsctl set interface patch-int options:peer=patch-tun
ovs-vsctl set interface patch-tun options:peer=patch-int
ifconfig eth0 up
ovs-vsctl set interface int-default options:peer=phy-br-ex
ovs-vsctl set interface phy-br-ex options:peer=int-br-ex
ovs-ofctl del-flows br-tun

ovs-vsctl add-port br-tun vtep -- set interface vtep type=gre option:remote_ip=10.10.1.8 option:key=flow ofport_request=3

Executing these commands would give us a topology:

ovs-vsctl show

91bc1bbb-36d2-4d17-a42c-2c36819adbc1

Bridge br-int

Port "vnet0"

Interface "vnet0"

Port br-int

Interface br-int

type: internal

Port int-br-ex

Interface int-br-ex

type: patch

options: {peer=phy-br-ex}

Port patch-tun

Interface patch-tun

type: patch

options: {peer=patch-int}

Bridge br-tun

Port patch-int

Interface patch-int

type: patch

options: {peer=patch-tun}

Port vtep

Interface vtep

type: gre

options: {key=flow, remote_ip="10.10.1.8"}

Port br-tun

Interface br-tun

type: internal

Bridge br-ext

Port phy-br-ex

Interface phy-br-ex

type: patch

options: {peer=int-br-ex}

Port br-ex

Interface br-ex

type: internal

Port "eth0"

Interface "eth0"

The patch ports are used to mirror traffics across two bridges. We want the traffic coming br-int to be mirrored into br-tun – so that the openflow rules can be put to action. So if a VM has to send out a packet – it would be sent via the OpenFlow rules chain and then sent out of the box. Similarly a packet arriving would have to go through a set of rules to ultimately reach the right VM.

Below is a sample set of rules – Openstack Neutron would typically setup when the neutron-openvswitch agent is booted.

table=0,priority=1,in_port=1,actions=resubmit(,2)

table=0,priority=0,actions=drop

priority=1,in_port=3,actions=resubmit(,2)

table=2,priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00,actions=resubmit(,20)

table=2,priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00,actions=resubmit(,21)

table=2,priority=1,tun_id=100,actions=resubmit(,10)

table=3,priority=0,actions=drop

table=4,priority=0,actions=drop

table=10,priority=1,actions=learn(table=20,priority=1,hard_timeout=300,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

table=20,priority=0,actions=resubmit(,21)

table=21,actions=set_field:100->tun_id,output:3

table=21,priority=0,actions=drop

This article is not meant to give a tutorial on Openflow and hence we would not get into the extreme details of things of what these rules are. But here's a brief idea on what it is trying to do:

1. Table 0 is where all the packets arrive. So we are listening to all the packets coming on port number 1 (patch port) and re-submitting them to table 2 and further processing is done on it.

2. The ouput from this bridge is being sent out of port 3 – which is the gre port. We are sending out the traffic with a GRE tunnel id of 100 out of the port.

Save these flows in a file and name it as neutron_flows.txt. Add these rules to the br-tun bridge by:

ovs-ofctl add-flows br-tun neutron_flows.txt

A similar configuration needs to be repeated at another PowerKVM compute host on which we want to form the point to point tunnel and hence all the steps would have to be performed there as well with the remote_ip reversed.

Once you are done doing this. Spin up a couple of a VMs on each of these PowerKVM nodes and assign them a completely non-routable IP addresses in context of each other. These two Virtual machine should be able to ping each other with their private Ips.

Thus we have isolated networks on the VMs but have provided L2 adjanceny via GRE tunnels for tenant ID 100. (Assuming your client has created a tunnel with GRE id 100).

Tips:

1. It is always good to ensure that the GRE kernel module is loaded via lsmod | grep GRE.

2. Verify that the GRE ports are properly created by issuing ovs-ofctl show br-tun.

3. Use tcpdump or any other packet tracing tool to verify the the packet flows. The traffic can be traced with private VM Ips as they leave br-int and the same packets would have the GRE header and the host IP encapsulation when it leaves the system on br-ex.