Manage network routes as Kubernetes resources.
Typically, the network on a Linux machine works out-of-the-box, especially when the machine is provisioned by a cloud provider. For example, in IBM Cloud Kubernetes Service, you can request a cluster by specifying some basic details—such as name, size, and location—and the result is a fully configured and secure cluster. The cluster provisioner takes care of configuring virtual interfaces, firewalls, the overlay network, the BGP mesh, network policies, cross zone communication, and routing rules. However, in some cases, you might have special requirements which are not met by the default network configuration for your cluster. To support network customization, IBM Cloud Kubernetes Service is introducing the ability to create and apply custom routing rules.
First, let’s talk about where routing rules are located and what are they are used for. All incoming and outgoing packages in the cluster network have at least a source and destination IP address. The path of the package, however, is calculated at every network junction according to routing rules. The Linux kernel checks the routing table on the worker node for each package and sends the package to the next junction in the network, where the process repeats until the package reaches its final destination.
In some cases, routing rules must be manually customized; for example, you might be establishing a network connection between two data centers. Customizing routing rules can introduce the following problems:
- Configuring routes requires root access to the target worker nodes, which is not permitted in IBM Cloud Kubernetes Service.
- Routing is a very sensitive part of the Linux kernel. It is easy to break an existing rule or, in a worst-case scenario, destroy the entire network configuration.
- IBM manages the underlaying infrastructure for your worker nodes, so the kernel itself is typically hidden from user management controls.
To address these issues, a Kubernetes-native component is required to automatically manage route configuration and to protect the kernel from misconfigurations. The solution is the Static Route Operator project, which is open sourced on GitHub and available as a cluster add-on for IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud clusters. The operator runs on the target Kubernetes cluster to implement Kubernetes Custom Resources definitions (CRDs) for routes.
There are two steps to configure a routing rule:
- Enable the Static Route add-on for your cluster
- From the IBM Cloud CLI, create a Custom Resource (CR) for a static route. The gateway parameter is optional and when CR omits it, the static route operator is able to dynamically detect the right value. In case of IBM Cloud, it will select the next-hop based on the route towards 10.0.0.1 IP address:
After you apply your static route CR, the static route operator uses the created resource to create a routing rule. In the above example, the operator would create a routing rule to route any packages with a destination IP address in the 192.168.0.0/24 CIDR to the 10.40.16.1 gateway first. Any time that you update or delete the CR, the operator makes automatic changes to the rule in the routing table.
You can also use node selectors to apply the rule to only a specific group of worker nodes. In the following example, the static route is created only on worker nodes in the
dal10 location in the
gateway worker pool:
If an error occurs, first look for errors in the CR. For each node where the rule is applied, check the
If you need to dig deeper, you can check for errors in the static route operator logs:
For more information, check out our official documentation.