IBM Cloud Kubernetes Service Introduces the Static Route Operator Add-On

Manage network routes as Kubernetes resources.

Typically, the network on a Linux machine works out-of-the-box, especially when the machine is provisioned by a cloud provider. For example, in IBM Cloud Kubernetes Service, you can request a cluster by specifying some basic details—such as name, size, and location—and the result is a fully configured and secure cluster. The cluster provisioner takes care of configuring virtual interfaces, firewalls, the overlay network, the BGP mesh, network policies, cross zone communication, and routing rules. However, in some cases, you might have special requirements which are not met by the default network configuration for your cluster. To support network customization, IBM Cloud Kubernetes Service is introducing the ability to create and apply custom routing rules.

Routing rules

First, let’s talk about where routing rules are located and what are they are used for. All incoming and outgoing packages in the cluster network have at least a source and destination IP address. The path of the package, however, is calculated at every network junction according to routing rules. The Linux kernel checks the routing table on the worker node for each package and sends the package to the next junction in the network, where the process repeats until the package reaches its final destination.

In some cases, routing rules must be manually customized; for example, you might be establishing a network connection between two data centers. Customizing routing rules can introduce the following problems:

• Configuring routes requires root access to the target worker nodes, which is not permitted in IBM Cloud Kubernetes Service.
• Routing is a very sensitive part of the Linux kernel. It is easy to break an existing rule or, in a worst-case scenario, destroy the entire network configuration.
• IBM manages the underlaying infrastructure for your worker nodes, so the kernel itself is typically hidden from user management controls.

The solution

To address these issues, a Kubernetes-native component is required to automatically manage route configuration and to protect the kernel from misconfigurations. The solution is the Static Route Operator project, which is open sourced on GitHub and available as a cluster add-on for IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud clusters. The operator runs on the target Kubernetes cluster to implement Kubernetes Custom Resources definitions (CRDs) for routes.

Usage

There are two steps to configure a routing rule:

1. Enable the Static Route add-on for your cluster
   • In the IBM Cloud Kubernetes Service console or the Red Hat OpenShift on IBM Cloud console, click your cluster and click the Add-ons tab. On the Static Route card, click Install:

     

   • From the IBM Cloud CLI, run the following command:

     ibmcloud ks cluster addon enable static-route --cluster [CLUSTER_NAME]

2. From the IBM Cloud CLI, create a Custom Resource (CR) for a static route. The gateway parameter is optional and when CR omits it, the static route operator is able to dynamically detect the right value. In case of IBM Cloud, it will select the next-hop based on the route towards 10.0.0.1 IP address:

   cat <<EOF | kubectl create -f -
   apiVersion: static-route.ibm.com/v1
   kind: StaticRoute
   metadata:
     name: example-static-route
   spec:
     subnet: "192.168.0.0/24"
     gateway: "10.40.16.1"
   EOF

After you apply your static route CR, the static route operator uses the created resource to create a routing rule. In the above example, the operator would create a routing rule to route any packages with a destination IP address in the 192.168.0.0/24 CIDR to the 10.40.16.1 gateway first. Any time that you update or delete the CR, the operator makes automatic changes to the rule in the routing table.

You can also use node selectors to apply the rule to only a specific group of worker nodes. In the following example, the static route is created only on worker nodes in the dal10 location in the gateway worker pool:

apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
  name: example-static-route-with-selector
spec:
  subnet: "192.168.0.0/24"
  selectors:
    -
      key: "ibm-cloud.kubernetes.io/zone"
      operator: In
      values:
        - "dal10"
    - key: "ibm-cloud.kubernetes.io/worker-pool-name"
      operator: In
      values:
        - "gateway"

Troubleshooting

If an error occurs, first look for errors in the CR. For each node where the rule is applied, check the status.nodeStatus.error field:

kubectl get StaticRoute example-static-route-wrong-gateway -o yaml

apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
  creationTimestamp: "2020-06-06T09:11:23Z"
  generation: 1
  name: example-static-route-wrong-gateway
  resourceVersion: "3027750"
  selfLink: /apis/static-route.ibm.com/v1/staticroutes/example-static-route
  uid: e0f37d84-f9cb-4ca3-8e51-4b3940623e4a
spec:
  gateway: 10.135.31.193
  subnet: 192.168.0.0/24
status:
  nodeStatus:
  - error: Given gateway IP is not directly routable, cannot setup the route
    hostname: 10.240.0.61
    state:
      gateway: 10.135.31.193
      subnet: 192.168.0.0/24
  - error: Given gateway IP is not directly routable, cannot setup the route
    hostname: 10.240.0.89
    state:
      gateway: 10.135.31.193
      subnet: 192.168.0.0/24

If you need to dig deeper, you can check for errors in the static route operator logs:

kubectl logs -n kube-system daemonset/static-route-operator --all-containers=true

More information

For more information, check out our official documentation.

Contact us

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.