Manage network routes as Kubernetes resources.

Typically, the network on a Linux machine works out-of-the-box, especially when the machine is provisioned by a cloud provider. For example, in IBM Cloud Kubernetes Service, you can request a cluster by specifying some basic details—such as name, size, and location—and the result is a fully configured and secure cluster. The cluster provisioner takes care of configuring virtual interfaces, firewalls, the overlay network, the BGP mesh, network policies, cross zone communication, and routing rules. However, in some cases, you might have special requirements which are not met by the default network configuration for your cluster. To support network customization, IBM Cloud Kubernetes Service is introducing the ability to create and apply custom routing rules.

Routing rules

First, let’s talk about where routing rules are located and what are they are used for. All incoming and outgoing packages in the cluster network have at least a source and destination IP address. The path of the package, however, is calculated at every network junction according to routing rules. The Linux kernel checks the routing table on the worker node for each package and sends the package to the next junction in the network, where the process repeats until the package reaches its final destination.

In some cases, routing rules must be manually customized; for example, you might be establishing a network connection between two data centers. Customizing routing rules can introduce the following problems:

  • Configuring routes requires root access to the target worker nodes, which is not permitted in IBM Cloud Kubernetes Service.
  • Routing is a very sensitive part of the Linux kernel. It is easy to break an existing rule or, in a worst-case scenario, destroy the entire network configuration.
  • IBM manages the underlaying infrastructure for your worker nodes, so the kernel itself is typically hidden from user management controls.

The solution

To address these issues, a Kubernetes-native component is required to automatically manage route configuration and to protect the kernel from misconfigurations. The solution is the Static Route Operator project, which is open sourced on GitHub and available as a cluster add-on for IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud clusters. The operator runs on the target Kubernetes cluster to implement Kubernetes Custom Resources definitions (CRDs) for routes.

Usage

There are two steps to configure a routing rule:

  1. Enable the Static Route add-on for your cluster
  2. From the IBM Cloud CLI, create a Custom Resource (CR) for a static route. The gateway parameter is optional and when CR omits it, the static route operator is able to dynamically detect the right value. In case of IBM Cloud, it will select the next-hop based on the route towards 10.0.0.1 IP address:
    cat <<EOF | kubectl create -f -
    apiVersion: static-route.ibm.com/v1
    kind: StaticRoute
    metadata:
      name: example-static-route
    spec:
      subnet: "192.168.0.0/24"
      gateway: "10.40.16.1"
    EOF

After you apply your static route CR, the static route operator uses the created resource to create a routing rule. In the above example, the operator would create a routing rule to route any packages with a destination IP address in the 192.168.0.0/24 CIDR to the 10.40.16.1 gateway first. Any time that you update or delete the CR, the operator makes automatic changes to the rule in the routing table.

You can also use node selectors to apply the rule to only a specific group of worker nodes. In the following example, the static route is created only on worker nodes in the dal10 location in the gateway worker pool:

apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
  name: example-static-route-with-selector
spec:
  subnet: "192.168.0.0/24"
  selectors:
    -
      key: "ibm-cloud.kubernetes.io/zone"
      operator: In
      values:
        - "dal10"
    - key: "ibm-cloud.kubernetes.io/worker-pool-name"
      operator: In
      values:
        - "gateway"

Troubleshooting

If an error occurs, first look for errors in the CR. For each node where the rule is applied, check the status.nodeStatus.error field:

kubectl get StaticRoute example-static-route-wrong-gateway -o yaml
apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
  creationTimestamp: "2020-06-06T09:11:23Z"
  generation: 1
  name: example-static-route-wrong-gateway
  resourceVersion: "3027750"
  selfLink: /apis/static-route.ibm.com/v1/staticroutes/example-static-route
  uid: e0f37d84-f9cb-4ca3-8e51-4b3940623e4a
spec:
  gateway: 10.135.31.193
  subnet: 192.168.0.0/24
status:
  nodeStatus:
  - error: Given gateway IP is not directly routable, cannot setup the route
    hostname: 10.240.0.61
    state:
      gateway: 10.135.31.193
      subnet: 192.168.0.0/24
  - error: Given gateway IP is not directly routable, cannot setup the route
    hostname: 10.240.0.89
    state:
      gateway: 10.135.31.193
      subnet: 192.168.0.0/24

If you need to dig deeper, you can check for errors in the static route operator logs:

kubectl logs -n kube-system daemonset/static-route-operator --all-containers=true

More information

For more information, check out our official documentation.

Contact us

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.

More from Announcements

IBM Hybrid Cloud Mesh and Red Hat Service Interconnect: A new era of app-centric connectivity 

2 min read - To meet customer demands, applications are expected to be performing at their best at all times. Simultaneously, applications need to be flexible and cost effective, and therefore supported by an underlying infrastructure that is equally reliant, performant and secure as the applications themselves.   Easier said than done. According to EMA's 2024 Network Management Megatrends report only 42% of responding IT professionals would rate their network operations as successful.   In this era of hyper-distributed infrastructure where our users, apps, and data…

IBM named a Leader in Gartner Magic Quadrant for SIEM, for the 14th consecutive time

3 min read - Security operations is getting more complex and inefficient with too many tools, too much data and simply too much to do. According to a study done by IBM, SOC team members are only able to handle half of the alerts that they should be reviewing in a typical workday. This potentially leads to missing the important alerts that are critical to an organization's security. Thus, choosing the right SIEM solution can be transformative for security teams, helping them manage alerts…

IBM and MuleSoft expand global relationship to accelerate modernization on IBM Power 

2 min read - As companies undergo digital transformation, they rely on APIs as the backbone for providing new services and customer experiences. While APIs can simplify application development and deliver integrated solutions, IT shops must have a robust solution to effectively manage and govern them to ensure that response times and costs are kept low for all applications. Many customers use Salesforce’s MuleSoft, named a leader by Gartner® in full lifecycle API management for seven consecutive times, to manage and secure APIs across…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters