Manage network routes as Kubernetes resources.

Typically, the network on a Linux machine works out-of-the-box, especially when the machine is provisioned by a cloud provider. For example, in IBM Cloud Kubernetes Service, you can request a cluster by specifying some basic details—such as name, size, and location—and the result is a fully configured and secure cluster. The cluster provisioner takes care of configuring virtual interfaces, firewalls, the overlay network, the BGP mesh, network policies, cross zone communication, and routing rules. However, in some cases, you might have special requirements which are not met by the default network configuration for your cluster. To support network customization, IBM Cloud Kubernetes Service is introducing the ability to create and apply custom routing rules.

Routing rules

First, let’s talk about where routing rules are located and what are they are used for. All incoming and outgoing packages in the cluster network have at least a source and destination IP address. The path of the package, however, is calculated at every network junction according to routing rules. The Linux kernel checks the routing table on the worker node for each package and sends the package to the next junction in the network, where the process repeats until the package reaches its final destination.

In some cases, routing rules must be manually customized; for example, you might be establishing a network connection between two data centers. Customizing routing rules can introduce the following problems:

  • Configuring routes requires root access to the target worker nodes, which is not permitted in IBM Cloud Kubernetes Service.
  • Routing is a very sensitive part of the Linux kernel. It is easy to break an existing rule or, in a worst-case scenario, destroy the entire network configuration.
  • IBM manages the underlaying infrastructure for your worker nodes, so the kernel itself is typically hidden from user management controls.

The solution

To address these issues, a Kubernetes-native component is required to automatically manage route configuration and to protect the kernel from misconfigurations. The solution is the Static Route Operator project, which is open sourced on GitHub and available as a cluster add-on for IBM Cloud Kubernetes Service and Red Hat OpenShift on IBM Cloud clusters. The operator runs on the target Kubernetes cluster to implement Kubernetes Custom Resources definitions (CRDs) for routes.

Usage

There are two steps to configure a routing rule:

  1. Enable the Static Route add-on for your cluster
  2. From the IBM Cloud CLI, create a Custom Resource (CR) for a static route. The gateway parameter is optional and when CR omits it, the static route operator is able to dynamically detect the right value. In case of IBM Cloud, it will select the next-hop based on the route towards 10.0.0.1 IP address:
    cat <<EOF | kubectl create -f -
    apiVersion: static-route.ibm.com/v1
    kind: StaticRoute
    metadata:
      name: example-static-route
    spec:
      subnet: "192.168.0.0/24"
      gateway: "10.40.16.1"
    EOF

After you apply your static route CR, the static route operator uses the created resource to create a routing rule. In the above example, the operator would create a routing rule to route any packages with a destination IP address in the 192.168.0.0/24 CIDR to the 10.40.16.1 gateway first. Any time that you update or delete the CR, the operator makes automatic changes to the rule in the routing table.

You can also use node selectors to apply the rule to only a specific group of worker nodes. In the following example, the static route is created only on worker nodes in the dal10 location in the gateway worker pool:

apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
  name: example-static-route-with-selector
spec:
  subnet: "192.168.0.0/24"
  selectors:
    -
      key: "ibm-cloud.kubernetes.io/zone"
      operator: In
      values:
        - "dal10"
    - key: "ibm-cloud.kubernetes.io/worker-pool-name"
      operator: In
      values:
        - "gateway"

Troubleshooting

If an error occurs, first look for errors in the CR. For each node where the rule is applied, check the status.nodeStatus.error field:

kubectl get StaticRoute example-static-route-wrong-gateway -o yaml
apiVersion: static-route.ibm.com/v1
kind: StaticRoute
metadata:
  creationTimestamp: "2020-06-06T09:11:23Z"
  generation: 1
  name: example-static-route-wrong-gateway
  resourceVersion: "3027750"
  selfLink: /apis/static-route.ibm.com/v1/staticroutes/example-static-route
  uid: e0f37d84-f9cb-4ca3-8e51-4b3940623e4a
spec:
  gateway: 10.135.31.193
  subnet: 192.168.0.0/24
status:
  nodeStatus:
  - error: Given gateway IP is not directly routable, cannot setup the route
    hostname: 10.240.0.61
    state:
      gateway: 10.135.31.193
      subnet: 192.168.0.0/24
  - error: Given gateway IP is not directly routable, cannot setup the route
    hostname: 10.240.0.89
    state:
      gateway: 10.135.31.193
      subnet: 192.168.0.0/24

If you need to dig deeper, you can check for errors in the static route operator logs:

kubectl logs -n kube-system daemonset/static-route-operator --all-containers=true

More information

For more information, check out our official documentation.

Contact us

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.

Categories

More from Announcements

IBM TechXchange underscores the importance of AI skilling and partner innovation

3 min read - Generative AI and large language models are poised to impact how we all access and use information. But as organizations race to adopt these new technologies for business, it requires a global ecosystem of partners with industry expertise to identify the right enterprise use-cases for AI and the technical skills to implement the technology. During TechXchange, IBM's premier technical learning event in Las Vegas last week, IBM Partner Plus members including our Strategic Partners, resellers, software vendors, distributors and service…

Introducing Inspiring Voices, a podcast exploring the impactful journeys of great leaders

< 1 min read - Learning about other people's careers, life challenges, and successes is a true source of inspiration that can impact our own ambitions as well as life and business choices in great ways. Brought to you by the Executive Search and Integration team at IBM, the Inspiring Voices podcast will showcase great leaders, taking you inside their personal stories about life, career choices and how to make an impact. In this first episode, host David Jones, Executive Search Lead at IBM, brings…

IBM watsonx Assistant and NICE CXone combine capabilities for a new chapter in CCaaS

5 min read - In an age of instant everything, ensuring a positive customer experience has become a top priority for enterprises. When one third of customers (32%) say they will walk away from a brand they love after just one bad experience (source: PWC), organizations are now applying massive investments to this experience, particularly with their live agents and contact centers.  For many enterprises, that investment includes modernizing their call centers by moving to cloud-based Contact Center as a Service (CCaaS) platforms. CCaaS solutions…

See what’s new in SingleStoreDB with IBM 8.0

3 min read - Despite decades of progress in database systems, builders have compromised on at least one of the following: speed, reliability, or ease. They have two options: one, they could get a document database that is fast and easy, but can’t be relied on for mission-critical transactional applications. Or two, they could rely on a cloud data warehouse that is easy to set up, but only allows lagging analytics. Even then, each solution lacks something, forcing builders to deploy other databases for…