Isolating Clusters with Calico Policies in IBM Cloud Kubernetes Service

4 min read

By: Rachael Graham

IBM Cloud Kubernetes Service now provides sets of Calico network policies to isolate your cluster on public and private networks.

Every IBM Cloud Kubernetes Service cluster is created with the Calico network plugin. Default Calico network policies are set up to secure the public network interface of every worker node in the cluster. If you have specific security requirements, you can create custom Calico network policies, such as blocking inbound (ingress) traffic to load balancer services exposing your apps. 

However, if you want to completely lock down egress from your cluster, you must write several Calico policies to deny traffic that you don't want, while still allowing all traffic that is required for your cluster to function. IBM Cloud Kubernetes Service now offers sets of pre-written Calico policies that isolate your cluster on the public and private network so that you don't have to.

Why should I isolate my clusters on the public or private network?

Consider a scenario in which you want to allow your cluster limited public access. You create worker nodes that are connected to private VLANs only and then create a worker pool of edge nodes that are connected to public and private VLANs. 

Edge nodes can improve the security of your cluster by allowing only a few worker nodes to be accessed externally and by isolating the networking workload to these workers. However, you want to make sure that only the minimally-required traffic is permitted to these edge nodes—like user requests to your apps through your Ingress service—while all other traffic is blocked. Or, you might want to allow pods to access other IBM Cloud services, like IBM Log Analysis with LogDNA, but want to make sure all other pod egress to public endpoints is blocked.

You might also want to lock down your cluster on the private network. For example, when you enable Virtual Routing and Forwarding (VRF) or VLAN spanning to allow worker nodes to communicate with each other on the private network, any instance that is connected to any of the private VLANs in the same IBM Cloud account can communicate with your worker nodes. Using Calico policies lets you isolate your cluster from other systems on the private network that you don't want to access your cluster.

Isolating clusters on the public network

This set of Calico policies work in conjunction with the default Calico policies to protect the public network traffic of a cluster while allowing communication on the public network that is necessary for the cluster to function. The policies target the public interface and the pod network of a cluster.

Get the set of public network Calico policies from the IBM Cloud kube-samples Git repository.

git clone https://github.com/IBM-Cloud/kube-samples.git

The policies are organized for each IBM Cloud Kubernetes Service region. Navigate to the directory for cluster's region, such as us-south.

cd ../IBM-Cloud/kube-samples/calico-policies/public-network-isolation/us-south

You can then use the Calico command line tool, calicoctl, to apply the policies in your cluster. Example:

calicoctl apply -f allow-egress-pods-public.yaml

Required policies

  • allow-egress-pods-public: Opens ports that are necessary for pods to function properly and allows pods to communicate with other pods in the cluster.
  • allow-ibm-ports-public: Opens ports that are necessary for worker nodes to function properly, including updates from IBM Cloud for your cluster.
  • allow-public-service-endpoint: Allows worker nodes to communicate with the cluster master through the public service endpoint.
  • deny-all-outbound-public: Denies all egress from worker nodes that is not allowed by the other egress policies.

Optional policies

  • allow-public-services: Allows workers to access specified IBM Cloud services over the public network.
  • allow-public-services-pods: Allows pods to access specified IBM Cloud services over the public network.

Isolating clusters on the private network

This set of Calico policies and host endpoints isolate the private network traffic of a cluster from other resources in the account's private network, while allowing communication on the private network that is necessary for the cluster to function. The policies target the private interface and the pod network of a cluster.

Get the set of private network Calico policies from the IBM Cloud kube-samples Git repository.

git clone https://github.com/IBM-Cloud/kube-samples.git

The policies are organized for each IBM Cloud Kubernetes Service region. Navigate to the directory for cluster's region, such as us-south.

cd ../IBM-Cloud/kube-samples/calico-policies/private-network-isolation/calico-v3/us-south

You can then use the Calico command line tool, calicoctl, to apply the policies in your cluster. Example:

calicoctl apply -f allow-all-workers-private.yam

Required policies

  • allow-all-workers-private: Limits worker node communication on the private network to other worker nodes and pods within the cluster.
  • allow-egress-pods-private: Opens ports that are necessary for pods to function properly and allows pods to communicate with other pods in the cluster. Also blocks pod egress to the 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16 private networks. If worker nodes are connected to a public VLAN, pod egress is permitted to public networks.
  • allow-ibm-ports-private: Opens ports that are necessary for worker nodes to function properly.
  • allow-icmp-private: Opens the ICMP protocol to allow infrastructure health monitoring.
  • allow-private-service-endpoint: Allows worker nodes to communicate with the cluster master through the private service endpoint.
  • allow-sys-mgmt-private: Allows egress to the IBM Cloud Classic infrastructure private subnets so that you can create worker nodes in your cluster.
  • generic-privatehostendpoint: Sets up private host endpoints for your worker nodes so that the other policies in this set can target the worker node private interface (eth0) of worker nodes. Note: Each time you add a worker node to a cluster, you must update the host endpoints file with the new entries.
  • allow-vrrp-private: Opens the VRRP protocol to use network load balancer (NLB) and Ingress application load balancer (ALB) services for app networking.

Optional policies

  • allow-private-services: Allows workers to access other IBM Cloud services that support communication over the private network through private service endpoints.
  • allow-private-services-pods: Allows pods to access other IBM Cloud services that support communication over the private network through private service endpoints.

More details

For more information, including how to set up calicoctl and how to apply the policies in your cluster, see the IBM Cloud Kubernetes Service documentation.

Contact us

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.

Be the first to hear about news, product updates, and innovation from IBM Cloud