IBM Cloud Private deployment topologies that use Calico
Depending on your environment, you can use Calico for different IBM Cloud Private deployment topologies.
Calico network via specific NICs
Because of consistent network device naming, network interface controllers (NICs) can be named differently for each cluster node, but are physically connected to the same network. For example:
- master -
ens192
- worker1 -
ens168
- worker2 -
ens168
- worker3 -
ens172
In this scenario, install IBM Cloud Private by using the following config.yaml
configuration:
calico_ip_autodetection_method: interface=ens192,ens168,ens172
Note: If a node has multiple matches in the list, the first one is considered to create the Calico network mesh. This can, however, break the Calico network mesh if unintended interfaces are used. This method is suitable only when a network interface matches only one value in the list.
Separate network for pod-to-pod communication and control plane by using multiple NICs
In environments where multiple networks are available, IBM Cloud Private can be configured to use separate physical networks for control and data planes so that the Kubernetes management traffic is on a separate layer 2 or 3 network segment. In
this scenario two networks are configured, 192.168.30.0/24
on eth0
for management traffic, and 172.68.32.0/24
for application traffic.
In this scenario, install IBM Cloud Private by using the following configuration:
cluster/hosts
:
The network that each kubelet
communicates with the kube-apiserver
on uses the management subnet IP address on 192.168.30.0/24
:
[master]
192.168.30.10
[proxy]
192.168.30.10
[worker]
192.168.30.20
192.168.30.21
192.168.30.22
cluster/config.yaml
:
Configure the Calico IP mesh either to use the data network interface eth1
or to specify the interface that routes the data network 172.68.32.0/24
.
calico_ip_autodetection_method: interface=eth1
Or
calico_ip_autodetection_method: can-reach=<Gateway IP for subnet 172.68.32.0/24>
Workload isolation by using different layer 2 network segments
In multi-tenant scenarios or in scenarios where compliance requires that some workloads need to be isolated from each other, deploy workloads on separate layer 2 segments. Use a router or firewall control and inspect the flow of network traffic between them.
If worker nodes and proxy nodes are already installed on separate layer 2 network segments, IBM Cloud Private can use the podNodeSelector
admission controller to deploy pods to a particular namespace target or a particular group of
worker nodes. In cases where stateful packet inspection is required for traffic that flows between groups of worker nodes, it is necessary for IP-in-IP encapsulation to be disabled and the router joins the BGP mesh.
For more information on how to configure IBM Cloud Private during installation for isolation, see Isolating network and compute environments.
Pod-to-pod communication encryption with IPsec
Network traffic can be secured if all of the applications that are running inside of pods implement SSL or TLS encryption. This scenario is not always possible, for example when you are running third-party Helm charts that do not support TLS-encrypted connections. To secure pod-to-pod traffic, IBM Cloud Private can be configured to encrypt the pod mesh network by using IPsec in transport mode so that all pod traffic is transparently encrypted. This configuration might be the setup you want in shared untrusted networks like a public cloud.
Enabling the IPsec mesh requires at least two network interfaces; one for management traffic and one for data plane. IPsec also requires IP-in-IP mode to be enabled and adds additional CPU overhead to perform encryption and decryption of pod traffic. For more information, see Encrypting cluster data network traffic with IPsec.
cluster/hosts
[master]
192.168.30.10
[proxy]
192.168.30.10
[worker]
192.168.30.20
192.168.30.21
192.168.30.22
cluster/config.yaml
calico_ipip_enabled: true
calico_tunnel_mtu: 1390
calico_ip_autodetection_method: interface=eth1
ipsec_mesh:
enable: true
interface: eth1
subnets: [172.68.32.0/24]
exclude_ips: [172.68.32.1, 172.68.32.2, 172.68.32.3]
cipher_suite: aes128gcm16!
-
calico_ipip_enabled
must be set to true to enable IPsec. -
calico_tunnel_mtu
must be adjusted such that it can accommodate Calico IP-in-IP (20 bytes) and the additional IPsec header (40 bytes) = 60 bytes. It must be at least 60 bytes less compared to the MTU size of the data network interface that you provided. To mitigate performance problems, jumbo frames (large MTU) can be enabled on the data network interface and uplink to reduce the probability of packet fragmentation between nodes. -
calico_ip_autodetection_method
must always specify the interface on which IPsec is enabled, that is, the data network. -
ipsec_mesh.enable
must be set totrue
. -
ipsec_mesh.interface
must be set to the same value ascalico_ip_autodetection_method: interface=...
. Post 3.1.1, this value is deprecated andcalico_ip_autodetection_method: interface=...
is used to identify the interface on which IPsec is enabled. -
ipsec_mesh.subnets
is the list of subnets for which the IPsec must be enabled. You can provide comma-separated values like[172.68.32.20/30, 172.68.32.48/29, 172.68.32.100/31]
-
ipsec_mesh.exclude_ips
is the list of IPs to be excluded from the IPsec subnet and packets to those IP addresses are not encrypted. For example, you can include the network management endpoints such as DHCP, DNS, and Gateway IP, in the exclusion list to not encrypt packets that are destined to these endpoints. Leave the list empty if you want all traffic to be encrypted. -
ipsec_mesh.cipher_suite
specifies the ESP encryption/authentication algorithms to be used during encryption. The default for Ubuntu isaes128gcm16!
and RHEL isaes_gcm_c128
. Ensure that the appropriate crypto kernel modules are loaded.