Investigating Network Performance in an IBM Cloud Kubernetes Service Cluster

1 min read

Sometimes network connections on your Kubernetes cluster can be very slow, and you need to understand where there are bottlenecks.

Since IBM Cloud Kubernetes Service is a managed service, debugging the network can be very difficult. In this post, we will show you how to debug network performance on your own by using open source tools.

iPerf3

iPerf3 is an open source software used frequently to measure the maximum bandwidth allowed for a network interface. This is not installed by default in every container image, so in order to install, you will need to define a container image by yourself.

For example:

FROM nginx
RUN apt-get update -y && apt-get install iperf3 -y

If you would like to use a predefined container image, you could use the iperf image from ClearLinux.

Once you have the container image, you have to deploy it into your IBM Cloud Kubernetes Service cluster. To do that, you can use a DaemonSet resource: 

apiVersion: apps/v1
kind: DaemonSet
metadata:
   name: iperf3
   labels:
      app: iperf3
spec:
   selector:
      matchLabels:
        app: iperf3
   template:
      metadata:
         labels:
            app: iperf3
      spec:
         containers:
         -  name: iperf3
            image: clearlinux/iperf:3
            command: ['/bin/sh', '-c', 'sleep 1d']
            ports:
            - containerPort: 5201

As per DaemonSet definition, one replica for each node will be created.

Tests

Once the DaemonSet is ready, list all pods created:

kubectl get pods -o wide -l app=iperf3

The output should be something similar to this:

NAME           READY     STATUS    RESTARTS   AGE       IP              NODE             NOMINATED NODE
iperf3-29dwc   1/1       Running   0          4m        172.30.24.203   10.187.165.223   <none>
iperf3-2xctw   1/1       Running   0          4m        172.30.20.85    10.94.110.40     <none>
iperf3-8zlhb   1/1       Running   0          4m        172.30.106.81   10.94.110.35     <none>
iperf3-hsv7g   1/1       Running   0          4m        172.30.133.17   10.185.69.203    <none>
iperf3-kx6h6   1/1       Running   0          4m        172.30.177.47   10.187.165.224   <none>
iperf3-wzkp5   1/1       Running   0          4m        172.30.32.22    10.185.69.204    <none>

Then, select one pod as master (which will receive all the packets that are sent) and the others as clients.

Opening a terminal, run the following on the master pod:

kubectl exec -it POD_NAME -- iperf3 -s -f M

Opening a separate terminal, run the following for each client pod:

kubectl exec -it POD_NAME -- iperf3 -c MASTER_POD_IP -f M

The result should be something similar to the following:

Connecting to host 172.30.70.199, port 5201
[  5] local 172.30.35.143 port 34202 connected to 172.30.70.199 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   617 MBytes   617 MBytes/sec   41   1.43 MBytes
[  5]   1.00-2.00   sec   714 MBytes   714 MBytes/sec    0   1.75 MBytes
[  5]   2.00-3.00   sec   635 MBytes   635 MBytes/sec   74   1.12 MBytes
[  5]   3.00-4.00   sec   669 MBytes   669 MBytes/sec   49   1.17 MBytes
[  5]   4.00-5.00   sec   649 MBytes   648 MBytes/sec  340    908 KBytes
[  5]   5.00-6.00   sec   641 MBytes   642 MBytes/sec    0   1.32 MBytes
[  5]   6.00-7.00   sec   649 MBytes   649 MBytes/sec    0   1.63 MBytes
[  5]   7.00-8.00   sec   670 MBytes   670 MBytes/sec   57   1.42 MBytes
[  5]   8.00-9.00   sec   661 MBytes   661 MBytes/sec   11   1.26 MBytes
[  5]   9.00-10.00  sec   634 MBytes   634 MBytes/sec  362    969 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.38 GBytes   654 MBytes/sec  934             sender
[  5]   0.00-10.04  sec  6.38 GBytes   650 MBytes/sec                  receiver
iperf Done.

Conclusion

This is just one way to debug network performances in a Kubernetes cluster.

If the output of iPerf tests return low bitrates, there is something misconfigured on the cluster, and you should immediately ask to IBM Cloud support to have a look.

Engage the support via the IBM Cloud Support Center.

Be the first to hear about news, product updates, and innovation from IBM Cloud