August 8, 2020 By Gabriele de Capoa
John Power
2 min read

Sometimes network connections on your Kubernetes cluster can be very slow, and you need to understand where there are bottlenecks.

Since IBM Cloud Kubernetes Service is a managed service, debugging the network can be very difficult. In this post, we will show you how to debug network performance on your own by using open source tools.

iPerf3

iPerf3 is an open source software used frequently to measure the maximum bandwidth allowed for a network interface. This is not installed by default in every container image, so in order to install, you will need to define a container image by yourself.

For example:

FROM nginx
RUN apt-get update -y && apt-get install iperf3 -y

If you would like to use a predefined container image, you could use the iperf image from ClearLinux.

Once you have the container image, you have to deploy it into your IBM Cloud Kubernetes Service cluster. To do that, you can use a DaemonSet resource: 

apiVersion: apps/v1
kind: DaemonSet
metadata:
   name: iperf3
   labels:
      app: iperf3
spec:
   selector:
      matchLabels:
        app: iperf3
   template:
      metadata:
         labels:
            app: iperf3
      spec:
         containers:
         -  name: iperf3
            image: clearlinux/iperf:3
            command: ['/bin/sh', '-c', 'sleep 1d']
            ports:
            - containerPort: 5201

As per DaemonSet definition, one replica for each node will be created.

Tests

Once the DaemonSet is ready, list all pods created:

kubectl get pods -o wide -l app=iperf3

The output should be something similar to this:

NAME           READY     STATUS    RESTARTS   AGE       IP              NODE             NOMINATED NODE
iperf3-29dwc   1/1       Running   0          4m        172.30.24.203   10.187.165.223   <none>
iperf3-2xctw   1/1       Running   0          4m        172.30.20.85    10.94.110.40     <none>
iperf3-8zlhb   1/1       Running   0          4m        172.30.106.81   10.94.110.35     <none>
iperf3-hsv7g   1/1       Running   0          4m        172.30.133.17   10.185.69.203    <none>
iperf3-kx6h6   1/1       Running   0          4m        172.30.177.47   10.187.165.224   <none>
iperf3-wzkp5   1/1       Running   0          4m        172.30.32.22    10.185.69.204    <none>

Then, select one pod as master (which will receive all the packets that are sent) and the others as clients.

Opening a terminal, run the following on the master pod:

kubectl exec -it POD_NAME -- iperf3 -s -f M

Opening a separate terminal, run the following for each client pod:

kubectl exec -it POD_NAME -- iperf3 -c MASTER_POD_IP -f M

The result should be something similar to the following:

Connecting to host 172.30.70.199, port 5201
[  5] local 172.30.35.143 port 34202 connected to 172.30.70.199 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   617 MBytes   617 MBytes/sec   41   1.43 MBytes
[  5]   1.00-2.00   sec   714 MBytes   714 MBytes/sec    0   1.75 MBytes
[  5]   2.00-3.00   sec   635 MBytes   635 MBytes/sec   74   1.12 MBytes
[  5]   3.00-4.00   sec   669 MBytes   669 MBytes/sec   49   1.17 MBytes
[  5]   4.00-5.00   sec   649 MBytes   648 MBytes/sec  340    908 KBytes
[  5]   5.00-6.00   sec   641 MBytes   642 MBytes/sec    0   1.32 MBytes
[  5]   6.00-7.00   sec   649 MBytes   649 MBytes/sec    0   1.63 MBytes
[  5]   7.00-8.00   sec   670 MBytes   670 MBytes/sec   57   1.42 MBytes
[  5]   8.00-9.00   sec   661 MBytes   661 MBytes/sec   11   1.26 MBytes
[  5]   9.00-10.00  sec   634 MBytes   634 MBytes/sec  362    969 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.38 GBytes   654 MBytes/sec  934             sender
[  5]   0.00-10.04  sec  6.38 GBytes   650 MBytes/sec                  receiver
iperf Done.

Conclusion

This is just one way to debug network performances in a Kubernetes cluster.

If the output of iPerf tests return low bitrates, there is something misconfigured on the cluster, and you should immediately ask to IBM Cloud support to have a look.

Engage the support via the IBM Cloud Support Center.

Was this article helpful?
YesNo

More from Cloud

Enhance your data security posture with a no-code approach to application-level encryption

4 min read - Data is the lifeblood of every organization. As your organization’s data footprint expands across the clouds and between your own business lines to drive value, it is essential to secure data at all stages of the cloud adoption and throughout the data lifecycle. While there are different mechanisms available to encrypt data throughout its lifecycle (in transit, at rest and in use), application-level encryption (ALE) provides an additional layer of protection by encrypting data at its source. ALE can enhance…

Attention new clients: exciting financial incentives for VMware Cloud Foundation on IBM Cloud

4 min read - New client specials: Get up to 50% off when you commit to a 1- or 3-year term contract on new VCF-as-a-Service offerings, plus an additional value of up to USD 200K in credits through 30 June 2025 when you migrate your VMware workloads to IBM Cloud®.1 Low starting prices: On-demand VCF-as-a-Service deployments begin under USD 200 per month.2 The IBM Cloud benefit: See the potential for a 201%3 return on investment (ROI) over 3 years with reduced downtime, cost and…

The history of the central processing unit (CPU)

10 min read - The central processing unit (CPU) is the computer’s brain. It handles the assignment and processing of tasks, in addition to functions that make a computer run. There’s no way to overstate the importance of the CPU to computing. Virtually all computer systems contain, at the least, some type of basic CPU. Regardless of whether they’re used in personal computers (PCs), laptops, tablets, smartphones or even in supercomputers whose output is so strong it must be measured in floating-point operations per…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters