Compute Services

IBM Cloud Kubernetes Service Diagnostics and Debug Tool

Share this post:

Introducing the Diagnostics and Debug Tool for IBM Cloud Kubernetes Service

At the IBM Cloud Kubernetes Service, we thrive on customer feedback because you help us become better every day. While interacting with our customers, it struck us that the support response times and overall time to resolution are way higher than where we want them to be. We deeply understand and appreciate the frustration that comes when something is not working as expected and it is not clear what is wrong, especially when you don’t have visibility or understand each component of the underlying platform and infrastructure. Our goal is to get you faster resolution and turnaround, ideally without even opening a ticket.

With these goals in mind, we are excited to introduce the IBM Cloud Kubernetes Service Diagnostics and Debug Tool (Beta)—a tool that customers can run in order to collect information and identify potential issues with their IBM Cloud-managed Kubernetes cluster.

At this stage, the Diagnostics and Debug Tool is more of an experiment to determine if it is beneficial for you, the customer. Feedback is more than welcome—both positive and negative—because it will help us guide this project in the right direction.

Why is this tool even needed?

With the IBM Cloud Kubernetes Service, your security is paramount. Although we provide a managed Kubernetes service on IBM Cloud infrastructure, we do not access your clusters, your environment, or your applications. It is a fair request when the customers ask IBM to just make it work at all times, but at the same time, the same customers will also ask that IBM employees not have access to my environment.

The continuous improvement of our compliance posture and the journey to increase the number of certifications has taught us that when customers need help, the fastest way to provide it is to use tools that are self-serve and can run any time and that allow the customer to control and see the diagnostic data output that is shared via IBM Cloud support.

How does it work?

The Diagnostics and Debug Tool is installed to the IBM Cloud Kubernetes Service cluster via a helm chart. There is a DaemonSet component that is run on each worker node and a Deployment component that acts as the server side that you can talk to and ask tests to run. The client side is your browser on your laptop. You can reach the server side by running the kubectl proxy …command, which will create a private tunnel between your laptop and the server side running in your cluster. There is no external access available to the tool other than through kubectl proxy.

At any time, the tool can be deleted by simply deleting the helm chart. When deleting the helm chart, all remnants of the tool that were installed will be removed from the cluster in which it was installed.

What does this tool do?

The main purpose of the Diagnostics and Debug Tool is two-fold:

  • Collect basic information about the cluster status and details, which can be useful to you and allow you to share with IBM Cloud support if necessary. You can review all data that is gathered. The tool will compress this data in a zip file if you wish to send to IBM Cloud support or if you wish to use the data at a later time. This data can then be loaded back into the tool or opened directly in the operating system in which it was downloaded.
  • Run pre-defined tests that we created to catch and identify issues in an automated way.

Running tasks to gather data about network and Kubernetes and also run some tests.

Examples of collecting information

  • Gather information about the Kubernetes cluster’s resources: Pods, Deployments, Services, Ingress, Nodes, Events.
  • Collect information about the present network settings:
    • Print the routes on the worker nodes
    • Collect Calico pods, policies, global network policies, profiles, etc.
    • Display actual iptables rules deployed to the worker nodes
    • Show the VLAN, Ingress, and other configmaps
  • Display information about the ALB and related authentication container versions.
  • Collect VPN status, routes, rules, NAT information, logs, resources, etc. for strongSwan.

Examples of pre-defined tests

  • Ping all worker nodes and pods. This test pings all known IPs (as well as an external public IP) within a cluster from a Daemonset running on each node in the cluster.
  • Deploy a basic coffee and tea ingress resource (deploys the complete example in a unique namespace and validates ingress works correctly).
  • Validate ingress annotation syntax (that kubectl apply just accepts without checking).
  • Validate whether the VLAN configuration associated with the ALB is matching the environment.
  • Verify the existence of necessary secrets (accidental deletes happen surprisingly often).
  • Scan for errors in the ALB logs.
  • Test the strongSwan VPN connection and transport, check for the ipsec ports availability, etc. These tests will only produce test results if the strongSwan VPN Helm chart is installed.

How can I use it?

Installing the Diagnostics and Debug Tool is very easy. In its current form, it exists in a helm chart that you can install from the official IBM Cloud Helm repository. Please visit the following link to install the IBM Cloud Kubernetes Service Diagnostics and Debug Tool.

Installing without Tiller

Although you need to have helm installed on your laptop, you don’t necessarily have to install Tiller onto your cluster. Here is how.

Add the IBM repo, update (if you already had it), and download the chart into ibm-diagnostics-tool-chart folder:

$ helm repo add ibm https://registry.bluemix.net/helm/ibm
"ibm" has been added to your repositories
$ helm repo update
[...]
$ helm fetch ibm/ibmcloud-iks-debug --untar --untardir ./ibm-diagnostics-tool-chart

Make sure kubectl is set up correctly (kubectl get nodes should work). By running the following, the helm will generate the .yamls, which you then apply directly to your cluster. No Tiller installed:

$ helm template --namespace ibm-system --name my-debugger ./ibm-diagnostics-tool-chart/ibmcloud-iks-debug/ | kubectl apply -f -

(It will create a service account, role binding, daemonset, and the debug tool itself.)

Make sure the pods are running:

$ kubectl get pods -n ibm-system |grep my-debugger
my-debugger-ibmcloud-iks-debug-55d86f766c-v95b7 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-check-state 0/1 Completed 0 116s
my-debugger-ibmcloud-iks-debug-daemonset-czdzc 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-daemonset-klbsc 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-daemonset-qsmp5 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-validate-tests 0/1 Completed 0 116s

From here you can just follow the documented steps and run the kubectl proxy:

$ kubectl proxy --port 8080

Then, open the debug tool UI in your browser:

$ open \

http://localhost:8080/api/v1/namespaces/ibm-system/services/my-debugger-ibmcloud-iks-debug:8822/proxy/page

Summary

We hope this tool will accelerate and simplify the process of finding issues and get you to faster resolution of problems. Please give us feedback and tell us what tests you would like to see in the future.

Contact us

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.

Chief Architect, Networking – IBM Cloud Kubernetes Service

Cale Rath

Networking, Diagnostics Tool Dev Lead — IBM Cloud Kubernetes Service

More Compute Services stories
May 7, 2019

We’ve Moved! The IBM Cloud Blog Has a New URL

In an effort better integrate the IBM Cloud Blog with the IBM Cloud web experience, we have migrated the blog to a new URL: www.ibm.com/cloud/blog.

Continue reading

May 6, 2019

Use IBM Cloud Certificate Manager to Obtain Let’s Encrypt TLS Certificates for Your Public Domains

IBM Cloud Certificate Manager now lets you obtain TLS certificates signed by Let’s Encrypt. Let’s Encrypt is an automated, ACME-protocol-based CA that issues free certificates valid for 90 days.

Continue reading

May 6, 2019

Are You Ready for SAP S/4HANA Running on Cloud?

Our clients tell us SAP applications are central to their success and strategy for cloud, with a deadline to refresh the business processes and move to SAP S/4HANA by 2025. Now is the time to assess, plan and execute the journey to cloud and SAP S/4HANA

Continue reading