Compute Services

IBM Cloud Kubernetes Service Diagnostics and Debug Tool

Share this post:

Introducing the Diagnostics and Debug Tool for IBM Cloud Kubernetes Service

At the IBM Cloud Kubernetes Service, we thrive on customer feedback because you help us become better every day. While interacting with our customers, it struck us that the support response times and overall time to resolution are way higher than where we want them to be. We deeply understand and appreciate the frustration that comes when something is not working as expected and it is not clear what is wrong, especially when you don’t have visibility or understand each component of the underlying platform and infrastructure. Our goal is to get you faster resolution and turnaround, ideally without even opening a ticket.

With these goals in mind, we are excited to introduce the IBM Cloud Kubernetes Service Diagnostics and Debug Tool (Beta)—a tool that customers can run in order to collect information and identify potential issues with their IBM Cloud-managed Kubernetes cluster.

At this stage, the Diagnostics and Debug Tool is more of an experiment to determine if it is beneficial for you, the customer. Feedback is more than welcome—both positive and negative—because it will help us guide this project in the right direction.

Why is this tool even needed?

With the IBM Cloud Kubernetes Service, your security is paramount. Although we provide a managed Kubernetes service on IBM Cloud infrastructure, we do not access your clusters, your environment, or your applications. It is a fair request when the customers ask IBM to just make it work at all times, but at the same time, the same customers will also ask that IBM employees not have access to my environment.

The continuous improvement of our compliance posture and the journey to increase the number of certifications has taught us that when customers need help, the fastest way to provide it is to use tools that are self-serve and can run any time and that allow the customer to control and see the diagnostic data output that is shared via IBM Cloud support.

How does it work?

The Diagnostics and Debug Tool is installed to the IBM Cloud Kubernetes Service cluster via a helm chart. There is a DaemonSet component that is run on each worker node and a Deployment component that acts as the server side that you can talk to and ask tests to run. The client side is your browser on your laptop. You can reach the server side by running the kubectl proxy …command, which will create a private tunnel between your laptop and the server side running in your cluster. There is no external access available to the tool other than through kubectl proxy.

At any time, the tool can be deleted by simply deleting the helm chart. When deleting the helm chart, all remnants of the tool that were installed will be removed from the cluster in which it was installed.

What does this tool do?

The main purpose of the Diagnostics and Debug Tool is two-fold:

  • Collect basic information about the cluster status and details, which can be useful to you and allow you to share with IBM Cloud support if necessary. You can review all data that is gathered. The tool will compress this data in a zip file if you wish to send to IBM Cloud support or if you wish to use the data at a later time. This data can then be loaded back into the tool or opened directly in the operating system in which it was downloaded.
  • Run pre-defined tests that we created to catch and identify issues in an automated way.

Running tasks to gather data about network and Kubernetes and also run some tests.

Examples of collecting information

  • Gather information about the Kubernetes cluster’s resources: Pods, Deployments, Services, Ingress, Nodes, Events.
  • Collect information about the present network settings:
    • Print the routes on the worker nodes
    • Collect Calico pods, policies, global network policies, profiles, etc.
    • Display actual iptables rules deployed to the worker nodes
    • Show the VLAN, Ingress, and other configmaps
  • Display information about the ALB and related authentication container versions.
  • Collect VPN status, routes, rules, NAT information, logs, resources, etc. for strongSwan.

Examples of pre-defined tests

  • Ping all worker nodes and pods. This test pings all known IPs (as well as an external public IP) within a cluster from a Daemonset running on each node in the cluster.
  • Deploy a basic coffee and tea ingress resource (deploys the complete example in a unique namespace and validates ingress works correctly).
  • Validate ingress annotation syntax (that kubectl apply just accepts without checking).
  • Validate whether the VLAN configuration associated with the ALB is matching the environment.
  • Verify the existence of necessary secrets (accidental deletes happen surprisingly often).
  • Scan for errors in the ALB logs.
  • Test the strongSwan VPN connection and transport, check for the ipsec ports availability, etc. These tests will only produce test results if the strongSwan VPN Helm chart is installed.

How can I use it?

Installing the Diagnostics and Debug Tool is very easy. In its current form, it exists in a helm chart that you can install from the official IBM Cloud Helm repository. Please visit the following link to install the IBM Cloud Kubernetes Service Diagnostics and Debug Tool.

Installing without Tiller

Although you need to have helm installed on your laptop, you don’t necessarily have to install Tiller onto your cluster. Here is how.

Add the IBM repo, update (if you already had it), and download the chart into ibm-diagnostics-tool-chart folder:

$ helm repo add ibm https://registry.bluemix.net/helm/ibm
"ibm" has been added to your repositories
$ helm repo update
[...]
$ helm fetch ibm/ibmcloud-iks-debug --untar --untardir ./ibm-diagnostics-tool-chart

Make sure kubectl is set up correctly (kubectl get nodes should work). By running the following, the helm will generate the .yamls, which you then apply directly to your cluster. No Tiller installed:

$ helm template --namespace ibm-system --name my-debugger ./ibm-diagnostics-tool-chart/ibmcloud-iks-debug/ | kubectl apply -f -

(It will create a service account, role binding, daemonset, and the debug tool itself.)

Make sure the pods are running:

$ kubectl get pods -n ibm-system |grep my-debugger
my-debugger-ibmcloud-iks-debug-55d86f766c-v95b7 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-check-state 0/1 Completed 0 116s
my-debugger-ibmcloud-iks-debug-daemonset-czdzc 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-daemonset-klbsc 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-daemonset-qsmp5 1/1 Running 0 116s
my-debugger-ibmcloud-iks-debug-validate-tests 0/1 Completed 0 116s

From here you can just follow the documented steps and run the kubectl proxy:

$ kubectl proxy --port 8080

Then, open the debug tool UI in your browser:

$ open \

http://localhost:8080/api/v1/namespaces/ibm-system/services/my-debugger-ibmcloud-iks-debug:8822/proxy/page

Summary

We hope this tool will accelerate and simplify the process of finding issues and get you to faster resolution of problems. Please give us feedback and tell us what tests you would like to see in the future.

Contact us

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.

Chief Architect, Networking – IBM Cloud Kubernetes Service

Cale Rath

Networking, Diagnostics Tool Dev Lead — IBM Cloud Kubernetes Service

More Compute Services stories
March 18, 2019

Kubernetes Clusters: Architecture for Rapid, Controlled Cloud App Delivery

Kubernetes clusters are the building blocks of Kubernetes, and they provide the architectural foundation for the platform. The modularity of this building block structure enables availability, scalability, and ease of deployment.

Continue reading

February 27, 2019

Autoscale Applications in the IBM Cloud Foundry Enterprise Environment

We are excited to announce the availability of application auto-scaling capabilities in v2.1.0 of the IBM Cloud Foundry Enterprise Environment (CFEE) offering. This is built on the open-source App-Autoscaler project in the Cloud Foundry community.

Continue reading

February 21, 2019

IBM Cloud Kubernetes Service Available in Mexico City

IBM Cloud Kubernetes Service is a managed Kubernetes offering that delivers powerful management tools, an intuitive user experience, and built-in security and isolation. Today, we are excited to announce the availability of the IBM Cloud Kubernetes Service in Mexico City, Mexico.

Continue reading