Monitoring Kubernetes

Supported managed kubernetes

• IBM Cloud Kubernetes Service Monitoring and Performance Management
• Amazon Elastic Container Service for Kubernetes (EKS)
• Azure Kubernetes Service (AKS)
• Google Kubernetes Engine (GKE)
• IBM Cloud Kubernetes Service
• VMware Tanzu Kubernetes Grid (TKG) and VMware Tanzu Kubernetes Grid Integration (TKGI), formerly known as Pivotal Container Service (PKS)

Note: Only Linux workers are supported. Workers that run on Windows are not supported.

Supported service meshes

• Instana supports the last three stable versions of Istio.

Kubernetes menu item

First and foremost Kubernetes is a top level item in the menu. This gives you direct access to your Kubernetes clusters and namespaces.

From application perspectives

Kubernetes information is also accessible from within all your application perspectives or services. If a service is running on a Kubernetes cluster, the respective context information is shown in the "Infrastructure" tab:

For containers the pod and namespace, and for hosts the cluster and node are shown and directly linked.

From infrastructure

In the Infrastructure map you will see Kubernetes information in the sidebar for either the host or the container you have selected.

You can use Dynamic Focus to filter the data. For example search for a specific deployment in a cluster. Additionally, the keywords entity.kubernetes.cluster.distribution and entity.kubernetes.cluster.managedBy enable searching for a Kubernetes cluster by distribution and management layer. Supported values for entity.kubernetes.cluster.distribution are gke, eks, openshift and kubernetes. Supported values for entity.kubernetes.cluster.managedBy are rancher and none.

CPU and memory usage

For Kubernetes pods, deployments, services, namespaces and nodes it is possible to see an aggregated view of the current CPU and Memory usage as it compares to the CPU and Memory limits and requests set for these resources.

If available, the usage information is calculated from data gathered from the container runtime that is executing the containers that make up the given resource.

Single kubernetes service to multiple logical services

Multiple logical services can be related to a single Kubernetes service when the service mapping rules match up, and there are calls generated on that Kubernetes service. For example, a Kubernetes service with the label selector "service=my-service" may contain pods that have the additional labels "env=dev" and "env=staging" — combined with a custom service mapping configuration in Instana with the following tags kubernetes.container.name and kubernetes.pod.label, key: env, results in multiple logical services linked to that single Kubernetes service and is displayed on the Kubernetes Service dashboard.

Single logical service to multiple kubernetes services

Multiple Kubernetes services can be related to a single logical service when those Kubernetes services are destroyed and recreated over time. For example, if the Kubernetes service shop-service-a with generated calls is replaced over time with shop-service-b with generated calls, both services are displayed on the logical service dashboard when the selected period of time overlapped when the calls were generated.

Cluster

CronJob

DaemonSet

Deployment

Job

K8s service

Namespace

Node

Pod

StatefulSet

Built-in

There are a couple of built-in health rules that will trigger an issue for Kubernetes entities

• Cluster
  • Kubernetes reports a Master-Component (api-server, scheduler, controller manager) is unhealthy. Note that due to a bug in Kubernetes the health is not always reported reliably. We try to filter these out, not causing an alert but only showing up on the Cluster detail page.
• Node
  • Requested CPU is approaching max capacity (requested CPU / CPU capacity ratio is greater than 80%).
  • Requested Memory is approaching max capacity (requested memory / memory capacity ratio is greater than 80%).
  • Allocated pods are approaching maximum capacity (allocated pods / pods capacity ratio is greater than 80%). For a node pods in the phases 'Running' and 'Unknown' are counted as allocated. See Kubernetes docs for details on node capacity.
  • Node reports a condition which is not ready for more than one minute. For a node that's all conditions besides the Ready condition. See Kubernetes docs for details on all node conditions.
• Namespace
  • Requested CPU is approaching max capacity (requested CPU / CPU capacity ratio is greater than 80%).
  • Requested Memory is approaching max capacity (requested memory / memory capacity ratio is greater than 80%)
  • Allocated pods are approaching maximum capacity (allocated pods / pods capacity ratio is greater than 80%). For a namespace pods in the phases 'Pending', 'Running', and 'Unknown' are counted as allocated. The namespace capacity values are based on ResourceQuotas which can be set per Namespace. See Kubernetes docs for details.
• Deployment
  • Available replicas less than desired replicas.
• Pod
  • A pod is not ready for more than one minute, and the reason is not that it's completed. (PodCondition=Ready, Status=False, Reason != PodCompleted). See Kubernetes docs for details on all pod conditions.

Custom

In addition to the built-in rules, you can also create custom rules on metrics of a cluster, namespace, deployment, and pod. E.g. if the threshold for node capacity warnings is too high you can disable them and create a custom rule with a lower threshold. See Events & Incidents configuration for details.

OpenShift ServiceMesh

Please see the OpenShift FAQs on the OpenShift ServiceMesh.

Istio

The default installation should work out of the box with Instana. If however you deploy Istio with a default deny policy (mode: REGISTRY_ONLY). To work effectively with this configuration it is necessary to enable Instana's service mesh by-pass. This can be enabled with the following agent configuration:

com.instana.container:
  serviceMesh:
    enableServiceMeshBypass: true

The setting will bypass blocked network connectivity in 2 different ways:

• Allow outgoing traffic from the application pod to the agent (on all ipv4 addresses the agent listens on, all ports).
• Allow incoming traffic to the application pod from the agent for JVM applications (from all ipv4 addresses the agent listens on, all ports).

Debugging the mesh by-pass

There are a couple of steps that can be taken to debug the service mesh by-pass.

1. verify it is enabled.
2. verify the iptable rules are applied to the container.

kubectl logs -l app.kubernetes.io/instance=instana-agent -n instana-agent -c instana-agent

2021-04-26T08:13:57.065+0000 | INFO  | -client-thread-2 | DefaultServiceMeshSupport        | 51 - com.instana.agent - 1.1.597 | Applying inbound service mesh bypass for process '764670'

2021-04-26T08:13:57.140+0000 | INFO  | -client-thread-2 | DefaultServiceMeshSupport        | 51 - com.instana.agent - 1.1.597 | Applying outbound service mesh bypass for process '764670'

kubectl -n instana-agent exec -it ${INSTANA_AGENT_POD} -c instana-agent  -- /bin/bash
nsenter -n -t ${PID} iptables -t nat -n -L INSTANA_OUTPUT

Chain INSTANA_OUTPUT (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            10.128.15.237
ACCEPT     tcp  --  0.0.0.0/0            10.64.0.1
ACCEPT     tcp  --  0.0.0.0/0            169.254.123.1

nsenter -n -t ${PID} iptables -t nat -n -L INSTANA_INBOUND

Chain INSTANA_INBOUND (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  10.128.15.237        10.64.0.14
ACCEPT     tcp  --  10.64.0.1            10.64.0.14
ACCEPT     tcp  --  169.254.123.1        10.64.0.14

Why am I not seeing any Kubernetes clusters or namespaces?

If there are no clusters or namespaces listed on the Kubernetes page, either no cluster is actively being monitored due to an agent not being installed, or no cluster was monitored during your selected timeframe.

Click Live to check for any clusters and namespaces in live mode, and if none are listed, see our Install Kubernetes section.

Missing clusterRole permissions

Monitoring issue type: kubernetes_missing_permissions

The Instana Agent requires the appropriate ClusterRole permissions for specific resources to be able to monitor a Kubernetes cluster successfully. If these permissions are missing, there will be corresponding resources missing on the Instana Kubernetes dashboards. To resolve this issue, install the latest version of the Instana Agent YAML, Helm chart or Operator. See our Kubernetes or OpenShift documentation for more information on the latest version of each installation method.

Deprecations

• We have deprecated support for the extensions/v1beta1 and apps/v1beta2 API versions for DaemonSet, Deployment, and ReplicaSet in our Kubernetes Sensor, following the announcement from Kubernetes that these deprecated API versions will soon be removed in Kubernetes v1.16. Please follow the latest installation documentation for Kubernetes or OpenShift and update to our latest Helm chart, agent YAML or operator.