Monitoring Kubernetes

Installing

If you need the Legacy Kubernetes sensor for compatibility reasons, then install the agent Helm chart with the --set k8s_sensor.deployment.enabled=false setting.

Checking the status and version of the legacy sensor

To check the status and running version of the Legacy Kubernetes sensor, complete the following steps:

1. From the navigation menu, select Infrastructure.
2. On the Maps tab, click your Kubernetes node.
3. In the node details panel, expand Instana Agent (1) and then click on your agent.
4. On the agent panel, click Open Dashboard.
5. On the agent page, click Sensors Info in the Info area.
6. In the Sensors Info window, search Instana - Kubernetes - Sensor in the search field.

If the sensor is properly enabled, you see a match in the table. The State column displays the value Active, and the Version column shows the version number, for example, 1.2.143.

Troubleshooting

If the search for Instana - Kubernetes - Sensor and filtering the agent logs for com.instana.sensor-kubernetes yield no results, then check the configuration map of the Instana agent by running the following command:

kubectl -n instana-agent get cm instana-agent -o yaml

Make sure that the following setting is not present:

    com.instana.plugin.kubernetes:
      enabled: false

If the State column for the Instana - Kubernetes - Sensor displays Waiting for several minutes, it means that the sensor failed to activate.

Search the agent logs, filter for com.instana.sensor-kubernetes, and look for Activating Kubernetes Sensor. Specifically, check for the following message:

ERROR : bundle com.instana.sensor-kubernetes:1.2.143 (246)[com.instana.agent.kubernetes.sensor.Kubernetes(479)] : The activate method has thrown an exception                                
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.43.0.1/apis/apps/v1/deployments?labelSelector=app%3Dk8sensor. Message: Forbidden!Configured serv
ice account doesn't have access. Service account may have been revoked. deployments.apps is forbidden: User "system:serviceaccount:instana-agent:instana-agent" cannot list resource "deployme
nts" in API group "apps" at the cluster scope.

This error message indicates that a service account is not usable for accessing the Kubernetes API server.

Installing

When you install the agent, the Next Generation K8sensor is installed automatically by default. If you specify the k8s_sensor.deployment.enabled value, make sure that it is set to true (the default value).

Checking the status and version of K8sensor

To determine if the K8sensor is running on a cluster, run the following command to list deployments with the label app: k8sensor.

kubectl get deployments --all-namespaces -l app=k8sensor

If K8sensor is in the list, then K8sensor is enabled. When K8sensor is enabled, the Legacy Kubernetes sensor is automatically disabled. In the configuration.yaml file, you can see the enabled value is changed to false as follows:

    com.instana.plugin.kubernetes:
      enabled: false

The K8sensor version is best identified by the sha256 hash of its image. To check the image hash of the running K8sensor container, run the following command:

kubectl get po -n instana-agent -l app=k8sensor  -o jsonpath='{ .items[0].status.containerStatuses[0].imageID }'

Troubleshooting

If listing the deployments with the label app: k8sensor returns no results, the cluster is not running K8sensor. This issue means that K8sensor was disabled when the agent was deployed. To resolve this issue, redeploy the agent with K8sensor enabled.

Kubernetes dashboards

On the Kubernetes page, click a cluster or a namespace. You can see Kubernetes dashboards that present all the information for a certain Kubernetes entity. The context is always accessible by the context path in the upper left. In the following screenshot, you can see a namespace that is named "robot-shop" in a cluster called "k8s-demo-cluster".

Kubernetes dashboards are structured as follows:

• Summary shows the most relevant information for a certain entity. This dashboard starts with a status line that shows the status and related information, such as age. In the next section, you can see the CPU, memory, and pod information, which provides the consumed resources, including the pods. Sections like Top Deployments and Top Pods in the following screenshot show potential hotspots, which you might want to have a look at. The Logs section shows you the distribution chart of relevant logs for that entity, which complements the entity metrics. The chart is interactive and allows selection and highlight over all the measured values. You can focus on the selected time period or jump to the Analyze section to continue a troubleshooting journey.

• Details shows detailed information like "labels", "annotation", and the "spec".

• Events shows all relevant Kubernetes events and links them to the respective dashboards.

• Related Entities like "Deployments", "K8s Services", and "Pods" are shown as tabs of Kubernetes dashboards. What is shown depends on the entity that you selected.

Using the agent.serviceMesh.enabled flag

You can enable the Instana agent JVM monitoring with Istio service mesh by using the agent.serviceMesh.enabled flag. This Kubernetes-native approach uses a single dedicated network port for all Java workloads that are monitored on a single host or node. The default value is set to true. For more information about the configuration parameter, see Helm Chart configuration.

If the Istio configuration is set to REGISTRY_ONLY, additional steps are required for the agent socket service to work properly.

You need to deploy the following resource definition for each individual cluster node. Make sure to define a unique metadata.name property for each host or node. Also, set the value for spec.hosts to <node-ip-address>.instana-agent-headless.instana-agent.svc, where <node-ip-address> is the node's IP address.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: instana-agent-worker-<node-unique-counter>
spec:
  hosts:
  - <node-ip-address>.instana-agent-headless.instana-agent.svc
  ports:
  - number: 42699
    name: agent
    protocol: TCP
  resolution: DNS
  location: MESH_EXTERNAL

Using service mesh bypass

The alternative legacy approach is to enable service mesh bypass. The default installation of Istio works out-of-the-box with Instana. If you deploy Istio with a default deny policy (mode: REGISTRY_ONLY), you can enable Instana's service mesh bypass by using the following agent configuration:

com.instana.container:
  serviceMesh:
    enableServiceMeshBypass: true

The setting bypass blocked network connectivity in two different ways:

• Allow outgoing traffic from the application pod to the host agent (on all IPv4 addresses which the agent listens on, and all ports).
• Allow incoming traffic to the application pod from the agent for JVM applications (from all ipv4 addresses which the host agent listens on, all ports).

Verify enabled

To verify whether the service mesh by-pass is enabled, check in the Instana agent logs by running the following command:

kubectl logs -l app.kubernetes.io/instance=instana-agent -n instana-agent -c instana-agent

If the service mesh by-pass is enabled, you can find the following log lines, which indicate that an inbound or outbound by-pass entry is written for the denoted process:

Inbound by-pass:

2021-04-26T08:13:57.065+0000 | INFO  | -client-thread-2 | DefaultServiceMeshSupport        | 51 - com.instana.agent - 1.1.597 | Applying inbound service mesh bypass for process '764670'

Outbound by-pass:

2021-04-26T08:13:57.140+0000 | INFO  | -client-thread-2 | DefaultServiceMeshSupport        | 51 - com.instana.agent - 1.1.597 | Applying outbound service mesh bypass for process '764670'

Verify iptable rules

The easiest way to verify that the iptable rules is to shell into the Instana agent and list the target container iptables rules as follows. Replace ${PID} with the pid of the JVM process:

kubectl -n instana-agent exec -it ${INSTANA_AGENT_POD} -c instana-agent  -- /bin/bash
nsenter -n -t ${PID} iptables -t nat -n -L INSTANA_OUTPUT

If the chains are applied, you can see an output as follows:

Chain INSTANA_OUTPUT (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            10.128.15.237
ACCEPT     tcp  --  0.0.0.0/0            10.64.0.1
ACCEPT     tcp  --  0.0.0.0/0            169.254.123.1

Check whether bidirectional communication between the Instana agent and your JVM processes is supported by running the following command:

nsenter -n -t ${PID} iptables -t nat -n -L INSTANA_INBOUND

The result is similar to the following output:

Chain INSTANA_INBOUND (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  10.128.15.237        10.64.0.14
ACCEPT     tcp  --  10.64.0.1            10.64.0.14
ACCEPT     tcp  --  169.254.123.1        10.64.0.14

Depending on when the iptable rules were applied, it can take a few minutes for the process to be instrumented and the data to be visible in Instana's dashboards.