Installing the host agent on Kubernetes

Supported Versions

• Instana supports the current stable version of Kubernetes. If the current stable version of Kubernetes is 1.26, the Kubernetes sensor is pinned to version 1.24. According to the Kubernetes version compatibility guarantee, Instana supports Kubernetes versions 1.22 to 1.26. However, the lowest 2 versions in that range are considered as a soft deprecation.

JVM monitoring prerequisites

The Instana host agent creates temporary JAR files in the $TMP/.instana directory. These JAR files are required for the JVM monitoring. For successful JVM monitoring, ensure that the root user that runs the agent and the user that runs the JVM have the necessary read and write permissions for the $TMP directory.

If the $TMP directory on the host has enforced read-only permissions (readOnlyRootFilesystem: true), you must add the following configuration to the Java workloads to ensure successful monitoring. By using the configuration, the agent can write files to the $TMP location.

    securityContext:
      ...
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: tmp-volume
      mountPath: /tmp
  volumes:
  - name: tmp-volume
    emptyDir: {}

Installing the host agent in an air-gapped environment

You can install Instana agent in an air-gapped environment, which has limited internet access, with Kubernetes package manager HELM.

In the following example, the CentOS Linux distribution is used, but installation steps are valid for all distributions.

Installing an agent using an operator in an air-gapped environment

1. Pull the following images.

   docker pull containers.instana.io/instana/release/agent/static:latest
   docker pull icr.io/instana/k8sensor:latest
   docker pull icr.io/instana/instana-agent-operator:latest
   

2. Re-tag pulled images to a local image registry.

   docker tag <public-image> <local-registry-image>
   

3. Push re-tagged images to the local image registry.

   docker push <local-registry-image>
   

4. Create an image pull secret (Docker secret) in your target Kubernetes namespace:

   kubectl create secret -n instana-agent docker-registry local-registry-image-pull-secret \
        --docker-server=<your_local_image_registry> \
        --docker-username= # Replace with ARTIFACTORY_USERNAME \
        --docker-password= # Replace with ARTIFACTORY_PASSWORD 
   

5. Fetch yaml from GitHub:

   curl -L -o instana-agent-operator.yaml https://github.com/instana/instana-agent-operator/releases/latest/download/instana-agent-operator.yaml
   

6. Replace image reference with the private registry. For example, icr.io/instana/instana-agent-operator:2.1.8 with a private registry operator image.

7. Apply the configuration to deploy the operator.

   kubectl apply -f instana-agent-operator.yaml
   

8. If you do not have OpenShift CLI (oc) installed, you can download it from Getting started with the OpenShift CLI and patch the service account to have access to the image pull secret that you created in the previous section:

   oc patch serviceaccount instana-agent-operator -p '{"imagePullSecrets": [{"name": "local-registry-image-pull-secret"}]} -n instana-agent'
   

9. Create a file agent.yaml and update the agent custom resource to specific image versions:

   apiVersion: instana.io/v1
   kind: InstanaAgent
   metadata:
     name: instana-agent
     namespace: instana-agent
   spec:
     zone:
       name: # (optional) name of the zone of the host
     cluster:
       name: # cluster name.
     agent:
       image:
         name: # private registry static agent image.
         tag: # agent tag
         pullSecrets:
           - name: local-registry-image-pull-secret
       key: # fetch the download key from the Instana UI > Agents > Install Agent > Kubernetes Operator
       endpointHost: <your-ingress-endpoint>
       endpointPort: "443"
       env: {}
       configuration_yaml: |
         # You can leave this empty, or use this to configure your instana agent.
         # See https://docs.instana.io/setup_and_manage/host_agent/on/kubernetes/
     k8s_sensor:
       image:
         name: # private registry k8 sensor image.
         tag: # k8 sensor tag
   

10. Deploy the custom resource:

$ kubectl apply -f agent.yaml

11. Watch for all pods starting correctly.

Installing the host agent in an online environment

You can use several available methods to install the Instana agent onto a Kubernetes cluster. If you install the agent on a host manually, only containers and processes are monitored, and Kubernetes data is not collected. You are recommended to install the agent by using the Operator. Alternative methods are the Helm Chart or YAML file (DaemonSet) deployment.

Install by using the Operator

Instana provides a Kubernetes operator to install and manage the Instana agent.

For configuration options that you can set by using the Instana Agent Custom Resource Definition and environment variables, see Operator configuration.

Install by using the Helm Chart

With a DaemonSet, the Helm chart adds the Instana agent to all schedulable nodes in your cluster.

1. Sign in to the Instana UI, and then select an option to display the agent catalog; for example, on the home page, click Deploy Agent.

   If you are starting a new trial instance of Instana, the agent catalog is displayed with a prompt to select a host agent to install.

2. Click the tile Kubernetes - Helm chart

3. Enter the cluster name and (optionally) the agent zone that you want the cluster to be part of.

   The cluster name (<your_cluster_name>) is the customised name of the cluster monitored by this daemonset.

   The agent zone (<your_zone_name>) is used to customize the zone grouping displayed on the infrastructure map.

   The agent deployment code is updated with the values that you provide. All of the other required parameters are pre-populated in the agent deployment code, which looks like the following:

   helm install instana-agent
   --repo https://agents.instana.io/helm \
   --namespace instana-agent \
   --create-namespace \
   --set agent.key='<your_agent_key>' \
   --set agent.endpointHost='<your_host_agent_endpoint>' \
   --set agent.endpointPort=443 \
   --set cluster.name='<your_cluster_name>' \
   --set zone.name='<your_zone_name>' \
   instana-agent
   

4. Copy and then run the agent deployment code with Helm 3.

   To configure the installation, you can specify the values on the command line by using the --set flag or can provide a YAML file with your values by using the -f flag.

   If you want to deploy the static host agent, set the flag —set agent.image.name=containers.instana.io/instana/release/agent/static in the agent deployment code.

   For a detailed list of all the configuration parameters, see Instana Helm Chart.

--set 'agent.tls.certificate=<YOUR_CERTIFICATE_BASE64_ENCODED>'
--set 'agent.tls.key=<YOUR_PRIVATE_KEY_BASE64_ENCODED>'

helm uninstall instana-agent

Install as a DaemonSet

To install and configure the Instana host agent as a DaemonSet within your Kubernetes cluster, customize the instana-agent.yaml file to create the instana-agent namespace in which the DaemonSet is created. This enables you to tag agents for quick identification or to stop all of them by deleting the namespace.

1. Sign in to the Instana UI, and then select an option to display the agent catalog; for example, on the home page, click Deploy Agent.

   If you are starting a new trial instance of Instana, the agent catalog is displayed with a prompt to select a host agent to install.

2. Click the tile Kubernetes - YAML

3. Enter the cluster name and (optionally) the agent zone that you want the cluster to be part of.

   The cluster name (<your_cluster_name>) is the customised name of the cluster monitored by this daemonset.

   The agent zone (<your_zone_name>) is used to customize the zone grouping displayed on the infrastructure map.

   The agent deployment code is updated with the values that you provide. All of the other required parameters are pre-populated in the agent deployment code.

4. Download (or copy and save) the agent deployment code as a YAML file; for example: deployment.yaml.

5. To install Instana within your Kubernetes Cluster, run the following command:

   kubectl apply -f deployment.yaml
   

   Where deployment.yaml is the name of the file that you created in the previous step.

   Note: If you make any more edits to the deployment.yaml file, you must recreate the DaemonSet. To apply changes, run the following commands:

   kubectl delete -f deployment.yaml
   kubectl apply -f deployment.yaml
   

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account)

kubectl label --overwrite ns instana-agent pod-security.kubernetes.io/enforce=privileged

Checking the status of the host agent

After you install the host agent, you can check the status of the host agent in the Instana UI or on the host. For more information, see Checking the status of the host agent.

Configure network access for monitored applications

Some types of applications need to reach out to the agent first. Currently they are

• Node.js
• Go
• Ruby
• Python
• .NET Core

Those applications need to know on which IP the agent is listening. As the agent will listen on the host IP automatically, use the following Downward API snippet to pass it in an environment variable to the application pod:

spec:
  containers:
    env:
      - name: INSTANA_AGENT_HOST
        valueFrom:
          fieldRef:
            fieldPath: status.hostIP

Monitor master nodes

Per default, the agent does not get scheduled on Kubernetes master nodes, as the deployment respects the default taint node-role.kubernetes.io/master:NoSchedule that is set on most master nodes. To overwrite these add the following toleration to the agent daemonset:

kind: DaemonSet
metadata:
  name: instana-agent
  namespace: instana-agent
spec:
  template:
  ...
    spec:
      tolerations:
        - key: "node-role.kubernetes.io/master"
          effect: "NoSchedule"
          operator: "Exists"
    ...

For more direct control install the Agent separately on the master nodes. Please contact support on advice for your environment.

Monitor Kubernetes NGINX Ingress

For guidelines on how to configure the Kubernetes NGINX Ingress and our Agent for capturing NGINX metrics, see the Monitoring NGINX page. Tracing of the Kubernetes NGINX Ingress is also possible via the OpenTracing project, see Distributed Tracing for NGINX Ingress on guidelines how to set that up.

Secrets

Kubernetes has built-in support for storing and managing sensitive information. However, if you do not use that built-in capability but still need the ability to redact sensitive data in Kubernetes resources the agent secrets configuration is extended to support that.

To enable sensitive data redaction for selected Kubernetes resources (specifically annotations and container environment variables), set the INSTANA_KUBERNETES_REDACT_SECRETS environment variable to true as shown in the following agent yaml snippet:

spec:
  containers:
      env:
        - name: INSTANA_KUBERNETES_REDACT_SECRETS
          value: "true"

Then configure the agent with the desired list of secrets to match on as described in the agent secrets configuration.

It is important to note that enabling this capability can possibly cause a decrease in performance in the Kubernetes sensor.

Report to multiple backends

To add additional backends by using helm and operator, run the following command. You can use additional backends to configure your Instana agent or leave it empty.

When you run the following command, only the agent data is forwarded to multiple backends. The k8s sensor and metrics data are not forwarded to multiple backends.

• Using helm:

     helm install instana-agent \
     --repo https://agents.instana.io/helm \
     --namespace instana-agent \
     --create-namespace \
     --set agent.key=my-key \
     --set agent.endpointHost='<your_host_agent_endpoint>' \
     --set agent.endpointPort=443 \
     --set cluster.name='<your_cluster_name>' \
     --set zone.name='zone-name' \
     --set "agent.additionalBackends[0].endpointHost=<your_host_agent_endpoint>" \
     --set "agent.additionalBackends[0].key=<your_agent_key>" \
     --set "agent.additionalBackends[0].endpointPort=443" \
     --set "agent.additionalBackends[1].endpointHost=<your_other_host_agent_endpoint>" \
     --set "agent.additionalBackends[1].key=<your_other_agent_key>" \
     --set "agent.additionalBackends[1].endpointPort=443" \
     instana-agent
  

• Using operator:

    apiVersion: instana.io/v1
    kind: InstanaAgent
    metadata:
    name: instana-agent
    namespace: instana-agent
    spec:
    zone:
      name: zone-name
    cluster:
      name: cluster-name
    agent:
      key: <your_agent_key>
      endpointHost: <your_host_agent_endpoint>
      endpointPort: "443"
      env: {}
      additionalBackends:
      - endpointHost: <your_host_agent_endpoint>
      key: <your_agent_key>
      endpointPort: "443"
      - endpointHost: <your_other_host_agent_endpoint>
      key: <your_other_agent_key>
      endpointPort: "443"
      configuration_yaml: |
  

To enable reporting to multiple backends from a Kubernetes agent, see the Docker agent configuration.

Instana agent security considerations

Since the Instana agent needs to connect to application pods and list/open own ports on bridge networks on the node it is deployed to, it requires host network access and host level process ID lookup for infrastructure correlation. This is equivalent to the permissions granted on Linux & Unix host environments and requires the following flags as set in the DaemonSet deployment:

• privileged: true provides full access to /proc without overlay, change UID/GID for JVM attachment and access application namespaces.
• hostPID: true provides host level PIDs in /proc, required for infrastructure correlation.
• hostNetwork: true provides access to host level & bridge network interfaces.

In addition, when deploying to OpenShift, the Security Context Constraint (SCC) will be set to privileged to grant the afore mentioned permissions.

The following cluster role rules are required by the Kubernetes sensor to detect all resources and applications running in the Kubernetes cluster and making sure its monitored correctly:

rules:
  - nonResourceURLs:
    - "/version"
    - "/healthz"
    verbs: ["get"]- apiGroups: ["batch"]
    resources:
      - "jobs"
      - "cronjobs"
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources:
      - "deployments"
      - "replicasets"
      - "ingresses"
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - "deployments"
      - "replicasets"
      - "daemonsets"
      - "statefulsets"
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - "namespaces"
      - "events"
      - "services"
      - "endpoints"
      - "nodes"
      - "pods"
      - "replicationcontrollers"
      - "componentstatuses"
      - "resourcequotas"
      - "persistentvolumes"
      - "persistentvolumeclaims"
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources:
      - "endpoints"
    verbs: ["create", "update", "patch"]
  - apiGroups: ["networking.k8s.io"]
    resources:
      - "ingresses"
    verbs: ["get", "list", "watch"]

Example YAML file

A typical instana-agent.yaml file can be downloaded form Instana public GitHub repository. It gets rendered from the Helm chart with typical defaults. Individual properties are defined as dangling anchors as layed out in the next step.

Download this file and view the latest changelog.

Troubleshooting agent deployment

If installing the agent is not successful at first, you can check log messages and troubleshooting tips. If this troubleshooting section does not answer the questions you have, contact the IBM Instana support team with information about your experience, so that we can help you and update our documentation accordingly.

For troubleshooting information that is general to all host agents, see Managing host agents / Troubleshooting.

If reinstalling the agent fails with the following message, delete the agent's cluster level objects before reinstalling the agent:

installation Instana Agent failed: rendered manifests contain a resource that already exists. Unable to continue with install: ...

For more information about deleting the agent's cluster level objects, see Uninstalling the agent.