Instana Selective monitoring (public preview)

You can customize how the Instana agent discovers and monitors applications according to your organizational or technical requirements by using Selective monitoring. Instead of the agent automatically discovering all applications or processes, Instana Selective monitoring enables you to opt-out or opt-in for monitoring on a per-process basis. In addition to the per-process configuration, Selective monitoring also supports namespace-level control using Kubernetes labels to opt-out or opt-in all workloads that are running in a Kubernetes namespace.

Selective monitoring is an Instana agent feature and is available on agent version 1.2.28 or later. Namespace labels are supported for agent operator 2.1.25 or later and Helm chart 2.0.21 or later.

Currently, Selective monitoring is in public preview.

Opt-out process monitoring

The default mode of Instana Agent is to discover and monitor all processes on the host it is installed on. If certain processes need to be ignored during discovery and also ignored for monitoring, the respective process can opt-out of monitoring by setting the INSTANA_MONITORING environment variable to false. Instana Agent ignores every process with INSTANA_MONITORING=false in the process environment.

Opt-In process monitoring

To change the default discovery behavior of Instana Agent, it needs to be started with the environment variable INSTANA_SELECTIVE_MONITORING set to OPT_IN. In this mode, the Agent discovers only processes, which deliberately opt-in to monitoring. Each process that needs to be monitored in OPT_IN mode, requires the environment variable INSTANA_MONITORING set to true. Instana Agent monitors only processes with INSTANA_MONITORING=true in the process environment.

Opt-in and Opt-out matrix

Monitoring mode	Description	Agent environment	Process environment	Result
Opt-out	All processes monitored by default	n/a	n/a	The process is monitored
Opt-out	All processes monitored by default	n/a	`INSTANA_MONITORING=false`	The process is ignored
Opt-in	No process monitored by default	`INSTANA_SELECTIVE_MONITORING=OPT_IN`	n/a	The process is ignored
Opt-in	No process monitored by default	`INSTANA_SELECTIVE_MONITORING=OPT_IN`	`INSTANA_MONITORING=true`	The process is monitored

Applying Selective monitoring to Kubernetes

For Kubernetes-based workloads, configure the opt-in and opt-out modes either at the container level or on the namespace level. The configuration for a Kubernetes deployment to opt out "example-workload" from monitoring is shown in the following example. The environment variable that is described in Opt-put process monitoring is set as an environment variable on the workload's container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-workload
spec:
  ...
  template:
    spec:
      containers:
      - name: example-workload
        image: example-workload-image
        env:
        - name: INSTANA_MONITORING
          value: false

For the opt-in mode, you must set the appropriate environment variable for the agent. If you deploy your agents by using the operator, you can set the agent CustomResource as shown in the following example:

apiVersion: instana.io/v1
kind: InstanaAgent
metadata:
  name: instana-agent
  namespace: instana-agent
spec:
  cluster:
      name: example-cluster
  agent:
    key: <agent-key>
    endpointHost: ingress-red-saas.instana.io
    endpointPort: "443"
    env:
      INSTANA_SELECTIVE_MONITORING: "OPT_IN"

For a Helm chart-based deployment, apply the environment variable either in values.yaml or when you run Helm by specifying the Helm command option:

--set agent.env.INSTANA_SELECTIVE_MONITORING='OPT_IN'

Applying Selective monitoring on namespace-level

To opt-out or opt-in all workloads that are deployed on a Kubernetes namespace, you can apply the label instana-workload-monitoring=false or instana-workload-monitoring=true to the namespace labels. The Instana agent operator syncs the namespace labels with all host agents and ignore the workloads, or start monitoring the workloads accordingly.

The following example creates a namespace "demo" and sets the label and removes it on the last step. This example assumes the agent is deployed in opt-in mode as described in the section Applying Selective Monitoring to Kubernetes.

# create a new namespace
$ kubectl create ns demo
namespace/demo created
# monitor workloads in the demo namespace
$ kubectl label ns demo instana-workload-monitoring=true
namespace/demo labeled
# remove label
$ kubectl label ns demo instana-workload-monitoring-
namespace/demo unlabeled

Limitations to Selective monitoring on Kubernetes

Live update of monitored workloads

If a workload is already monitored by Instana, it remains under monitoring regardless of whether the namespace labels are updated during runtime to ignore all processes in a namespace.

Kubernetes resources

Kubernetes resources from the control plane are always collected for all the resource types. The Instana Kubernetes sensor can pull this data from the control plane, such as Deployments, DaemonSets, Namespaces, and Pods. Metrics at this level are not application-specific.

The agent collects metrics for all the containers that run on the host, regardless of the process that is ignored in the container. Only the container-level metrics are collected. For example, the agent collects metrics for several system CRIO containers that run on an OpenShift Container Platform node, but the processes are not monitored. On the process level, the agent does not start sensors and tracers if the process is opted out of monitoring. The ignored process is not listed as a child entity for the host, but the corresponding container is listed as a child entity. To know whether a process is ignored, find the corresponding container and check whether the process entity is attached as a child entity.

AutoTrace webhook

In the current public preview state, the AutoTrace webhook does not respect the opt-out or opt-in configuration on a process-level. If the AutoTrace webhook is deployed in the cluster and related workloads are configured to be ignored for monitoring, also the respective label instana-autotrace: "false" is required. For more information about opt-out and opt-in for the AutoTrace webhook, see Ignoring resources.