Instana Selective monitoring

Edit online

You can customize how the Instana agent discovers and monitors applications according to your organizational or technical requirements by using Selective monitoring. Instead of the agent automatically discovering all applications or processes, Instana Selective monitoring enables you to opt-out or opt-in for monitoring on a per-process basis. In addition to the per-process configuration, Selective monitoring also supports namespace-level control using Kubernetes labels to opt-out or opt-in all workloads that are running in a Kubernetes namespace.

Selective monitoring is an Instana agent feature and is available on agent version 1.2.28 or later. Namespace labels are supported for agent operator 2.1.25 or later and Helm chart 2.0.21 or later.

Opt-out process monitoring

Edit online

The default mode of Instana Agent is to discover and monitor all processes on the host it is installed on. This is the default behavior when the INSTANA_SELECTIVE_MONITORING environment variable is not set or is explicitly set to OPT_OUT. If certain processes need to be ignored during discovery and also ignored for monitoring, the respective process can opt-out of monitoring by setting the INSTANA_MONITORING environment variable to false. Instana Agent ignores every process with INSTANA_MONITORING=false in the process environment.

Opt-In process monitoring

Edit online

To change the default discovery behavior of Instana Agent, it needs to be started with the environment variable INSTANA_SELECTIVE_MONITORING set to OPT_IN. In this mode, the Agent discovers only processes, which deliberately opt-in to monitoring. Each process that needs to be monitored in OPT_IN mode, requires the environment variable INSTANA_MONITORING set to true. Instana Agent monitors only processes with INSTANA_MONITORING=true in the process environment.

Opt-in and Opt-out matrix

Edit online


Monitoring mode	Description	Agent environment	Process environment	Result
Opt-out	All processes monitored by default	Not set or `INSTANA_SELECTIVE_MONITORING=OPT_OUT`	Not set	The process is monitored
Opt-out	All processes monitored by default	Not set or `INSTANA_SELECTIVE_MONITORING=OPT_OUT`	`INSTANA_MONITORING=false`	The process is ignored
Opt-in	No process monitored by default	`INSTANA_SELECTIVE_MONITORING=OPT_IN`	Not set	The process is ignored
Opt-in	No process monitored by default	`INSTANA_SELECTIVE_MONITORING=OPT_IN`	`INSTANA_MONITORING=true`	The process is monitored

Applying Selective monitoring to Kubernetes

Edit online

For Kubernetes-based workloads, configure the opt-in and opt-out modes either at the container level or on the namespace level. The configuration for a Kubernetes deployment to opt out "example-workload" from monitoring is shown in the following example. The environment variable that is described in Opt-put process monitoring is set as an environment variable on the workload's container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-workload
spec:
  ...
  template:
    spec:
      containers:
      - name: example-workload
        image: example-workload-image
        env:
        - name: INSTANA_MONITORING
          value: false

For the opt-in mode, you must set the appropriate environment variable for the agent. If you deploy your agents by using the operator, you can set the agent CustomResource as shown in the following example:

apiVersion: instana.io/v1
kind: InstanaAgent
metadata:
  name: instana-agent
  namespace: instana-agent
spec:
  cluster:
      name: example-cluster
  agent:
    key: <agent-key>
    endpointHost: ingress-red-saas.instana.io
    endpointPort: "443"
    env:
      INSTANA_SELECTIVE_MONITORING: "OPT_IN"

For a Helm chart-based deployment, apply the environment variable either in values.yaml or when you run Helm by specifying the Helm command option:

--set agent.env.INSTANA_SELECTIVE_MONITORING='OPT_IN'

Applying Selective monitoring on namespace-level

Edit online

To opt-out or opt-in all workloads that are deployed on a Kubernetes namespace, you can apply the label instana-workload-monitoring=false or instana-workload-monitoring=true to the namespace labels. The Instana agent operator syncs the namespace labels with all host agents and ignore the workloads, or start monitoring the workloads accordingly.

The following example creates a namespace "demo" and sets the label and removes it on the last step. This example assumes the agent is deployed in opt-in mode as described in the section Applying Selective Monitoring to Kubernetes.

# create a new namespace
$ kubectl create ns demo
namespace/demo created
# monitor workloads in the demo namespace
$ kubectl label ns demo instana-workload-monitoring=true
namespace/demo labeled
# remove label
$ kubectl label ns demo instana-workload-monitoring-
namespace/demo unlabeled

Limitations to Selective monitoring on Kubernetes

Edit online

z/OS platform

Selective monitoring is supported on z/OS for WebSphere or Liberty and CICS Liberty applications. To enable selective monitoring on z/OS, follow the same opt-in and opt-out patterns that are described in the Opt-in and Opt-out matrix section:

Set the agent environment variable to enable opt-in mode:
```
export INSTANA_SELECTIVE_MONITORING=OPT_IN
```
Set the process environment variable as a JVM argument for the application that you want to monitor:
```
-DINSTANA_MONITORING=true
```

The configuration follows the same matrix logic:

Opt-out mode: By default, the agent monitors all processes unless -DINSTANA_MONITORING=false is set as a JVM argument.
Opt-in mode: When you set INSTANA_SELECTIVE_MONITORING=OPT_IN is set for the agent, the agent monitors only processes with -DINSTANA_MONITORING=true as a JVM argument.

Selective monitoring is not supported for other z/OS applications that are not WebSphere or Liberty and CICS Liberty.

Live update of monitored workloads

If a workload is already monitored by Instana, it remains under monitoring regardless of whether the namespace labels are updated during runtime to ignore all processes in a namespace.

Kubernetes resources

Kubernetes resources from the control plane are always collected for all the resource types. The Instana Kubernetes sensor can pull this data from the control plane, such as Deployments, DaemonSets, Namespaces, and Pods. Metrics at this level are not application-specific.

The agent collects metrics for all the containers that run on the host, regardless of the process that is ignored in the container. Only the container-level metrics are collected. For example, the agent collects metrics for several system CRIO containers that run on an OpenShift Container Platform node, but the processes are not monitored. On the process level, the agent does not start sensors and tracers if the process is opted out of monitoring. The ignored process is not listed as a child entity for the host, but the corresponding container is listed as a child entity. To know whether a process is ignored, find the corresponding container and check whether the process entity is attached as a child entity.

AutoTrace webhook

The AutoTrace webhook does not respect the opt-out or opt-in configuration on a process-level. If the AutoTrace webhook is deployed in the cluster and related workloads are configured to be ignored for monitoring, also the respective label instana-autotrace: "false" is required. For more information about opt-out and opt-in for the AutoTrace webhook, see Ignoring resources.