Monitoring OS process

Instant automatically deploys and installs the OS Process sensor after you install the Instana agent.

Configuration

Note:

Monitoring OS processes involves two distinct Instana sensors that work together to provide full OS process observability:

OS Process sensor (com.instana.plugin.process): Collects process monitoring metrics, such as CPU, memory, open files, and context switches.
eBPF sensor (com.instana.plugin.ebpf): Detects abnormal process termination events and out of memory (OOM) killer events.

Process abnormal termination

The Instana agent can automatically detect abnormal termination of processes (for example, crashes) of monitored processes and the issuing and outcome of out of memory killer events issues to monitored processes on the host.

Requirements

Instant supports detection of abnormal process termination, and OOM killer is only supported on Linux on the AMD64 architecture. A 4.8 or later Linux kernel is required or, in the case of RHEL, a Linux kernel 3.10.0-957 or later.
debugfs must be mounted, which is the case for all Linux OSes supported by the Instana host agent, except Amazon Linux 1.

Anomalies detected

Abnormal termination

Exit with erroneous status codes, for example, exit 1
Kill of the process by using kill, also known as SIGKILL
Segmentation faults
Unhandled signals

Abnormal process termination displayed on the process's dashboard.

Out of memory killer

The out of memory killer event is an event sent by the operating system or a container runtime to a process, called target process that consumes (as in: "allocates") more memory than what it is allowed to. The target process can then decide to terminate, or indicate which of its children processes to terminate instead.

A common use case is selecting a child process to terminate in leader‑worker architectures, such as NGINX or PHP‑FPM, where the leader delegates work to workers that usually consume more resources than the leader.

In Instana, OOM killer events are documented based on two dedicated events:

The Out of memory event records which process was terminated, helping you understand what happened when the target process does not terminate itself.
The Killed by out of memory killer event documents which process was terminated as the result of an Out of memory event. The Killed by out of memory killer event includes an Abnormal Termination event due to an uncaught SIGKILL signal that documents how the termination of the process occurred.

Depending on whether the target decided to terminate itself or one of its children, there are two possible scenarios in Instana:

If the target process decides to terminate itself, you see on the target process three events: Out of memory event, the Killed by out of memory killer event, and of course the Abnormal Termination event.
If the target process selects one of its children to terminate, you see the Out of memory event on the target process, and Killed by out of memory killer and the Abnormal Termination events on the children process that was selected.

At first glace, one might think that the Abnormal Termination and Killed by out of memory killer are redundant. However, they explain two different aspects of the termination of a process: the Abnormal Termination event explains the how and the Killed by out of memory killer event explains the why. Also, Instana finds that it provides a better user-experience to be able to find all the Abnormal Termination events looking the same, irrespective of whether they occur due to an out of memory killer event or otherwise.

Deactivation

The eBPF sensor detects abnormal process termination and OOM killer events. To disable this functionality, set enabled: false for com.instana.plugin.ebpf in the configuration.yaml file:

com.instana.plugin.ebpf:
  enabled: false

Note:

Disabling the eBPF sensor affects only the detection of abnormal termination and OOM killer events. Process metrics, custom process monitoring, and environment variable collection, which are provided by the OS Process sensor (com.instana.plugin.process), remain unaffected.

Custom processes

Instana automatically monitors process metrics for higher‑level sensors, such as Java or MySQL. To monitor an operating system–level process that Instana does not detect by default, you can configure it as shown in the following example:

com.instana.plugin.process:
  poll_rate: 3 # values are in seconds. Default value is 3 seconds.
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - 'config'
    - 'port'
  services:
    - 'upower.service'
    - 'polkit.service'

Process

To monitor a specific process in Instana, you must add the process names in the processes: section in the configuration file as a list. Instana automatically monitors any process whose name matches the strings that you provide. For example, if you list sshd and slapd in processes:, Instana monitors all processes that contain sshd or slapd in their names.

Arguments

To monitor specific processes based on certain arguments in Instana, you must add those keywords in the arguments: section in the configuration file as a list. Instana automatically monitors any process whose argument key or value matches the strings that you provide. For example, if you add config and port in arguments:, Instana monitors all processes that contain config or port in their arguments.

Service

To monitor specific services on Linux in Instana, you must add the service names in the services: section in the configuration file as a list. Instana automatically monitors any service whose name matches the strings that you specify. For example, if you list upower.service and polkit.service in services:, Instana monitors all services that match those names.

Voluntary and non-voluntary context switches

Note:

This functionality is supported only on Linux-based operating systems.

You can manually enable monitoring of context switches by editing the host's agent configuration file (/opt/instana/agent/etc/instana/configuration.yaml):

...
com.instana.plugin.process:
  ctx_switches_enabled: true

OS process environment variables

Instana's process sensor automatically captures all the environment variables of any monitored process. Because environments often contains sensitive or secret data, the process sensor will take any configured secrets into account when filtering.

You can find more about configuring secrets at Agent Configuration secrets

You can also manually disable the monitoring of process environment variables by editing the host agent configuration file (/opt/instana/agent/etc/instana/configuration.yaml) as follows:

...
com.instana.plugin.process:
  env_vars_enabled: false

configuring secrets

Metrics collection

To view the metrics, select Infrastructure in the navigation panel of the Instana UI, click a specific monitored host, and then you can see a host dashboard with all the collected metrics and monitored processes.

Configuration data

PID
Executable
Started At
User
Group
Max Open Files
Arguments

Performance metrics

CPU usage

CPU usage values as a percentage; user and system. You can view the values on a graph over a selected time period.

Data point: Filesystem

Granularity: 3 seconds

Normalized CPU usage

Normalized CPU usage values show the amount of CPU usage (in percentage) for executing user-mode and system-mode code of a process. The graph displays the values over a selected time period.

Data point: Filesystem

Granularity: 3 seconds

Memory

Memory usage values as a byte; virtual, resident and share. The values are displayed on a graph over a selected time period.

Data point: Filesystem

Granularity: 3 seconds

Open files

Open files values used as a total number and current as a percentage. The values are displayed on a graph over a selected time period.

Open files current versus max are visible when they are available on the operating system.

Data point: Filesystem

Granularity: 3 seconds

Number of context switches

Number of times the process was context-switched; voluntary and nonvoluntary. The values are displayed on a graph over a selected time period.

Data point: Filesystem

Granularity: 3 seconds

Health signatures

For each sensor, there is a curated knowledgebase of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents depending on user impact.

Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any given entity.

For information about the built-in event for the OS process sensor, see the Built-in events reference.