Monitoring Containerd

Support information

To make sure that the Containerd sensor is compatible with your current setup, check the following support information sections:

Supported operating systems

Edit online

The supported operating systems of the Containerd sensor are consistent with host agents requirements, which can be checked in the Supported operating systems section of each host agent, such as Supported operating systems for Unix.

Supported versions and support policy

Edit online

The sensor supports the following versions of Containerd:

1.6.x
1.7.x
2.0.x

The following table shows the latest supported version and support policy:


Technology	Support policy	Latest technology version	Latest supported version
Containerd	45 days	2.2.1	2.2.1

For more information about the support policy, see Support strategy for sensors.

Supported environments

Edit online

If the environment runs Docker or Garden containers, then the Containerd sensor is automatically disabled to prevent multiple sensors from unnecessarily monitoring the same containers.

Supported containers

Edit online

All containers that run the Containerd container runtime are supported except for the pause containers.

The pause container is a container that holds the network namespace for the pod. Kubernetes creates pause containers to acquire the IP address of respective pod and set up the network namespace for all other containers that join that pod.

The pause containers are not supported for the following reasons:

Monitoring pause containers does not bring much information on the infrastructure level because they act as sidecar helper containers.
When the pause containers are included, it doubles the number of monitored containers in an environment and therefore increases the Instana monitoring costs.

Supported container orchestration platforms

Edit online

Most Kubernetes distributions, for example upstream Kubernetes with kubeadm, GKE, EKS, and AKS will use Containerd as the default container runtime. However, this is not the case for OpenShift, which uses the CRI-O container runtime. For Kubernetes clusters such as OpenShift, Instana supports container metric collection for CRI-O through the CRIO sensor. Container log collection for CRI-O containers is not directly supported by the CRIO sensor, hence you are recommended to use the OpenTelemetry Collector to collect logs for CRI-O containers in these Kubernetes distributions that use CRI-O as the container runtime.

Configuring

Edit online

The Containerd sensor queries container metadata, including Kubernetes labels that are displayed in the Kubernetes Labels tab in the Instana UI. It uses the CRI-API interface for supported Kubernetes environments.

A native CRI-API client can be enabled in Kubernetes by setting the environment variable INSTANA_USE_CONTAINERD_CRI_CLIENT (default: true). When this variable is set to true, container metadata is collected by using the Containerd socket, typically located at /var/run/containerd/containerd.sock, unless alternative sockets are specified in the sensor configuration.

Note: When the INSTANA_USE_CONTAINERD_CRI_CLIENT flag is set to false or the Kubernetes cluster does not support the CRI-API, the sensor falls back to use the crictl binary. The crictl binary is not packaged with the Instana agent. You must install on the host server where the Instana host agent is installed and made available to the agent container. After installation, add the path of the crictl binary to the $PATH environment variable for the user under which the host agent is running. If the crictl command-line tool is not installed, the Containerd sensor's performance metric collection for CPU and memory usage can work, but the Kubernetes labels that are associated with the container can't be displayed in the Kubernetes Labels tab in the Instana UI. Additionally, container log collection remains disabled. For more information about log collection, see Configuring the Containerd-Log sensor.

Configuring the Containerd sensor

Edit online

You can configure the Containerd sensor in the agent configuration file as shown in the following example:

  com.instana.plugin.containerd:
    enabled: true
    poll_rate: 10
    unixSockets:
      - '/path/to/containerd.sock'


Parameter	Description	Mandatory or Optional	Value range
`com.instana.plugin.containerd.enabled`	The flag to enable the Containerd sensor.	Optional	`false` or `true`. Default: `true`
`poll_rate`	Sets the polling rate for the plug-in in seconds.	Optional	1 to 3600. Default: 10
`com.instana.plugin.containerd.unixSockets`	Available since: Containerd sensor 1.0.30. List of fully qualified paths to `containerd.sock` files. The sensor has automatic socket file discovery that searches for the `containerd.sock` file in `/run/` and `/var/run/` directories. If a different location is configured for the socket, then the fully qualified path needs to be added.	Optional	List of fully qualified file paths.

Enabling metrics collection

Edit online

The Containerd sensor with ctr versions after 1.2.0 can automatically collect metrics by using the ctr task metrics command. In this case, you don't need to manually enable the metrics collection.

The Containerd sensor with ctr 1.2.0 and earlier versions does not support the metrics command. If you are using ctr 1.2.0 or earlier versions, the sensor will attempt to query metrics from the Prometheus endpoint. To enable the Prometheus endpoint, you need to specify the "metrics" address "127.0.0.1:1338" in the /etc/containerd/config.toml configuration file and then restart the containerd.service.

[metrics]
  address = "127.0.0.1:1338"

Viewing metrics

Edit online

To view the metrics, complete the following steps:

In the sidebar of the Instana UI, select Infrastructure.
Click a host that has a Containerd container.

You can see a host dashboard with all the collected metrics and monitored processes.

Viewing container details

Edit online

To fetch the container details, click Get Container Info.

The following information is displayed:

Container ID
Image
Created at
Updated at
Containerd namespace
Labels

Performance metrics

Edit online


Metric	Description	Granularity
CPU usage	The metrics include total CPU usage, kernel CPU usage, user CPU usage, and normalized CPU usage	10 seconds
CPU throttling	Throttling time and count	10 seconds
Memory usage	The metrics include total memory usage in bytes, resident set size in bytes, cache usage in bytes, total memory usage in percentage, and working set usage in percentage	10 seconds
Memory active	The amount of anonymous and cache active memory	10 seconds
Memory inactive	The amount of anonymous and cache inactive memory	10 seconds
Block IO	The amount of data, in bytes, that the monitored container writes to and reads from the block device	10 seconds

Derived performance metrics

Edit online

You can view the following derived metrics on the Instana UI:

CPU Total Normalized
Memory Usage %
Memory Working Set Usage %

Total Normalized CPU Usage

Edit online

The CPU Total Normalized metric is the normalized CPU usage of the container in percentage. The value can range from 0% to 100%. The metric is computed by using the following formula: total_normalized_cpu_usage_percentage = cpu_total_usage / cpu_limit

The cpu_limit value depends on the configuration of the container that is deployed. The value of the total_normalized_cpu_usage_percentage metric is not known if cpu_limit is not specified in the configuration of the container that is deployed.

Memory Usage

Edit online

The Memory Usage % metric is the memory usage in percentage, including resident set size and file system cache usage. The metric is computed by using the following formula: metric_usage_percentage = memory_used / memory_limit

The value of the memory_limit metric depends on the configuration of the container that is deployed. The value of the memory_used metric is not known if memory_limit is not specified in the configuration of the container that is deployed.

Memory Working Set Usage

Edit online

The Memory Working Set Usage % is the memory usage in percentage excluding the file system cache. The metric is computed by using the following formula: memory_working_set_usage_percentage = (memory_used - total_inactive_file) / memory_limit

The memory_limit value depends on the container's deployment configuration. The memory_working_set_usage_percentage is not provided when memory_limit is not specified in container's deployment configuration.

Viewing Containerd logs

Edit online

Note: To ingest Containerd logs, your account must include the logging add-on to your license. To verify your license, see License and entitlement requirements before proceeding.

Containerd log collection is supported for Containerd sensor version 1.0.30 and later.

Note: If the Containerd sensor CRI-API client is disabled, or the crictl executable is not available to the agent, then the Containerd sensor does not collect logs. For more information, see the Configuring section.

When this feature is enabled, the Containerd sensor collects only the container logs that are written to a file because Instana currently supports only the default logging driver, which writes log entries to a file.

For example, in Kubernetes deployments, Containerd logs might be written to files in the /var/log/pods/{namespace}_{podName}_{podUID}/{containerName} directory.

You can monitor the collected logs by clicking Analytics > Logs in the Instana UI and filter logs by using the "Container Snapshot Id" filter, which can filter both Docker and Containerd log messages.

Configuring the Containerd-Log sensor

Edit online

You can configure the Containerd-Log sensor in the agent configuration file as shown in the following example with the logs section.

  com.instana.plugin.containerd:
    enabled: true
    poll_rate: 10
    logs:
      enabled: true
      sendInterval: 60
      maxBufferSize: 16777216
    unixSockets:
      - '/path/to/containerd.sock'

The additional logging configuration parameters are described in the following table:


Parameter	Description	Mandatory or Optional	Value range
`com.instana.plugin.containerd.logs.enabled`	Available since: Containerd sensor 1.0.30. The flag to enable log collection for the Containerd sensor.	Optional	`true` or `false`. Default: `false`
`com.instana.plugin.containerd.logs.sendInterval`	Available since: Containerd sensor 1.0.30. The maximum duration (in seconds) that container logs remain cached in the log-sensor before they are uploaded to the backend.	Optional	1 to 600. Default: 60
`com.instana.plugin.containerd.logs.maxBufferSize`	Available since: Containerd sensor 1.0.30. The maximum internal log buffer size (in bytes). If internal log buffer contents exceed the size, then the log-sensor flushes the contents to the Instana backend.	Optional	1000 to 32000000. Default: 16777216 (16 MB)