IDOT and the OpenTelemetry project
The Instana Distribution of OpenTelemetry Collector (IDOT) is built on top of the upstream OpenTelemetry Collector open source project. While the core functions come from the upstream project, IDOT provides several Instana-specific enhancements:
- Pre-configured components optimized for sending telemetry data to Instana backends
- Streamlined installation and configuration process
In addition, IDOT maintains a carefully curated subset of components from the community version, focusing on stability and security for enterprise environments.
- IBM maintains security fixes and patches for IDOT
- Enterprise-grade support through IBM's support channels
- Regular security audits and vulnerability management
- Compatibility testing with Instana's backend services
IDOT maintains compatibility with the OpenTelemetry Protocol (OTLP) standard and provides a streamlined experience for Instana users. The distribution is regularly updated to incorporate improvements from the upstream project and helps ensure reliable integration with Instana's observability platform.
Before you begin, review the system requirements to help ensure a smooth installation experience.
Instana Distribution of OpenTelemetry Collector (IDOT)
For Instana to receive the telemetry data that is sent by the vendor-agnostic OpenTelemetry Collector, you must configure the OTLP or gRPC and OTLP or HTTP endpoints correctly. The IDOT automates the collector configuration process and ensures that Instana receives the telemetry data by default.
The IDOT supports the following operating systems, platforms and architectures:
| Operating system or Platforms | Architectures | Valid on air-gapped environment |
|---|---|---|
| Linux | x86-64 (amd64), s390x (IBM Z) | Yes |
| AIX | ppc64 | Yes |
| Windows | amd64 | Yes |
| Kubernetes | amd64 | Yes |
| OpenShift | amd64 | Yes |
Installing the IDOT
To install the IDOT in your target environment, complete the following steps:
-
Download the appropriate installer for your architecture:
For x86-64 or amd64 systems:
curl -Lo instana_otelcol_setup.sh https://github.com/instana/instana-otel-collector/releases/latest/download/instana-otel-collector-installer-latest-linux-amd64.sh chmod +x instana_otelcol_setup.shFor s390x or zLinux systems:
curl -Lo instana_otelcol_setup.sh https://github.com/instana/instana-otel-collector/releases/latest/download/instana-otel-collector-installer-latest-linux-s390x.sh chmod +x instana_otelcol_setup.shFor AIX systems:
curl -Lo instana_otelcol_setup.sh https://github.com/instana/instana-otel-collector/releases/latest/download/instana-otel-collector-installer-latest-aix-ppc64.sh chmod +x instana_otelcol_setup.shNote:The Instana UI provides a simplified installation script (instana-collector-installer-latest.sh) that automatically detects your system architecture. Both the UI script and the architecture-specific scripts install the same collector.For Windows (one-liner download and installation)
powershell -Command "Invoke-WebRequest -Uri 'https://github.com/instana/instana-otel-collector/releases/download/<latest-version>/instana-otel-collector-installer-latest-windows-amd64.zip' -OutFile '$env:TEMP\instana-collector.zip'; Expand-Archive -Path '$env:TEMP\instana-collector.zip' -DestinationPath 'C:\Program Files\Instana\' -Force; Set-Location 'C:\Program Files\Instana\instana-collector\bin'; .\setenv.bat -a <INSTANA_KEY> -e <INSTANA_OTEL_ENDPOINT_GRPC> -H <INSTANA_OTEL_ENDPOINT_HTTP>; .\start.bat"For Windows (MSI installation)
Use the MSI installer as the recommended installation method for Windows environments. The MSI installer supports interactive GUI installation, automatically registers a Windows service, and integrates with enterprise deployment tools such as Group Policy and SCCM.
For more information, see Installing IDOT on Windows by using the MSI installer.
For Kubernetes Helm chart
helm install instana-otel-collector \ --repo https://instana.github.io/instana-otel-collector instana-otel-collector-chart \ --namespace instana-otel-collector \ --create-namespace \ --set clusterName=<CLUSTER_NAME> \ --set instanaEndpoint=<INSTANA_OTEL_ENDPOINT> \ --set instanaKey=<INSTANA_KEY>For more information about Kubernetes installation guidelines, including the operator and OpenShift OS support, see Installing the Instana Distribution of OpenTelemetry Collector on Kubernetes.
-
Run the installation script by using the following command:
sudo ./instana_otelcol_setup.sh -e <INSTANA_OTEL_ENDPOINT_GRPC> -a <INSTANA_KEY> [-H <INSTANA_OTEL_ENDPOINT_HTTP>] [-u USE_SUPERVISOR_SERVICE] [<install_path>]Use the following parameters with the installation script:<INSTANA_OTEL_ENDPOINT_GRPC>: Specifies the required gRPC endpoint for Instana. Use the<ip_address>:<port>format.<INSTANA_KEY>: Specifies the required Instana agent key for authentication.<INSTANA_OTEL_ENDPOINT_HTTP>: Specifies the optional HTTP endpoint for Instana. Use the<ip_address>:<port>format.<USE_SUPERVISOR_SERVICE>: Specifies whether the Instana Supervisor Service is enabled. The default value istrue. Set this parameter tofalseto disable the supervisor service.
Important:TheINSTANA_OTEL_ENDPOINT_GRPCandINSTANA_KEYparameters are required to run the installer. You can change these parameters later in theconfig.envfile that is located in the install_path/collector/config/config.env. The default path is /opt/instana/collector/config/config.env. In addition, root access is required for installing, as the collector typically needs to write to system directories and configure system-level services.
- Supervisor Service: Manages the lifecycle of the collector process, enabling remote management and automatic restarts. The supervisor service is enabled by default during installation.
- Collector Service: The main OpenTelemetry Collector process that receives, processes, and exports telemetry data.
For more information about OpenTelemetry terminology such as the Supervisor Service and Collector Service, see the OpenTelemetry Collector Management documentation.
Managing the Instana Collector Service
The IDOT Collector installation script installs and starts the service by default. You can further manage the Instana Collector Service by running the following commands within the bin directory:
-
Install the service:
./instana_collector_service.sh install -
Uninstall the service:
./instana_collector_service.sh uninstall -
Start the service:
./instana_collector_service.sh startOr
service instana-collector start -
Stop the service:
./instana_collector_service.sh stopOr
service instana-collector stop -
Restart the service:
./instana_collector_service.sh restartOr
service instana-collector restart -
Display the status of the service:
./instana_collector_service.sh statusOr
service instana-collector status
Managing the Instana Supervisor Service
As with the Instana Collector Service, the Instana Supervisor Service is installed and initialized during the IDOT installation process. Managing the Supervisor service is done the same way (seen in the preceding section) as managing the service for the collector by using ./instana_supervisor_service.sh instead of ./instana_collector_service.sh.
Configuring the OpenTelemetry Collector
You can configure the Collector according to your business requirements by modifying the /opt/instana/collector/config/config.yaml file. You can also define and modify the pipelines for telemetry data as required. See the following configuration example to support the data flow for all types of telemetry data:
receivers:
# Configure OTLP/gRPC endpoint
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:24317
# Specifies a file log receiver to include logs from a given path
filelog:
include: ["path/to/logs/*.log"]
processors:
# Set a limit for batch size
batch:
send_batch_size: 5000
send_batch_max_size: 10000
timeout: 180s
# Specify a transform processor to add a processed attribute for logs
transform:
log_statements:
- set(log.body, log.attributes["processed"])
exporters:
# Configure OTLP exporter for telemetry data to be sent to
otlp:
endpoint: INSTANA_OTEL_ENDPOINT_GRPC
headers:
x-instana-key: INSTANA_KEY
x-instana-host: INSTANA_HOST
tls:
insecure: true
# Assemble the data pipeline from the configured components
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
logs:
receivers: [filelog]
processors: [transform]
exporters: [otlp]
Optional: Configuring OTEL shell receiver
The shell receiver is an OpenTelemetry Collector component that executes shell commands and collects their output as metrics or logs. This receiver is useful for integrating command-line tools and scripts into your observability pipeline.
Security warning
Configuration options
The shell receiver supports the following configuration options:
receivers:
shell:
collection_interval: 60s # How often to execute commands (default: 60s)
commands:
- name: disk_usage # Required: Unique identifier for this command
command: df -h / # Required: Shell command to execute
shell: /bin/sh # Optional: Shell to use (default: /bin/sh)
args: [] # Optional: Additional arguments to pass to the shell
timeout: 5s # Optional: Maximum time the command can run
metric_name: disk.usage # Required for metrics: Name for the metric
log_name: disk_check # Optional: Name for log records (defaults to metric_name)
description: "Check disk usage" # Optional: Human-readable description
labels: # Optional: Additional key-value pairs as attributes
filesystem: "root"
output_type: both # Optional: How to process output (metric, log, or both; default: metric)
Command configuration parameters
Each command in the commands list supports the following options:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name |
string |
Yes | Unique identifier for this command | |
command |
string |
Yes | Shell command to execute | |
shell |
string |
No | /bin/sh |
Shell to use for execution |
args |
[]string |
No | [] | Additional arguments to pass to the shell |
timeout |
duration |
No | Maximum time the command can run | |
metric_name |
string |
Required whenoutput_type ismetric orboth |
Name for the metric. For simple outputs, it is the complete metric name. For tabular outputs, it acts as a metric prefix combined with column names (for example,prefix.column_name). |
|
log_name |
string |
Required whenoutput_type islog andmetric_name is not set |
metric_name |
Name for log records |
description |
string |
No | Human-readable description of what the command does | |
labels |
map[string]string |
No | {} |
Additional key-value pairs to add as attributes |
output_type |
string | No | metric |
Determines how to process the command output:metric,log, orboth |
row_regex |
string |
No | Regular expression to match and parse each output row (enables customized parsing) | |
metric_names |
[]string |
No (required ifrow_regex is set) |
Names for each captured field fromrow_regex |
|
descriptions |
[]string |
Required ifrow_regex is set |
Human-readable descriptions for each field (optional, must matchmetric_names length) |
|
field_types |
[]string |
No | How each field must be processed:Attribute orGauge |
Output processing modes
The receiver supports the following parsing modes:
Automatic parsing (default)
When row_regex is not specified, the receiver automatically detects the output format:
For Metrics (when output_type is metric or both):
- If the output can be parsed as a number, it creates a gauge metric with that value.
- If the output is tabular (multiple columns), it creates separate gauge metrics for each numeric column.
- If the output cannot be parsed as a number, it creates a sum metric with value 1 and includes the output as an attribute.
For Logs (when output_type is log or both):
- Creates a log record with the command output as the body.
- Sets
INFOseverity by default. - Includes command name, description, and labels as attributes.
- If the output can be parsed as a number, adds it as a
valueattribute.
Regex-based parsing (customized)
row_regex is specified, the receiver uses regular expressions to parse the output with full control over field types:
row_regex: Regular expression with capture groups for each fieldmetric_names: Names for each captured fieldfield_types: Specifies whether each field is anAttribute(string identifier) orGauge(numeric metric)descriptions: Optional descriptions for each field
Example configurations
The following examples show the different configuration options:
Basic usage
receivers:
shell:
collection_interval: 30s
commands:
- name: cpu_temperature
command: cat /sys/class/thermal/thermal_zone0/temp
metric_name: system.cpu.temperature
description: "CPU temperature in millidegrees Celsius"
Multiple commands with different output types
receivers:
shell:
collection_interval: 60s
commands:
- name: memory_free
command: free -m | grep Mem | awk '{print $4}'
metric_name: system.memory.free
description: "Free memory in MB"
labels:
unit: "MB"
output_type: metric
- name: disk_usage
command: df -h / | grep / | awk '{print $5}' | sed 's/%//'
metric_name: system.disk.usage
description: "Root filesystem usage percentage"
labels:
filesystem: "root"
unit: "percent"
output_type: both
- name: last_logins
command: last -n 5
log_name: system.logins
description: "Last 5 system logins"
output_type: log
Automatic parsing with multi-tabular output
receivers:
shell:
collection_interval: 60s
commands:
- name: system_metrics_auto
command: hyptop -b -n 1 | awk 'BEGIN {printf "%-12s %-10s %-8s %-12s\n","system","cpucount","cpu%","memoryinGB"}$2 ~ /^[0-9]+$/ {printf "%-12s %-10s %-8s %-12s\n",$1,$2,$3,$5}'
metric_name: system.metrics.shell.hypervisor
description: "System CPU and Memory metrics"
output_type: metric
Regex-based parsing for custom control
receivers:
shell:
collection_interval: 60s
commands:
- name: system_metrics_regex
command: hyptop -b -n 1 | awk 'BEGIN {printf "%-12s %-10s %-8s %-12s\n","system","cpucount","cpu%","memoryinGB"}$2 ~ /^[0-9]+$/ {printf "%-12s %-10s %-8s %-12s\n",$1,$2,$3,$5}'
metric_name: system.metrics.shell.hypervisor
description: "System CPU and Memory metrics"
output_type: metric
row_regex: '^\s*(\S+)\s+(\d+)\s+([\d.]+)\s+([\d.]+)\s*$'
metric_names:
- system
- cpucount
- cpu
- memoryinGB
descriptions:
- System Name
- CPU Count
- CPU Usage Percentage
- Memory Usage in GB
field_types:
- Attribute # system is an identifier, not a metric
- Gauge # cpucount is a numeric metric
- Gauge # cpu is a numeric metric
- Gauge # memoryinGB is a numeric metric
labels:
source: "hyptop"
- name: process_stats
command: ps aux | grep myapp | awk '{print $2, $3, $4}'
metric_name: process.metrics
description: "Process CPU and Memory usage"
output_type: metric
row_regex: '^\s*(\d+)\s+([\d.]+)\s+([\d.]+)\s*$'
metric_names:
- pid
- cpu_percent
- mem_percent
descriptions:
- Process ID
- CPU Usage Percentage
- Memory Usage Percentage
field_types:
- Attribute # PID is an identifier, not a gauge
- Gauge # CPU percentage is a metric
- Gauge # Memory percentage is a metric
labels:
app: "myapp"
Selecting a parsing mode
Use automatic parsing in the following instances:
- You have simple numeric outputs or standard tabular data.
- You want quick setup without regex configuration.
- All numeric fields should be treated as metrics.
Use regex-based parsing in the following instances:
- You need to distinguish between identifiers (such as, PID or system names) and actual metrics
- You have irregular or variable output formats.
- You want explicit control over field types and names.
- You need custom field descriptions.
Optional: Configuring the OTEL s390 receiver
The s390 receiver is an OpenTelemetry Collector component that collects metrics from Logical Partitions (LPAR) that run on IBM Z (s390x) systems under PR/SM. This receiver is supported only on zLinux LPAR and provides visibility into CPU utilization, capacity groups, and LPAR performance metrics.
Applicability
- IBM Z LinuxOne system that run Linux on the s390x architecture.
- zLinux LPAR environments that run under PR/SM.
- LPARs with the
s390_hypfsfile system mounted.
Prerequisites
- IBM Z LinuxOne systems that run Linux on the s390x architecture.
- The
s390_hypfsfile system mounted (typically at /sys/kernel/debug/s390_hypfs). - Global performance data control is enabled on the LPAR so that the receiver can access performance data for all LPARs.
- Read access to the
s390_hypfsfile system.
- Open the z-HMC console.
- Select the partition.
- In the Controls section, select the Access Global Performance Data checkbox.
Configuring the s390 receiver
To enable the s390 receiver, open the IDOT configuration file at /opt/instana/collector/config/config.yaml and complete the following changes:
Step 1: Enable the s390 receiver
receivers:
# s390 receiver: Collects Linux LPAR Resource utilization metrics sitting on PR/SM for IBM Z.
# IMPORTANT: Only enable if Linux LPAR running on PR/SM with s390_hypfs available
#
# Requirements:
# - Running on IBM Z Linux LPAR (not z/VM guest, KVM, or containers)
# - s390_hypfs filesystem mounted at /sys/kernel/debug/s390_hypfs
# - "Global performance data control" enabled on the LPAR
#
# Uncomment the following lines to enable s390 receiver:
s390:
collection_interval: 60s
hypfs_dir: /sys/kernel/debug/s390_hypfs
Step 2: Enable the s390 receiver in the metrics pipeline
service.pipelines.metrics/otlp section, uncomment the line that includes the s390 receiver and comment out the default receiver configuration:
service:
pipelines:
# Metrics data pipeline for OTLP data
# To enable s390 receiver for Linux LPAR sitting on PR/SM metrics, uncomment s390 in the receivers list below
metrics/otlp:
# receivers: [otlp/receiver] # Comment out this line when enabling s390
receivers: [s390, otlp/receiver] # Uncomment this line to enable s390 receiver
processors: [batch] # Process metrics with batching
exporters: [otlphttp/exporter] # Send metrics to Instana backend
Step 3: Restart the collector service
service instana-collector restart
./instana_collector_service.sh restart
Configuration parameters
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
collection_interval |
duration | No | 60s |
Specifies how often the receiver collects. Values under 30s and unset or zero values are automatically treated as 30s. |
hypfs_dir |
string | No | /sys/kernel/debug/s390_hypfs | The path to the s390_hypfs file system. |
Collected metrics
The s390 receiver collects the following metrics:
system.cpu.utilization.core:- Unit: %
- Description: Average LPAR core utilization percentage across the LPAR reported cores.
system.cpu.utilization.management:- Unit: %
- Description: Average management and accounting time percentage across the LPAR reported cores.
system.lpar.cores:- Unit: cores
- Description: Number of CPU or core blocks reported for the LPAR.
system.group.utilization:- Unit: %
- Description: Total utilization of the entire capacity group.
- Attributes:
capacity.group: Name of the capacity group.group.capacity: Capacity limit of the group, in cores.
- Available only when capacity groups exist.
Metric attributes
host.arch: s390xos.type: Linuxhost.id: Hostname of the LPAR
The receiver adds the following attributes to the metric data point level:
lpar.name: Name of the logical partition.capacity.group: Name of the capacity group, if applicable.
capacity.group: Name of the capacity group.group.capacity: Capacity limit of the capacity group, in cores.
Example output
LPAR: PROD1
- system.cpu.utilization.core: 45.23% (capacity.group="GROUP1", group.capacity=10.5, lpar.name="PROD1")
- system.cpu.utilization.management: 2.15% (capacity.group="GROUP1", group.capacity=10.5, lpar.name="PROD1")
- system.lpar.cores: 8 (capacity.group="GROUP1", group.capacity=10.5, lpar.name="PROD1")
LPAR: TEST1
- system.cpu.utilization.core: 12.45% (capacity.group="GROUP1", group.capacity=10.5, lpar.name="TEST1")
- system.cpu.utilization.management: 0.85% (capacity.group="GROUP1", group.capacity=10.5, lpar.name="TEST1")
- system.lpar.cores: 4 (capacity.group="GROUP1", group.capacity=10.5, lpar.name="TEST1")
Capacity Group: GROUP1
- system.group.utilization: 67.97% (capacity.group="GROUP1", group.capacity=10.5)
system.group.utilization metric represents the total utilization of all LPARs in the capacity group.Troubleshooting
Permission denied
Error: failed to open /sys/kernel/debug/s390_hypfs/diag_204: permission denied
Make sure that the collector process has read permissions for the s390_hypfs file system. Depending on your environment, you might need to run the collector as root or grant the required capabilities.
No metrics generated
- The collection interval is configured as expected. Values under
30sseconds are automatically treated as 30 seconds. - The
s390_hypfsfile system is accessible at the configured path. - The system is running on the s390x architecture.
- Global performance data control is enabled on the LPAR.
Connecting IDOT to different Instana environments
Self-hosted environments
For IDOT, you must configure the OTLP exporter to send data to the Instana backend which is a default option.
-
otlp-grpc.<base_domain>→ OTLP/gRPC -
otlp-http.<base_domain>→ OTLP/HTTP
By default, the Instana backend uses a self-signed certificate. If you do not configure the collector with the CA file, you must disable verification. Otherwise, the TLS handshake fails.
Example configuration without certificate verification:
exporters:
otlp:
endpoint: https://otlp-grpc.<base_domain>:443
tls:
insecure: false
insecure_skip_verify: true
headers:
x-instana-key: <your_instana_key>
x-instana-host: <your_instana_host>
Check the supervisor configuration. Make sure that it targets the right endpoint and handles the self-signed certificate correctly.
Connecting IDOT to the local agent
Alternatively, you can report data through the local Instana agent on the same host. This option requires specific configuration during installation and in the configuration files.
Installation command for local agent
127.0.0.1 (localhost) for the endpoint parameters:
sudo ./instana_otelcol_setup.sh -e 127.0.0.1:4317 -a <INSTANA_KEY> -H http://127.0.0.1:4318 -o <INSTANA_OPAMP_ENDPOINT> -u true
127.0.0.1 instead of the server IP address. The Instana agent listens on ports 4317 and 4318 only on localhost.-u true), you must provide -o by setting INSTANA_OPAMP_ENDPOINT in the command line.Post-installation configuration
/opt/instana/collector/config/config.env file to use HTTP protocol for localhost connections:
INSTANA_OTEL_ENDPOINT_GRPC=http://127.0.0.1:4317
INSTANA_OTEL_ENDPOINT_HTTP=http://127.0.0.1:4318
http:// instead of https:// for localhost connections. The local Instana agent does not use TLS encryption for localhost connections. Using HTTPS results in SSL handshake errors.INSTANA_OPAMP_ENDPOINT=wss://<opamp-endpoint-url>:443/v1/opamp
INSTANA_OPAMP_ENDPOINT is required when the supervisor service is enabled to allow remote management of the collector.YAML configuration for local agent
config.yaml to target localhost:
exporters:
otlphttp:
endpoint: http://127.0.0.1:4318
tls:
insecure: true
Installing the Instana Distribution of OpenTelemetry Collector on Kubernetes
For Kubernetes environments, IDOT can be deployed as a centralized collector using either Helm charts or YAML manifests. This deployment model allows you to perform the following tasks:
- Collect telemetry data from multiple applications across your cluster
- Configure high availability with multiple collector instances
- Manage the collector configuration through Kubernetes native tools
For detailed instructions on installing and configuring IDOT on Kubernetes, see Installing the Instana Distribution of OpenTelemetry Collector on Kubernetes.
For operator-based deployment and management of IDOT on Kubernetes, see Installing IDOT through operator.
Uninstalling the OpenTelemetry Collector
By default, the installation script also adds an uninstallation script under collector/bin in the install_path.
./uninstall.sh
Running this script stops the Instana Collector Service and removes all collector files from the system.
Troubleshooting
For common issues and solutions, see Troubleshooting the Instana Distribution of OpenTelemetry Collector. This topic covers installation problems, connectivity issues, and platform-specific troubleshooting steps.
Release Notes
For the latest updates, features, and bug fixes, see the IDOT release notes.
Support
For any support related issues, see Support Strategy for IDOT.
Additional OpenTelemetry resources
- Real-world configuration examples.
- Troubleshooting guides.
- Integration patterns for various technologies.
- Community-contributed solutions.