OpenTelemetry related service mappings and infrastructure correlation

This topic provides guidance for customers who manually send OpenTelemetry data to Instana.

If you are using the Instana Distribution of OpenTelemetry Collector (IDOT), you do not need to follow the manual configuration because IDOT automatically handles service mappings and infrastructure correlation by default.

You can follow the information in this document to integrate OpenTelemetry with Instana and enable comprehensive service mappings and infrastructure correlation when you send telemetry data manually. This document explains how to use specific OpenTelemetry resource attributes for optimal data linking and visibility within Instana, particularly in Kubernetes environments and IBM middleware.

OpenTelemetry service mapping

Instana aligns with OpenTelemetry Semantic Conventions to map metrics, spans, and logs to services. These conventions establish standardized attribute names and usage patterns, promoting consistency and interoperability across observability tools. By aligning to these standards, Instana make sure the seamless integration of telemetry data into its monitoring and observability platform.

The following are the key attributes that are used in service mapping:

Resource attribute service.name
Trace span attribute peer.service

Resource attribute `service.name`

The service.name attribute uniquely identifies a service in OpenTelemetry. Each application, microservice, or component have a unique and logical service.name to enable effective filtering, analysis, and visualization of telemetry data. Instana maps service.name to its services, linking traces, metrics, and logs to the emitting application. Consistent and descriptive naming of service.name helps ensure accurate grouping and improved observability within Instana.

In OpenTelemetry, you can set the service.name attribute at different levels of your application, depending on the specific needs of your observability setup. In the following table, you can see the different ways in which service.name can be set:


Level	Definition Method	Example	Use Case
Application Code	Directly in the application during manual instrumentation	`service.name="orders-service"`	When the `service.name` is hardcoded or configured in the application itself
Environment Variable	By using the `OTEL_SERVICE_NAME` environment variable	`OTEL_SERVICE_NAME=payments-service`	Common for environments that use auto-instrumentation across multiple services
Resource Attributes	By using `OTEL_RESOURCE_ATTRIBUTES` with key-value pairs	`OTEL_RESOURCE_ATTRIBUTES=service.name=inventory-service`	Useful for combining `service.name` with other attributes like `host.id`
Collector Configuration	In OpenTelemetry Collector by using the `resource` processor	`service.name: shipping-service`	Centralized control for telemetry data before it is sent to the backend

Trace span attribute `peer.service`

In OpenTelemetry, the peer.service attribute specifies the name of the remote service a span connects to, helping to identify external dependencies like databases, message brokers, or other microservices. Instana maps peer.service to an Instana service, enabling the visualization of service dependencies and interactions.

Instrumentation Libraries: Some libraries automatically set peer.service when they are properly configured. For example:
- JDBC: The OpenTelemetry JDBC instrumentation can infer peer.service from database connection details.
- HTTP Clients: HTTP libraries such as Apache HttpClient can derive peer.service based on the remote service’s hostname.

In the following table, you can see the different ways in which peer.service can be set or configured in OpenTelemetry:


Level	Description	Example	Use Case
Application Code	Directly set the `peer.service` attribute in your code by using the OpenTelemetry API	`span.setAttribute("peer.service", "backend-service")`	When you have specific knowledge about the target service and want to provide a custom name
Configuration Files (System Property)	Define the `peer.service` value in system properties that are used by the OpenTelemetry agent or SDK	`otel.instrumentation.common.peer-service-mapping: "api.example.com=example-api-service,1.2.3.4=cats-service"`	When you want to map specific hostnames, IP addresses, or a combination of both to service names in a configuration file
Environment Variables	Set the `peer.service` value as an environment variable accessible to your application	`OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING="api.example.com=example-api-service,1.2.3.4=cats-service"`	When you need to dynamically set the `peer.service` value based on runtime conditions or deployment environment (similar to system properties but by using environment variables)
Instrumentation Library	Automatically set by OpenTelemetry libraries	`peer.service="postgresql"`	Common for popular libraries such as database and HTTP clients which set `peer.service` automatically

Service mapping of well-known protocols

In addition to general attributes such as service.name and peer.service, OpenTelemetry defines a comprehensive set of semantic conventions for well-known protocols, including HTTP, RPC, databases, messaging systems, and more.

For a complete list of protocols, refer to the Trace Semantic Conventions.

HTTP span attributes

OpenTelemetry provides semantic conventions for HTTP spans to maintain consistency across tracing systems, including attributes that offer insights into HTTP requests and responses.

In the following table, you can see the different ways in which http span attributes can be set or configured in OpenTelemetry:


Attribute	Description	Example
`http.method`	HTTP method (GET, POST)	`http.method="GET"`
`http.route`	Route or path of the HTTP request	`http.route="/webshop/articles/:id"`
`url.path`	Path component of the full URL	`url.path="/webshop/articles/4"`
`http.host`	Hostname of the server	`http.host="api.example.com"`
`http.target`	Target URL of the request	`http.target="/users"`

While http.host and http.target can often infer the service, dynamic routing or service discovery might require additional information from configuration files or service registries for accurate mapping.

Remote procedure calls span attributes

OpenTelemetry provides semantic conventions for Remote Procedure Call (RPC) spans to maintain consistency across tracing systems, including attributes that offer insights. The key RPC span attributes used for service mapping in OpenTelemetry are:

rpc.method: Specifies the method or function which is invoked, providing detailed information about the operation.
rpc.service: Identifies the name of the service that is called, which is crucial for mapping RPC calls to specific services.


Attribute	Description	Example
`rpc.service`	Name of the remote service	`rpc.service="user-service"`
`rpc.method`	A method or function that is invoked	`rpc.method="GetUserInfo"`

By combining rpc.service and rpc.method with other attributes such as peer.service, RPC calls can be mapped to their corresponding services.

Database span attributes

OpenTelemetry provides semantic conventions for Database spans to maintain consistency across tracing systems, offering key attributes that provide valuable insights into database operations and interactions. These insights are essential for understanding the performance and behavior of database systems within a distributed application.

In the following table, you can see the key database span attributes used for service mapping in OpenTelemetry.


Attribute	Description	Example
`db.system`	Database system (For example, MySQL and PostgreSQL)	`db.system="mysql"`
`db.instance`	Database instance name or identifier	`db.instance="my_database"`
`db.name`	The name of the database which is accessed	`db.name="users_db"`
`db.statement`	SQL query or database command executed	`db.statement="SELECT * FROM users"`
`db.operation`	Type of database operation (For example, SELECT and INSERT)	`db.operation="SELECT"`

While these attributes primarily describe database interactions, they can indirectly assist with service mapping. For example, the db.instance attribute might help identify the service interacting with a specific database instance.

Messaging span attributes

OpenTelemetry provides semantic conventions for Messaging spans to maintain consistency across tracing systems, including attributes that offer insights.

The key Messaging span attributes used for service mapping in OpenTelemetry are:


Attribute	Description	Example
`messaging.system`	Messaging system used (For example, Kafka, RabbitMQ, and AMQP)	`messaging.system="kafka"`
`messaging.destination`	Destination address (For example, topic name and queue name)	`messaging.destination="user-events"`
`messaging.operation`	Operation performed (For example, SEND, RECEIVE, PUBLISH, and SUBSCRIBE)	`messaging.operation="SEND"`

These attributes help map services based on the messaging systems that they interact with and the specific topics or queues they use. For example, if a service is known to send messages to a specific Kafka topic, the messaging.destination attribute can help identify the service.

Instana supports the resource attributes of OpenTelemetry and many commonly used definitions in OpenTelemetry Resource Semantic Conventions. Instana displays OpenTelemetry service flows and resources as infrastructure entities. You can link these entities to physical entities such as processes, hosts, or containers, in addition to trace spans and log records, especially when an Instana agent runs on the same host.

If you are using language-based OpenTelemetry auto-instrumentation, use the latest OpenTelemetry agent. The latest agent generates sufficient resource attributes to establish links between entities, spans, and log records.

In addition to service.name and service.instance.id, the following resource attributes are critical for linking:

In Kubernetes environments, the k8s.pod.uid resource attribute is critical for linking OpenTelemetry entities to Kubernetes pods. It establishes relationships between spans, logs, and metrics associated with these pods, providing a cohesive view of application and infrastructure performance.

Correlation strategies

Instana uses different strategies to correlate OpenTelemetry spans with infrastructure entities, depending on the telemetry data available:

Strategy 1: OpenTelemetry traces and metrics (recommended)

If your service emits both OpenTelemetry traces and metrics, the following process occurs:

Instana creates an OpenTelemetry entity from the metrics data.
OpenTelemetry spans automatically link to this OpenTelemetry entity by using the service.name and service.instance.id resource attributes. This process establishes the correlation chain: OpenTelemetry span > OpenTelemetry entity.

The OpenTelemetry entity uses different resource attributes to link to the corresponding existing infrastructure entities:

k8s.pod.uid - links to Kubernetes pods.
Correlation chain: OpenTelemetry span > OpenTelemetry entity > Kubernetes pod > other infrastructure Kubernetes entities
container.id - links to container entities.
Correlation chain: OpenTelemetry span > OpenTelemetry entity > container entity > other infrastructure entities
process.pid and host.id - links to process entities.
Correlation chain: OpenTelemetry span > OpenTelemetry entity > Process entity > other infrastructure entities
host.id - links to host entity.
Correlation chain: OpenTelemetry span > OpenTelemetry entity > Host entity

Make sure that the resource attributes are consistent between metrics and traces from the same service. For instructions about how to set these resource attributes, see Configuration example.

Strategy 2: OpenTelemetry traces only (by using OpenTelemetry Collector)

If your application emits only traces (spans) and not metrics, use the spanmetrics connector in the OpenTelemetry Collector to generate metrics from spans. This enables the same correlation strategy as Strategy 1.

connectors:
  spanmetrics: null
service:
  pipelines:
    metrics:
      receivers:
      - spanmetrics
    traces:
      exporters:
      - spanmetrics

Strategy 3: Infrastructure correlation for OpenTelemetry tracing with IBM middleware

Instana provides infrastructure linking for OpenTelemetry tracing with IBM middleware, including IBM MQ, IBM App Connect Enterprise (ACE), and IBM DataPower.

For calls to IBM MQ, you can link to a queue entity that is collected by the Instana IBM MQ sensor.
For calls to IBM ACE, you can link to a message flow entity that is collected by the Instana IBM ACE sensor.
For calls to IBM DataPower, you can link to a domain entity that is collected by the Instana IBM DataPower sensor.

Infrastructure links are typically created automatically for OpenTelemetry calls to IBM middleware and require no additional configuration. These links are displayed on the call details page.

The following example shows infrastructure linking for OpenTelemetry calls to the IBM MQ queue entity Q1:

Strategy 4: Infrastructure correlation for OpenTelemetry tracing with process entity

If your application emits only traces (spans) and you cannot use the OpenTelemetry Collector to generate metrics from spans, process-level correlation provides an alternative way to link spans to infrastructure entities. This strategy establishes a direct link between OpenTelemetry spans and process entities by using the following resource attributes:

process.pid, host.id, and entity.type (with value process)
Correlation chain: OpenTelemetry span > Process entity > other infrastructure entities

Note:

If your service emits both OpenTelemetry traces and metrics, infrastructure correlation falls back to Strategy 1.

Limitation: In Kubernetes environments, this strategy is difficult to use reliably because most OpenTelemetry SDKs provide the in-container process.pid, but Instana process entities use the host-level process PID to do the infrastructure correlation.

Configuration example

The OpenTelemetry official demo illustrates the best practices for setting service.name, service.instance.id, k8s.pod.uid, and other OpenTelemetry resource attributes.

In applications that use OpenTelemetry (OTel) manual instrumentation, the service.name is typically defined within the application code. However, if OTel automatic instrumentation is used, you must set it using the OTEL_SERVICE_NAME environment variable (supported by all language SDKs) or OTEL_RESOURCE_ATTRIBUTES. The following example demonstrates their usage:

      - env:
        - name: OTEL_SERVICE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/component']
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: service.name=$(OTEL_SERVICE_NAME),service.namespace=opentelemetry-demo

To add the service.instance.id, container.id, and k8s.pod.uid resource attributes, use the resource and k8sattributes processors in the OpenTelemetry Collector. Send both traces and metrics to Instana.

The OpenTelemetry SDK usually sets the process.pid resource attribute automatically.

You need to provide the host.id resource attribute only when you send OpenTelemetry data directly to the Instana backend. For more information, see Sending OpenTelemetry data to the Instana backend. You do not need to provide host.id when you send OpenTelemetry data to the Instana agent.

If your application emits only traces (spans) and not metrics, use the spanmetrics connector. Instana uses spanmetrics to generate OpenTelemetry entities, enabling the correlation of application spans with infrastructure entities (such as Kubernetes pods and physical nodes). spanmetrics helps ensure complete correlation and visibility within Instana.

The following example shows how to configure OpenTelemetry Collector with the k8sattributes processor, resource processor, and spanmetrics connector for optimal infrastructure correlation:

    connectors:
      spanmetrics: null
    exporters:
      otlp:
        endpoint: instana-agent.instana-agent:4317
        tls:
          insecure: true
    processors:
      batch: {}
      k8sattributes:
        extract:
          metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
          - container.id
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
      resource:
        attributes:
        - action: insert
          from_attribute: k8s.pod.uid
          key: service.instance.id
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            cors:
              allowed_origins:
              - http://*
              - https://*
            endpoint: ${env:MY_POD_IP}:4318
    service:
      pipelines:
        logs:
          exporters:
          - otlp
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
        metrics:
          exporters:
          - otlp
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
          - spanmetrics
        traces:
          exporters:
          - otlp
          - spanmetrics
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp

Note:

If your OpenTelemetry (OTel) applications already generate metrics, you do not need to configure the spanmetrics connector.
Make sure that the k8sattributes processor is placed before the resource processor in the pipeline.
Set the passthrough attribute of the k8sattributes processor to false.
The resource processor copies k8s.pod.uid to service.instance.id for proper entity identification.
If your OpenTelemetry SDK already provides service.name, you can retain it. You do not need to override it in the collector configuration. Configure service.name in the collector only when the SDK does not set it or when you need to standardize naming across multiple services.

Troubleshooting missing infrastructure correlation

Missing correlation with OpenTelemetry entities

If OpenTelemetry spans are not linking to OpenTelemetry entities, complete the following checks:

Verify that the OpenTelemetry entity exists:
1. From the Instana UI, click Infrastructure > Analyze Infrastructure.
2. Select OpenTelemetry from the entity types list.
3. Use service.name or service.instance.id as a tag filter to search for the corresponding OpenTelemetry entity.
4. If the entity does not exist, verify that your service is emitting OpenTelemetry metrics (not just traces).
Verify resource attributes match:
1. Ensure the service.name and service.instance.id resource attributes in your OpenTelemetry spans matches the service.name and service.instance.id values in the OpenTelemetry entity.
2. If using the OpenTelemetry Collector with multiple pipelines, verify that the resource processor configuration is consistent across pipelines for the same service.

Missing correlation with Kubernetes pods

If OpenTelemetry spans are not correlating with Kubernetes pods, complete the following checks:

Verify that the OpenTelemetry entity exists:
1. From the Instana UI, click Infrastructure > Analyze Infrastructure > OpenTelemetry.
2. Use service.name or service.instance.id to filter and locate your OpenTelemetry entity.
Check whether the Kubernetes pod appears in the OpenTelemetry entity breadcrumb:
1. Open the OpenTelemetry entity details.
2. Check the breadcrumb navigation at the top of the page.
3. If the Kubernetes pod appears in the breadcrumb, the correlation is working.
Verify that the Kubernetes pod exists in Instana:
1. From the Instana UI, click Infrastructure > Analyze Infrastructure > Pods.
2. Filter by k8s.pod.uid to search for the specific pod.
3. If the pod does not exist, verify that the Instana agent is running in the same Kubernetes cluster.
4. Confirm that the k8s.pod.uid value in your OpenTelemetry data matches the actual pod UID in Kubernetes.

Missing correlation with process entities

If OpenTelemetry spans are not correlating with process entities, complete the following checks:

Verify that the process.pid is correct:
1. In Kubernetes environments, ensure that you use the host-level process PID, not the in-container PID.
2. From the Instana UI, click Infrastructure > Analyze Infrastructure > Processes.
3. Open the process entity details and check the Process ID in the left panel.
4. Ensure that the process.pid in your OpenTelemetry spans matches this value.
Verify that the host.id is correct:
1. From the process entity details page, use the breadcrumb navigation at the top to go to the corresponding host entity.
2. Check the Host ID in the left panel of the host entity details.
3. Ensure that the host.id in your OpenTelemetry spans matches this value.

Missing correlation with container entities

If OpenTelemetry spans are not correlating with container entities, complete the following checks:

Verify that the OpenTelemetry entity exists:
1. From the Instana UI, click Infrastructure > Analyze Infrastructure > OpenTelemetry.
2. Use service.name or service.instance.id to filter and locate your OpenTelemetry entity.
Check whether the container appears in the OpenTelemetry entity breadcrumb:
1. Open the OpenTelemetry entity details.
2. Check the breadcrumb navigation for the entity.
3. If the container appears in the breadcrumb, the correlation is working.
Verify that the container entity exists in Instana:
1. From the Instana UI, click Infrastructure > Analyze Infrastructure.
2. Search for type container, then select one of the supported container runtimes (for example, Containerd, CRI-O, Docker, OTel Kubernetes, or Podman).
3. Filter by containerId to search for the specific container.
4. If the container does not exist, verify that the Instana agent is running in the same host.
5. Confirm that the container.id value in your OpenTelemetry data matches the actual container ID.

Infrastructure correlation missing for calls to IBM middleware

If the hostname collected by OpenTelemetry tracing for IBM middleware differs from the hostname that is collected by the Instana middleware sensor, you might encounter an Infrastructure: Correlation missing message in the Instana UI.

Depending on the environment where the IBM middleware is deployed, you can resolve this issue in the OpenTelemetry configuration or in the IBM middleware configuration:

OpenTelemetry configuration

Non-Kubernetes environment

If the IBM middleware is deployed on a standard virtual machine (VM), set a host alias for OpenTelemetry calls as follows:

To resolve this issue, set a host alias for OpenTelemetry calls by following these steps:

On the server that is running IBM MQ, IBM ACE, or IBM DataPower, stop the respective service and export an environment variable:
```
OTEL_RESOURCE_ATTRIBUTES=host.alias=<alias_of_the_host>
```
Replace <alias_of_the_host> with the hostname collected by the Instana middleware sensor.

For example, if the Instana IBM MQ sensor collects the hostname instmq1.example.com, but IBM MQ OpenTelemetry tracing collects the host's IP address, you must export the variable as follows:
```
OTEL_RESOURCE_ATTRIBUTES=host.alias=instmq1.example.com
```
Restart IBM MQ, IBM ACE, or IBM DataPower.

Kubernetes environment

When IBM middleware (such as IBM ACE, IBM MQ, or DataPower) is installed on a Kubernetes cluster, it runs in a container, which is different from the standard VM environment. Unlike VMs, containers have dynamic hostnames or IPs, which leads to unmonitored calls and incorrect infrastructure correlation in Instana.

To resolve this issue, complete the following steps:

Export environment variables in the IBM middleware Custom Resource (CR) instance. In the CR configuration file, add the following environment variables in the deployment/demonset section:

    spec:
      containers:
      - env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "host.alias=$(POD_IP)"

Wait for resync of the CR with the related deployments.
Restart IBM MQ, IBM ACE, or IBM DataPower, if required.

This configuration help ensure that each IBM middleware container carries its POD_IP as its host.alias value, enabling proper infrastructure correlation.

IBM middleware configuration

For IBM App Connect Enterprise integration server, you can configure OpenTelemetry trace for all message flows in an integration server and export span data to an OpenTelemetry collector.

In the ResourceManagers section of the server.conf.yaml file, set the properties openTelemetryHostName in the OpenTelemetryManager subsection to override the Hostname attribute for telemetry spans. For more information, see Configuring OpenTelemetry trace for an integration server.

Note:

To find the hostname in the Instana UI, go to Infrastructure > Analyze Infrastructure. From the "Entity types" list, Select IBM ACE Message Flow. Then, select the required instance from the list to view its details. The Host name is displayed under the IBM ACE Message Flow section.

Additional OpenTelemetry resources

For community-curated OpenTelemetry examples, best practices, and integration patterns, see the OpenTelemetry for Instana repository. This resource provides the following information:

Real-world configuration examples.
Troubleshooting guides.
Integration patterns for various technologies.
Community-contributed solutions.

OpenTelemetry related service mappings and infrastructure correlation

OpenTelemetry service mapping

Resource attribute service.name

Trace span attribute peer.service

Service mapping of well-known protocols

HTTP span attributes

Remote procedure calls span attributes

Database span attributes

Messaging span attributes

OpenTelemetry related infrastructure correlation

Correlation strategies

Strategy 1: OpenTelemetry traces and metrics (recommended)

Strategy 2: OpenTelemetry traces only (by using OpenTelemetry Collector)

Strategy 3: Infrastructure correlation for OpenTelemetry tracing with IBM middleware

Strategy 4: Infrastructure correlation for OpenTelemetry tracing with process entity

Configuration example

Troubleshooting missing infrastructure correlation

Missing correlation with OpenTelemetry entities

Missing correlation with Kubernetes pods

Missing correlation with process entities

Missing correlation with container entities

Infrastructure correlation missing for calls to IBM middleware

OpenTelemetry configuration

IBM middleware configuration

Additional OpenTelemetry resources

Resource attribute `service.name`

Trace span attribute `peer.service`