OpenTelemetry related service mappings and infrastructure correlation
You can follow the guidance in this document to integrate OpenTelemetry with Instana and enable comprehensive service mappings and infrastructure correlation. This document explains how to use specific OpenTelemetry resource attributes for optimal data linking and visibility within Instana, particularly in Kubernetes environments and IBM middleware.
OpenTelemetry service mapping
Instana aligns with OpenTelemetry Semantic Conventions to map metrics, spans, and logs to services. These conventions establish standardized attribute names and usage patterns, promoting consistency and interoperability across observability tools. By aligning to these standards, Instana make sure the seamless integration of telemetry data into its monitoring and observability platform.
The following are the key attributes that are used in service mapping:
- Resource attribute
service.name
- Trace span attribute
peer.service
Resource attribute service.name
The service.name
attribute uniquely identifies a service in OpenTelemetry. Each application, microservice, or component have a unique and logical service.name
to enable effective filtering, analysis, and visualization
of telemetry data. Instana maps service.name
to its services, linking traces, metrics, and logs to the emitting application. Consistent and descriptive naming of service.name
helps ensure accurate grouping and improved
observability within Instana.
In OpenTelemetry, you can set the service.name
attribute at different levels of your application, depending on the specific needs of your observability setup. In the following table, you can see the different ways in which service.name
can be set:
Level | Definition Method | Example | Use Case |
---|---|---|---|
Application Code | Directly in the application during manual instrumentation | service.name="orders-service" |
When the service.name is hardcoded or configured in the application itself |
Environment Variable | By using the OTEL_SERVICE_NAME environment variable |
OTEL_SERVICE_NAME=payments-service |
Common for environments that use auto-instrumentation across multiple services |
Resource Attributes | By using OTEL_RESOURCE_ATTRIBUTES with key-value pairs |
OTEL_RESOURCE_ATTRIBUTES=service.name=inventory-service |
Useful for combining service.name with other attributes like host.id |
Collector Configuration | In OpenTelemetry Collector by using the resource processor |
service.name: shipping-service |
Centralized control for telemetry data before it is sent to the backend |
Trace span attribute peer.service
In OpenTelemetry, the peer.service
attribute specifies the name of the remote service a span connects to, helping to identify external dependencies like databases, message brokers, or other microservices. Instana maps peer.service
to an Instana service, enabling the visualization of service dependencies and interactions.
- Instrumentation Libraries: Some libraries automatically set
peer.service
when they are properly configured. For example:- JDBC: The OpenTelemetry JDBC instrumentation can infer
peer.service
from database connection details. - HTTP Clients: HTTP libraries such as Apache HttpClient can derive
peer.service
based on the remote service’s hostname.
- JDBC: The OpenTelemetry JDBC instrumentation can infer
In the following table, you can see the different ways in which peer.service
can be set or configured in OpenTelemetry:
Level | Description | Example | Use Case |
---|---|---|---|
Application Code | Directly set the peer.service attribute in your code by using the OpenTelemetry API |
span.setAttribute("peer.service", "backend-service") |
When you have specific knowledge about the target service and want to provide a custom name |
Configuration Files (System Property) | Define the peer.service value in system properties that are used by the OpenTelemetry agent or SDK |
otel.instrumentation.common.peer-service-mapping: "api.example.com=example-api-service,1.2.3.4=cats-service" |
When you want to map specific hostnames, IP addresses, or a combination of both to service names in a configuration file |
Environment Variables | Set the peer.service value as an environment variable accessible to your application |
OTEL_INSTRUMENTATION_COMMON_PEER_SERVICE_MAPPING="api.example.com=example-api-service,1.2.3.4=cats-service" |
When you need to dynamically set the peer.service value based on runtime conditions or deployment environment (similar to system properties but by using environment variables) |
Instrumentation Library | Automatically set by OpenTelemetry libraries | peer.service="postgresql" |
Common for popular libraries such as database and HTTP clients which set peer.service automatically |
Service mapping of well-known protocols
In addition to general attributes such as service.name
and peer.service
, OpenTelemetry defines a comprehensive set of semantic conventions for well-known protocols, including HTTP, RPC, databases, messaging systems,
and more.
For a complete list of protocols, refer to the Trace Semantic Conventions.
HTTP span attributes
OpenTelemetry provides semantic conventions for HTTP spans to maintain consistency across tracing systems, including attributes that offer insights into HTTP requests and responses.
In the following table, you can see the different ways in which http
span attributes can be set or configured in OpenTelemetry:
Attribute | Description | Example |
---|---|---|
http.method |
HTTP method (GET, POST) | http.method="GET" |
http.route |
Route or path of the HTTP request | http.route="/webshop/articles/:id" |
url.path |
Path component of the full URL | url.path="/webshop/articles/4" |
http.host |
Hostname of the server | http.host="api.example.com" |
http.target |
Target URL of the request | http.target="/users" |
While http.host
and http.target
can often infer the service, dynamic routing or service discovery might require additional information from configuration files or service registries for accurate mapping.
Remote procedure calls span attributes
OpenTelemetry provides semantic conventions for Remote Procedure Call (RPC) spans to maintain consistency across tracing systems, including attributes that offer insights. The key RPC span attributes used for service mapping in OpenTelemetry are:
- rpc.method: Specifies the method or function which is invoked, providing detailed information about the operation.
- rpc.service: Identifies the name of the service that is called, which is crucial for mapping RPC calls to specific services.
Attribute | Description | Example |
---|---|---|
rpc.service |
Name of the remote service | rpc.service="user-service" |
rpc.method |
A method or function that is invoked | rpc.method="GetUserInfo" |
By combining rpc.service
and rpc.method
with other attributes such as peer.service
, RPC calls can be mapped to their corresponding services.
Database span attributes
OpenTelemetry provides semantic conventions for Database spans to maintain consistency across tracing systems, offering key attributes that provide valuable insights into database operations and interactions. These insights are essential for understanding the performance and behavior of database systems within a distributed application.
In the following table, you can see the key database span attributes used for service mapping in OpenTelemetry.
Attribute | Description | Example |
---|---|---|
db.system |
Database system (For example, MySQL and PostgreSQL) | db.system="mysql" |
db.instance |
Database instance name or identifier | db.instance="my_database" |
db.name |
The name of the database which is accessed | db.name="users_db" |
db.statement |
SQL query or database command executed | db.statement="SELECT * FROM users" |
db.operation |
Type of database operation (For example, SELECT and INSERT) | db.operation="SELECT" |
While these attributes primarily describe database interactions, they can indirectly assist with service mapping. For example, the db.instance
attribute might help identify the service interacting with a specific database instance.
Messaging span attributes
OpenTelemetry provides semantic conventions for Messaging spans to maintain consistency across tracing systems, including attributes that offer insights.
The key Messaging span attributes used for service mapping in OpenTelemetry are:
Attribute | Description | Example |
---|---|---|
messaging.system |
Messaging system used (For example, Kafka, RabbitMQ, and AMQP) | messaging.system="kafka" |
messaging.destination |
Destination address (For example, topic name and queue name) | messaging.destination="user-events" |
messaging.operation |
Operation performed (For example, SEND, RECEIVE, PUBLISH, and SUBSCRIBE) | messaging.operation="SEND" |
These attributes help map services based on the messaging systems that they interact with and the specific topics or queues they use. For example, if a service is known to send messages to a specific Kafka topic, the messaging.destination
attribute can help identify the service.
OpenTelemetry related infrastructure correlation
Instana supports the resource attributes of OpenTelemetry and many commonly used definitions in OpenTelemetry Resource Semantic Conventions. Instana displays OpenTelemetry service flows and resources as infrastructure entities. These entities can be linked to physical entities such as processes, hosts, or containers, in addition to trace spans and log records, especially when an Instana agent runs on the same host. If an Instana agent is installed in the same host where apps enabled with OpenTelemetry are running, send the OpenTelemetry data to the local Instana agent (instead of the Instana backend). Sending the data to the local Instana agent achieves better links with the entities that are monitored by the Instana agent.
If you are using language-based OpenTelemetry auto-instrumentation, make sure that you use the latest OpenTelemetry agent. This makes sure that the agent can generate sufficient resource attributes to establish links between entities, spans,
and log records. In addition to service.name
and service.instance.id
, critical resource attributes for linking include process.pid
, container.id
, host.id
, and k8s.pod.uid
in Kubernetes environments.
In Kubernetes environments, the k8s.pod.uid
resource attribute is critical for linking OpenTelemetry entities to Kubernetes pods. It establishes relationships between spans, logs, and metrics associated with these pods, providing
a cohesive view of application and infrastructure performance.
Using service.name
, service.instance.id
and k8s.pod.uid
resource attributes
OpenTelemetry resource attributes, such as service.name
and service.instance.id
, are essential for linking entities, spans, and log records. The k8s.pod.uid
attribute is important in Kubernetes and Red
Hat OpenShift environments. For detailed definitions of these resource attributes, see Resource Semantic Conventions and Resource Semantic Conventions for Kubernetes.
The OpenTelemetry official demo illustrates the best practices for setting service.name
, service.instance.id
, k8s.pod.uid
, and other
OpenTelemetry resource attributes.
In applications that use OpenTelemetry (OTel) manual instrumentation, the service.name
is typically defined within the application code. However, if OTel automatic instrumentation is used, you must set it using the OTEL_SERVICE_NAME
environment variable (supported by all language SDKs) or OTEL_RESOURCE_ATTRIBUTES
. The following example demonstrates their usage:
- env:
- name: OTEL_SERVICE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.labels['app.kubernetes.io/component']
- name: OTEL_RESOURCE_ATTRIBUTES
value: service.name=$(OTEL_SERVICE_NAME),service.namespace=opentelemetry-demo
To reliably add the service.instance.id
and k8s.pod.uid
resource attributes, use the k8sattributes
and resource
processors in the OpenTelemetry Contrib Collector. Make sure that both traces
and metrics are sent to Instana.
If your application emits only traces (spans) and not metrics, use the spanmetrics
connector. Instana uses spanmetrics
to generate OpenTelemetry entities, enabling the correlation of application spans with infrastructure
entities (such as Kubernetes pods and physical nodes). spanmetrics
helps ensure complete correlation and visibility within Instana.
Refer to the following example YAML file of the OpenTelemetry Contrib Collector:
connectors:
spanmetrics: null
exporters:
otlp:
endpoint: 'http://instana-agent.instana-agent:4317'
tls:
insecure: true
processors:
batch: {}
k8sattributes:
extract:
metadata:
- k8s.namespace.name
- k8s.deployment.name
- k8s.statefulset.name
- k8s.daemonset.name
- k8s.cronjob.name
- k8s.job.name
- k8s.node.name
- k8s.pod.name
- k8s.pod.uid
- k8s.pod.start_time
- container.id
passthrough: false
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.ip
- sources:
- from: resource_attribute
name: k8s.pod.uid
- sources:
- from: connection
resource:
attributes:
- action: insert
from_attribute: k8s.pod.uid
key: service.instance.id
receivers:
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:4317
http:
cors:
allowed_origins:
- http://*
- https://*
endpoint: ${env:MY_POD_IP}:4318
service:
pipelines:
logs:
exporters:
- otlp
processors:
- k8sattributes
- resource
- batch
receivers:
- otlp
metrics:
exporters:
- otlp
processors:
- k8sattributes
- resource
- batch
receivers:
- otlp
- spanmetrics
traces:
exporters:
- otlp
- spanmetrics
processors:
- k8sattributes
- resource
- batch
receivers:
- otlp
Notes:
- If your OpenTelemetry (OTel) applications already generate metrics, you do not need to configure the
spanmetrics
connector. - Make sure that the
k8sattributes
processor is placed before theresource
processor in the pipeline. - Set the
passthrough
attribute of thek8sattributes
processor tofalse
.
Infrastructure correlation for OpenTelemetry tracing with IBM middleware
Instana provides infrastructure linking for OpenTelemetry tracing with IBM middleware, including IBM MQ, IBM App Connect Enterprise (ACE), and IBM DataPower.
- For calls to IBM MQ, you can link to a queue entity collected by the Instana IBM MQ sensor.
- For calls to IBM ACE, you can link to a message flow entity collected by the Instana IBM ACE sensor.
- For calls to IBM DataPower, you can link to a domain entity collected by the Instana IBM DataPower sensor.
Infrastructure links are typically created automatically for OpenTelemetry calls to IBM middleware, requiring no additional configuration. These links are displayed on the call details page.
The following example demonstrates infrastructure linking for OpenTelemetry calls to the IBM MQ queue entity Q1
:
Troubleshooting
Infrastructure correlation missing for calls to IBM middleware
If the hostname collected by OpenTelemetry tracing for IBM middleware differs from the hostname that is collected by the Instana middleware sensor, you might encounter an Infrastructure: Correlation missing
message
in the Instana UI.
To resolve this issue, set a host alias for OpenTelemetry calls by following these steps:
-
On the server that is running IBM MQ, IBM ACE, or IBM DataPower, stop the respective service and export an environment variable:
OTEL_RESOURCE_ATTRIBUTES=host.alias=<alias_of_the_host>
Replace <alias_of_the_host> with the hostname collected by the Instana middleware sensor.
For example, if the Instana IBM MQ sensor collects the hostname
instmq1.example.com
, but IBM MQ OpenTelemetry tracing collects the host's IP address, you must export the variable as follows:OTEL_RESOURCE_ATTRIBUTES=host.alias=instmq1.example.com
-
Restart IBM MQ, IBM ACE, or IBM DataPower.