OpenTelemetry related service mappings and infrastructure correlation

The service.name attribute uniquely identifies a service in OpenTelemetry. Each application, microservice, or component have a unique and logical service.name to enable effective filtering, analysis, and visualization of telemetry data. Instana maps service.name to its services, linking traces, metrics, and logs to the emitting application. Consistent and descriptive naming of service.name helps ensure accurate grouping and improved observability within Instana.

In OpenTelemetry, you can set the service.name attribute at different levels of your application, depending on the specific needs of your observability setup. In the following table, you can see the different ways in which service.name can be set:

In OpenTelemetry, the peer.service attribute specifies the name of the remote service a span connects to, helping to identify external dependencies like databases, message brokers, or other microservices. Instana maps peer.service to an Instana service, enabling the visualization of service dependencies and interactions.

• Instrumentation Libraries: Some libraries automatically set peer.service when they are properly configured. For example:
  • JDBC: The OpenTelemetry JDBC instrumentation can infer peer.service from database connection details.
  • HTTP Clients: HTTP libraries such as Apache HttpClient can derive peer.service based on the remote service’s hostname.

In the following table, you can see the different ways in which peer.service can be set or configured in OpenTelemetry:

In addition to general attributes such as service.name and peer.service, OpenTelemetry defines a comprehensive set of semantic conventions for well-known protocols, including HTTP, RPC, databases, messaging systems, and more.

For a complete list of protocols, refer to the Trace Semantic Conventions.

OpenTelemetry resource attributes, such as service.name and service.instance.id, are essential for linking entities, spans, and log records. The following are some examples of how to create name based on the OTel resource attributes of spans, metrics, or logs:

1. service.instance.id@service.name: Used when both values are not blank. For example:
   • myInst1@my-service1
2. process.pid@container.id: Used when container.id is not blank. For example:
   • 2135@e4a130e60c8e70958b95e887235047be0305431c3ab2dd895ee54442c5b92cd3
   • UNNOWN_PID@e4a130e60c8e70958b95e887235047be0305431c3ab2dd895ee54442c5b92cd3
3. process.pid@host.id: Used when process.pid is not 1 and both process.pid and host.id are not blank. For example:
   • 2135@e378ef316121abc73e6dd16262b34dc2
4. process.pid@host.name: Used when process.pid is not 1 and both process.pid and host.name are not blank. For example:
   • 2135@stand-dev.fyre.ibm.com

The k8s.pod.uid attribute is important in Kubernetes and Red Hat OpenShift environments. For more information about definitions of these resource attributes, see Resource Semantic Conventions and Resource Semantic Conventions for Kubernetes.

The OpenTelemetry official demo illustrates the best practices for setting service.name, service.instance.id, k8s.pod.uid, and other OpenTelemetry resource attributes.

In applications that use OpenTelemetry (OTel) manual instrumentation, the service.name is typically defined within the application code. However, if OTel automatic instrumentation is used, you must set it using the OTEL_SERVICE_NAME environment variable (supported by all language SDKs) or OTEL_RESOURCE_ATTRIBUTES. The following example demonstrates their usage:

      - env:
        - name: OTEL_SERVICE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/component']
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: service.name=$(OTEL_SERVICE_NAME),service.namespace=opentelemetry-demo
 

To reliably add the service.instance.id and k8s.pod.uid resource attributes, use the k8sattributes and resource processors in the OpenTelemetry Contrib Collector. Make sure that both traces and metrics are sent to Instana.

If your application emits only traces (spans) and not metrics, use the spanmetrics connector. Instana uses spanmetrics to generate OpenTelemetry entities, enabling the correlation of application spans with infrastructure entities (such as Kubernetes pods and physical nodes). spanmetrics helps ensure complete correlation and visibility within Instana.

Refer to the following example YAML file of the OpenTelemetry Contrib Collector:

    connectors:
      spanmetrics: null
    exporters:
      otlp:
        endpoint: instana-agent.instana-agent:4317
        tls:
          insecure: true
    processors:
      batch: {}
      k8sattributes:
        extract:
          metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
          - container.id
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
      resource:
        attributes:
        - action: insert
          from_attribute: k8s.pod.uid
          key: service.instance.id
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            cors:
              allowed_origins:
              - http://*
              - https://*
            endpoint: ${env:MY_POD_IP}:4318
    service:
      pipelines:
        logs:
          exporters:
          - otlp
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
        metrics:
          exporters:
          - otlp
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
          - spanmetrics
        traces:
          exporters:
          - otlp
          - spanmetrics
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
 

Notes:

• If your OpenTelemetry (OTel) applications already generate metrics, you do not need to configure the spanmetrics connector.
• Make sure that the k8sattributes processor is placed before the resource processor in the pipeline.
• Set the passthrough attribute of the k8sattributes processor to false.

Instana provides infrastructure linking for OpenTelemetry tracing with IBM middleware, including IBM MQ, IBM App Connect Enterprise (ACE), and IBM DataPower.

• For calls to IBM MQ, you can link to a queue entity collected by the Instana IBM MQ sensor.
• For calls to IBM ACE, you can link to a message flow entity collected by the Instana IBM ACE sensor.
• For calls to IBM DataPower, you can link to a domain entity collected by the Instana IBM DataPower sensor.

Infrastructure links are typically created automatically for OpenTelemetry calls to IBM middleware, requiring no additional configuration. These links are displayed on the call details page.

The following example demonstrates infrastructure linking for OpenTelemetry calls to the IBM MQ queue entity Q1:

If the hostname collected by OpenTelemetry tracing for IBM middleware differs from the hostname that is collected by the Instana middleware sensor, you might encounter an Infrastructure: Correlation missing message in the Instana UI.

Depending on the environment where the IBM middleware is deployed, you can resolve this issue in the OpenTelemetry configuration or in the IBM middleware configuration: