OpenTelemetry related service mappings and infrastructure correlation

OpenTelemetry related service mappings
OpenTelemetry related infrastructure correlation
- Using service.name, service.instance.id and k8s.pod.uid resource attributes
- Infrastructure correlation for OpenTelemetry tracing with IBM middleware
Troubleshooting
- Infrastructure correlation missing for calls to IBM middleware

Instana supports the resource attributes of OpenTelemetry and many commonly used definitions in OpenTelemetry Resource Semantic Conventions. Instana shows the OpenTelemetry service flows and OpenTelemetry resources as infrastructure entities and can link those entities with other physical entities (for example, process, host, or container), trace spans, and log records especially when an Instana agent runs in the same host. If an Instana agent is installed in the same host where apps enabled with OpenTelemetry are running, send the OpenTelemetry data to the local Instana agent (instead of the Instana backend). Sending the data to the local Instana agent achieves better links with the entities that are monitored by the Instana agent.

If you are using the language-based OpenTelemetry auto-instrumentation, use the latest OpenTelemetry agent, so that the agent can produce enough resource attributes for the linking between entities, spans, and log records. Besides service.name and service.instance.id, the most important resource attributes for linking include process.pid, container.id, host.id, and especially k8s.pod.uid when it is in Kubernetes environments.

Using `service.name`, `service.instance.id` and `k8s.pod.uid` resource attributes

OpenTelemetry resource attributes, such as service.name and service.instance.id, are used for linking entities, spans, and log records. Another resource attribute k8s.pod.uid is also very important in the Kubernetes or Red Hat OpenShift environments. For more information about the definitions of these resource attributes, see Resource Semantic Conventions and Resource Semantic Conventions for Kubernetes.

The OpenTelemetry official demo shows the optimized way to set the service.name, service.instance.id, k8s.pod.uid, and other OpenTelemetry resource attributes.

In an application with OTel manual instrumentation, the service.name is usually set in the built-in code. If the OTel automatic instrumentation is used, you need to use environment variable OTEL_SERVICE_NAME (supported by all language SDKs) or OTEL_RESOURCE_ATTRIBUTES. The following example shows their usage:

      - env:
        - name: OTEL_SERVICE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['app.kubernetes.io/component']
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: service.name=$(OTEL_SERVICE_NAME),service.namespace=opentelemetry-demo

The stable way of adding the service.instance.id and k8s.pod.uid resource attributes is to use the k8sattributes processor and resource processor of the OpenTelemetry Contrib Collector. Make sure to send both traces and metrics to Instana.

Refer to the following example yaml file of OpenTelemetry Contrib Collector:

    connectors:
      spanmetrics: null
    exporters:
      otlp:
        endpoint: 'instana-agent.instana-agent:4317'
        tls:
          insecure: true
    processors:
      batch: {}
      k8sattributes:
        extract:
          metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
          - container.id
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
      resource:
        attributes:
        - action: insert
          from_attribute: k8s.pod.uid
          key: service.instance.id
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
          http:
            cors:
              allowed_origins:
              - http://*
              - https://*
            endpoint: ${env:MY_POD_IP}:4318
    service:
      pipelines:
        logs:
          exporters:
          - otlp
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
        metrics:
          exporters:
          - otlp
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp
          - spanmetrics
        traces:
          exporters:
          - otlp
          - spanmetrics
          processors:
          - k8sattributes
          - resource
          - batch
          receivers:
          - otlp

Note:

If you already have metrics of your OTel apps, you do not need to define spanmetrics.
Make sure to set the k8sattributes processor before the resource processor in the pipeline.
Make sure to set the passthrough attribute of the k8sattributes processor to false.

Infrastructure correlation for OpenTelemetry tracing with IBM middleware

Instana supports infrastructure linking for OpenTelemetry tracing with IBM middleware, including IBM MQ, IBM App Connect Enterprise (ACE), and IBM DataPower. For a call to IBM MQ, you can link to a queue entity that is collected by the Instana IBM MQ sensor. For a call to IBM ACE, you can link to a message flow entity that is collected by the Instana IBM ACE sensor. For a call to IBM DataPower, you can link to an IBM DataPower domain entity that is collected by the Instana IBM DataPower sensor.

Typically, infrastructure links are created automatically for OpenTelemetry calls to IBM middleware, including IBM MQ, IBM ACE, and IBM DataPower. You don't need to do anything. The infrastructure link is displayed automatically in the call details page.

In the following example, you can see the infrastructure linking for OpenTelemetry calls to the queue entity Q1 of IBM MQ:

Troubleshooting

Infrastructure correlation missing for calls to IBM middleware

If the hostname that is collected by OpenTelemetry Tracing for IBM middleware differs from the hostname that is collected by the Instana middleware sensor, you might get an Infrastructure: Correlation missing message in the Instana UI.

To resolve this issue, specify a host alias for OpenTelemetry calls. Follow the steps:

On the server where the IBM MQ, IBM ACE, or IBM Datapower is running, stop IBM MQ, IBM ACE, or IBM Datapower, and then export an environment variable as follows:
```
OTEL_RESOURCE_ATTRIBUTES=host.alias=<alias_of_the_host>
```
<alias_of_the_host> must match the hostname that is collected by the Instana middleware sensor.

For example, if the hostname that is collected by IBM MQ sensor is instmq1.example.com but the hostname that is collected by IBM MQ OTel tracing is the IP address of the host, you need to export an environment variable as follows:
```
OTEL_RESOURCE_ATTRIBUTES=host.alias=instmq1.example.com
```
Start IBM MQ, IBM ACE, or IBM Datapower.

OpenTelemetry related service mappings and infrastructure correlation