OpenTelemetry tracing with IBM MQ on z/OS

Queue managers running on z/OS® can take part in OpenTelemetry traces initiated either on distributed or z/OS and automatically propagate those traces to other z/OS middleware.

High level overview of OpenTelemetry trace

An OpenTelemetry trace provides information about the processing performed on behalf of a specific application request through the components that make up an application. Each component contributes to the trace by emitting a span. A span contains information, span attributes, about the work performed by the component. Spans build up parent-child relationships which allow their position in the trace to be calculated.

A trace is represented by a globally unique number – a trace ID. Each span in the trace refers to the same trace ID.

The first span in a trace is known as the root span. Each span, apart from the root span, has a span immediately preceding it, this is known as the parent of the span. Each span might have zero, one or more spans immediately following it, these are known as child spans. Every span in a trace has a unique number – a span ID. Span IDs are used to build up the parent – child relationships in a trace. Span IDs must be unique inside a given trace.

Information about a trace is stored in an environment specific manner as trace context. At a minimum trace context consists of the trace ID and the most recent span ID. Trace context is propagated between the components that make up an application. As those component emit spans, the trace ID in the trace context stays constant, but the span ID in the trace context is changed to represent the most recent span that has been emitted.

Spans are emitted to an observability backend which stores the spans, and allows them to be visualized as a trace. Example observability backends include IBM® Instana® as well as open source and third-party tools.

This process is shown in Figure 1:

This figure is described in the surrounding text. — Figure 1. OpenTelemetry trace overview

High level overview of OpenTelemetry trace in IBM MQ for z/OS

In IBM MQ, trace context is represented by message properties. One of these message properties, traceparent contains the trace ID and span ID. Applications, or the environment that the applications run in, can set these properties to enable IBM MQ for z/OS to take part in a trace.

If IBM MQ is enabled for OpenTelemetry, then a queue manager generates a span when a message is put to, or destructively got from, a queue. The span generated when a message is put is known as a producer span, the span generated when a message is got is known as a consumer span.

A queue manager emits spans to the z/OS operating system and the spans are then sent to any observability backend that support the OTLP protocol. The observability backend, may, or may not, run on z/OS.

Emission of spans is controlled by the OTELTRAC attribute, on the queue manager and queue objects. For more information, see Span emission.

Trace context can be propagated from an application component into a message when it is put, or from a message into an application component when it is got. This propagation can be either manual or automatic.

Manual propagation is performed by the application using the IBM MQ message property APIs.
Automatic propagation is performed by the environment the application component is running in. Automatic propagation is available between IBM MQ for z/OS and CICS® using either the IBM MQ-CICS adapter, or the IBM MQ-CICS bridge.
Third party instrumentation provides automatic propagation in other environments, for example JMS.

Trace context automatically flows between queue managers using channels. For example trace context will flow between z/OS queue managers, or between z/OS and distributed queue managers.

Trace context propagation is controlled by the OTELPCTL attribute, on the queue manager and queue objects. For more information on trace context propagation, see Trace propagation.

Figure 2 shows a queue manager taking part in a trace between two z/OS applications. Spans 2 and 3 are emitted by the queue manager.