Kafka Header Migration

Summary

Instana will change its header format for Kafka trace correlation over the next few months. You don't need to take any actions. The only reason you might want to take a closer look is if you are very concerned about message size. While the migration is ongoing, the message size will increase slightly. The overhead Instana adds to messages is negligible for almost all use cases. But if you want to save a few bytes per message, read on.

Background

To enable distributed tracing, Instana adds trace correlation information to requests and messages. The exact format depends on the underlying communication protocol. For Kafka messages, Instana currently sends trace correlation information in a binary format. The Kafka protocol allows arbitrary binary values for headers, so this behavior is fully compliant with the Kafka message format. However, some Kafka client implementations don't deal well with binary headers. For this reason, Instana will move to trace correlation headers with string values instead of binary values. Changing the format of trace correlation headers for a communication protocol is not trivial, because Instana does not have control over the point in time when tracer components are updated for your applications. Instana cannot assume that all tracers that participate in any given trace are updated at the same time. Furthermore, the header format that a tracer injects into a message needs to be understood by the tracer that monitors the service that receives that message. Therefore it is vital that the migration is performed in a way that is backwards compatible. We have prepared this move for a while, and all our tracers can already deal with both the old as well as the new header format.

In summary, this migration will be implemented in multiple phases, and allow for a transitional period, to ensure that trace correlation across Kafka always works out of the box.

Tracer Versions

The following tracer versions are ready to process the new string header format, and they also allow configuring which header format to use when messages are sent. You don't need to update to these tracer versions immediately, unless you want to configure the header format explicitly for your services that send Kafka messages. That being said, Instana always recommends to update tracers regularly.

Runtime Version Release Date Comment
Golang instasarama@v1.2.0 May 25 2022 This tracer is updated manually by installing the latest package version.
Java Java Trace Sensor 1.2.413 May 31 2022 This tracer is usually automatically updated by the Instana agent.
.NET Core Instana.Tracing.Core@1.228.1 July 7 2022 This tracer is updated manually by installing the latest nuget version (unless you use the Kubernetes AutoTrace webhook).
Node.js @instana/collector@2.3.0 May 24 2022 This tracer is updated manually by installing the latest npm package version (unless you use the Kubernetes AutoTrace webhook).

Other runtimes besides the ones listed here are not affected by this change.

Additionally, the following tracer versions send both header formats by default:

Runtime Version Release Date Comment
Golang instasarama@v1.5.0 October 4 2022 This tracer is updated manually by installing the latest package version.
Java Java Trace Sensor 1.2.425 October 4 2022 This tracer is usually automatically updated by the Instana agent.
.NET Core Instana.Tracing.Core@1.235.1 October 5 2022 This tracer is updated manually by installing the latest nuget version (unless you use the Kubernetes AutoTrace webhook).
Node.js @instana/collector@2.10.0 October 5 2022 This tracer is updated manually by installing the latest npm package version (unless you use the Kubernetes AutoTrace webhook).

Migration Phases

Phase 0

In this phase, tracers only send the binary headers X_INSTANA_C and X_INSTANA_L by default.

Sending string headers instead of binary headers, or sending both sets of headers is possible. For more information, see Configuration Options.

When a message is received, tracers will look for both sets of headers, and are able to continue a trace from both header formats.

This phase has ended with the start of phase 1.

Phase 1

This is the current phase of the migration.

At the moment, all tracers send both binary and string headers by default, that is, X_INSTANA_C and X_INSTANA_L as well as X_INSTANA_T and X_INSTANA_S. This transitional phase allows tracers to be gradually upgraded across your IT infrastructure.

If you are concerned about having four Instana headers attached instead of two, you can apply the following migration strategy to optimize the message header overhead:

  1. Configure all services that send Kafka message to send only binary headers. See Configuration Options.
  2. At some point before September 2023, make sure that all tracers are upgraded to a version that supports both header formats.
  3. Configure all services that send Kafka message to send only string headers. See Configuration Options. By using this setting, the services are no longer affected by the ongoing migration.
  4. After all tracers are upgraded to a phase 2 version, the configuration for the header format can be removed. But it is also not an issue if the configuration stays in place, it will simply be ignored.

You can also skip step 1 and use the new header format directly, if all tracers have already been upgraded to a version that is new enough.

Phase 2

This phase will start approximately in September 2023.

Starting with this phase, Instana tracers will switch from sending the legacy binary header format to the new string header format by default. The configuration for the header format (if present) will be ignored. This is the final phase of the migration.

Configuration Options

The Kafka header format can be configured with two different approaches, either in the host agent configuration or by using an environment variable.

Agent Configuration Options

Set com.instana.tracing.kafka.header-format to binary, string, or both. Here is an example configuration:

com.instana.tracing:
  kafka:
    header-format: both # possible values: binary, both, string

An option is provided to disable injecting headers into Kafka messages completely. See the following example settings:

This setting will disable trace correlation for Kafka entirely. It is not recommended to set this, because it will split traces - the first trace will end at the service that produces the Kafka message, and the service that receives the Kafka message will start a new trace.

com.instana.tracing:
  kafka:
    trace-correlation: false

Environment Variables

  • Set the environment variable INSTANA_KAFKA_HEADER_FORMAT on the monitored process. Valid values for the environment variable are binary, string, or both.
  • An option is also provided to disable injecting headers into Kafka messages completely. This will also disable trace correlation for Kafka. The setting for this is INSTANA_KAFKA_TRACE_CORRELATION=false.

Header Names

The header formats are described in detail in the section Kafka Tracing Headers.

Legacy/Binary Headers

In legacy mode, trace correlation for Kafka messages uses the two headers X_INSTANA_C and X_INSTANA_L, with binary content.

Modern/String Headers

After the migration, or when you configure to send string headers, trace correlation for Kafka message uses the headers X_INSTANA_T, X_INSTANA_S (and optionally X_INSTANA_L_S). All three headers are sent with string values.