Concepts of tracing

Callstacks

A callstack is an ordered list of code executions. Whenever code invokes other code, the new code is put onto the top of the stack. Callstacks are used by runtimes of all programming languages and are usually print as a stacktrace. When an error occurs, the stacktrace allows you to trace back to the calls that led to the error.

For example, the following error message states Apple is not a number. Combined with the callstack, it's possible to narrow down where in a complex system the error occurred. The message alone is usually insufficient, as the NumberUtil algorithm might be used in many places.

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      ShoppingCart.update()
        ShoppingCart.updateCart()
          ShoppingCart.parseQuantity()
            ShoppingCart.convertParams()
              NumberUtil.convert()  <-- Error: "Apple is not a number"

To understand why the error occurred, use the callstack to trace back from the error to the relevant business method, which in this case is ShoppingCart.parseQuantity().

Callstacks themselves are insufficient for monitoring. They are not easy to read and do not provide information to correlate performance and availability of a system to overall health. To see what happens on a code execution and to correlate much more information like process activity, resource usage, queuing, access patterns, load, and throughput, system, and application health needs to be taken into account.

Distributed tracing

With the introduction of service oriented architectures (SOA), the callstack was broken apart. For example, the ShoppingCart logic may now reside on server A, while NumberUtil resides on server B. An error trace on server B only contains the short callstack of the parse error, while on server A a new error was produced stating that something went wrong on server B, but not stating the problem.

Instead of a single error callstack that is easy to troubleshoot, you ended up with two callstacks with two errors. Along with this, there was no connection between the two, making it impossible to have access to both at the same time.

Server A:

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      ShoppingCart.update()
        ShoppingCart.updateCart()
          ShoppingCart.parseQuantity()
            ShoppingCart.convertParams()
              RestClient.invokeConversion() <-- Error: Unkown

Server B:

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      NumberUtil.convert()  <-- Error: "Apple is not a number"

The idea behind distributed tracing was to fix this problem by connecting the two error call stacks with each other. Most implementations use a simple mechanism to so; when server A calls server B, the APM tool adds an identifier to the call that serves as a common reference point between the callstacks in the APM system. This mechanism is called correlation and to produce one error, it joins the two callstacks.

Thread.run()
  HttpFramework.service()
    HttpFramework.dispatch()
      ShoppingCart.update()
        ShoppingCart.updateCart()
          ShoppingCart.parseQuantity()
            ShoppingCart.convertParams()
              RestClient.invokeConversion()
                Thread.run()
                  HttpFramework.service()
                    HttpFramework.dispatch()
                      NumberUtil.convert()  <-- Error: "Apple is not a number"

With added decoration, where the remote call takes place and on which servers parts of the callstack were executed, enables you to find out that the ShoppingCart was the context of the error, and the NumberUtil caused the shopping cart activity to fail.

Measuring performance

However, the preceding examples have illustrated error tracing APM tools use the same mechanism for taking and presenting performance measurements. The trace is annotated with performance numbers like this:

413 Thread.run()
413   HttpFramework.service()
413     HttpFramework.dispatch()
412       ShoppingCart.update()
411         ShoppingCart.updateCart()
211           ShoppingCart.parseQuantity()
210             ShoppingCart.convertParams()
200               RestClient.invokeConversion()
 10                 Thread.run()
 10                   HttpFramework.service()
 10                     HttpFramework.dispatch()
  5                       NumberUtil.convert()

The total time for executing the shopping cart update was approximately 413 ms. The number conversion (NumberUtil.convert()) took 5 ms. The time spent in between is distributed among many calls, so you are looking for bigger cliffs. In the example, updating the cart (ShoppingCart.updateCart()) took a total of 411 ms, while the parsing (ShoppingCart.parseQuantity()) only required 211 ms, which itself spent most of the time doing the remote call.

Trace continuity

Trace continuity means that calls triggered by one external request are collected into one trace. Instana employs protocol-specific means to add metadata, such as HTTP headers, gRPC metadata, Kafka message headers, AMQP headers, JMS headers, etc. This ensures trace continuity across all protocols and services.

Communication protocols without support for any metadata do not support trace continuity. This means that when calling another service over such a protocol, the outgoing call is a leaf in the trace tree. The work happening in the receiver of the call is not part of that trace. Instead, receiving the call starts a new trace and all subsequent calls triggered in the receiver belong to this new trace.

Trace continuity is not supported in the following cases:

• Kafka up to version 0.10 (Kafka introduced headers in version 0.11),
• sending or receiving Kafka messages with the Node.js package kafka-node (That package does not have support for headers. It is recommended to use the npm package kafkajs instead of kafka-node when working with Kafka in Node.js. With kafkajs, trace continuity is supported, note these additional remarks for continuing the trace for incoming messages).
• NATS and NATS streaming messaging
• Microsoft Message-Queue

W3C Trace Context Support

The following Instana tracers support the W3C trace context specification for HTTP/HTTPS communication in addition to the proprietary headers X-INSTANA-T/X-INSTANA-S:

• .NET Core
• .NET
• Apache HTTPd
• Clojure
• Go
• Haskell
• Java
• Kotlin
• Node.js
• Python
• Ruby
• Scala

The following Instana tracers currently do not support the W3C trace context specifictiation. Only the proprietary headers X-INSTANA-T/X-INSTANA-S are supported:

• Envoy
• NGINX
• PHP

Tracing headers

To ensure the trace continuity across different services, Instana tracers utilize different headers or metadata properties, depending on the protocol.