Tutorial: Instrumenting a Custom Java HTTP Framework with the Instana Tracing SDK

The example code for the custom-http-sample tutorial can be found on github.com. It implements the following scenario:

If you run the example and open http://localhost:8080, you will get the response Hello, Stan!. However, if you analyze the calls in Instana, you will see that the trace is broken.

Trace 1:

Trace 2:

The outgoing requests are implemented using the Apache HTTP client, which is supported by Instana's automatic tracing. The missing link is the entry span (incoming call) GET /greeting at the custom HTTP server.

For supported frameworks, the Instana agent instruments Java applications automatically. For the custom HTTP server, we need to add some annotations to tell the Instana agent which method calls should start a new span.

First, add mvn dependency

<dependency>
  <groupId>com.instana</groupId>
  <artifactId>instana-java-sdk</artifactId>
  <version>1.2.0</version>
</dependency>

Second, add a @Span annotation to the method handling the HTTP request in the custom HTTP server.

@Span(type = Span.Type.ENTRY, value = "my-custom-http-server")
public String apply(Map<String, String> headers) {
  // ...
}

This annotation tells the Java agent to generate an ENTRY span (aka. server span) whenever this method is called.

Third, add the following to the Instana agent's configuration.yaml file to tell it that Java classes in package com.instana.sample.* should be scanned for the @Span annotation:

com.instana.plugin.javatrace:
  instrumentation:
    sdk:
      packages:
        - 'com.instana.sample'

Now, the custom server gets instrumented, and you see that the outgoing call to /name is triggered from the incoming call to /greeting.

However, the trace is still broken, because the trace ID is not passed on between the incoming call and the outgoing call.

Instana uses the following HTTP headers to pass tracing information from one service to the next:

• X-INSTANA-T: Trace ID
• X-INSTANA-S: Parent span ID
• X-INSTANA-L: Level 0 means the trace should be suppressed, which might be useful for health checks

In order to correlate the trace context in the customer HTTP server, you need to add the following method to GreetingHandler:

private void correlateTracing(Map<String, String> request) {
  String level = request.remove(SpanSupport.LEVEL);
  String traceId = request.remove(SpanSupport.TRACE_ID);
  String parentSpanId = request.remove(SpanSupport.SPAN_ID);

  if (SpanSupport.SUPPRESS.equals(level)) {
    SpanSupport.suppressNext();
  } else if (traceId != null && parentSpanId != null) {
    SpanSupport.inheritNext(traceId, parentSpanId);
  }
}

Note that we remove the headers after evaluating them, leaving only the original headers as they would be sent without Instana tracing.

However, you cannot just call correlateTracing() in apply(), because the headers need to be evaluated before the method with the @Span annotation is called. As a workaround, you could rename the apply() method to doApply() and add a step as follows:

@Override
public String apply(Map<String, String> headers) {
  correlateTracing(headers);
  return doApply(headers);
}


@Span(type = Span.Type.ENTRY, value = "my-custom-http-server")
private String doApply(Map<String, String> headers) {
  // ...
}

When you now run curl http://localhost:8080, you will see the complete Trace in Instana

So far, our my-custom-http-server span is just a generic ENTRY span without any additional annotations. If you look at the details, they look quite empty.

You can call the SDK's SpanSupport.annotate() method within an active @Span context to add annotations to the current span. Instana supports part of OpenTracing's semantic conventions as described in the Java Trace SDK documentation. In order to add HTTP annotations, add the following lines at the start of the doApply() method:

@Span(type = Span.Type.ENTRY, value = SPAN_NAME)
private String doApply(Map<String, String> headers) {
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.http.url", "http://localhost:8080/greeting");
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.http.method", "GET");
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.http.status_code", "200");
  // ...
}

Now the details of the ENTRY span for the custom HTTP server show the specified HTTP metadata.

Note that the corresponding EXIT span from Jetty to the custom HTTP server had HTTP metadata already, because the client uses the Apache HTTP client which is supported by Instana's auto tracing.

SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.error", "true");