Tutorial: Instrumenting a Custom Scheduler with the Instana Tracing SDK

Example Code

The example code for the custom-scheduler-example tutorial can be found on github.com. It implements a simple scheduler, that calls a Hello World service on http://localhost:8080/ every 4 seconds and prints the response to the console.

Source Not Monitored By Instana

If you run the example, you will see that the HTTP calls are coming from an unmonitored source.

Why is the source not monitored?

The example code uses the Apache HTTP Client for the outgoing call. The Apache HTTP Client is supported by the Instana agent. So why do we not see the source of the request in Instana?

The reason is that Instana instruments outgoing calls only if they happen in an existing trace context. In the example, the outgoing call is triggered by the custom scheduler which is not instrumented. Therefore, the call is not instrumented because no existing trace context is found.

Enabling Tracing for the Custom Scheduler

In order to enable tracing for the custom scheduler, you first need to add the Java Trace SDK as a dependency. For Maven, it looks like this:

<dependency>
  <groupId>com.instana</groupId>
  <artifactId>instana-java-sdk</artifactId>
  <version>1.2.0</version>
</dependency>

In order to create a new trace context whenever the HttpClientJob is called, add a @Span annotation to the job's run() method:

@Span(type = Span.Type.ENTRY, value = "my-custom-scheduler")
public void run() {
  // ...
}

This annotation tells the Java agent to generate an ENTRY span whenever the run() method is called.

Third, add the following to the Instana agents configuration.yaml file to tell it that Java classes in package com.instana.sample.* should be scanned for the @Span annotation:

com.instana.plugin.javatrace:
  instrumentation:
    sdk:
      packages:
        - 'com.instana.sample'

Now, each call to HelloWorldJob.run() creates a trace context, and the outgoing HTTP request is performed within this trace context. You see this in Instana as follows:

The source of the HTTP GET request is now traced, including all metadata gathered from the automatic instrumentation of the Apache HTTP Client:

The scheduler job itself is visualized as an empty call with an unmonitored source:

Adding Tags

As shown above, the scheduler job is visualized as an empty call without any annotations. You can call the SDK's SpanSupport.annotate() method to add custom metadata. Just call SpanSupport.annotate() at the beginning of the HttpClientJob.run() method:

Override
@Span(type = Span.Type.ENTRY, value = SPAN_NAME)
public void run() {
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME, "tags.batch.job", "hello world job");
  // ...

In setting tags.batch.job for the ENTRY span of the job, as specified in OpenTracing's semantic conventions, Instana now treats the span as if it were of type BATCH and knows that there is no missing source.

You can also search for the custom tag in the Unbounded Analytics view:

SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME, "tags.error", "true");

Summary and Outlook

In this tutorial, we created a new trace context for our custom scheduler, thus enabling the batch job to be recognized as a source for outgoing HTTP calls in Instana.

For a full reference on our SDK, view the SDK's JavaDoc. For example, the @SpanParam and @SpanReturn annotations make it really easy to capture parameters or return values as additional information far spans, which might be more convenient than explicitly calling SpanSupport.annotate() in some scenarios.