Best practices for custom tracing

Use well annotated spans

To best monitor, learn from, and alert on your applications and infrastructure, Instana performs advanced processing and analysis on all incoming data. This includes all incoming spans from all sources.

To get the most benefit out Instana's automatic analysis and processing, adding the appropriate tags to your spans allows Instana to best analyze and act upon incoming spans.

For example, the following Python OpenTracing code provides very little information.

The following example uses OpenTracing to illustrate but the intention applies to all custom tracing code.

import opentracing

with opentracing.tracer.start_active_span('vanilla') as pscope:
  # ...
  # do something that takes 50ms
  # ...
  pscope.span.log_kv({"foo": "bar"})

From this code, we only know that it's a span named vanilla that took 50 ms of time. We don't know the type of span it was; HTTP, RPC, or a Messaging span. We don't know what happened during that time, such as communicating with any other component in your infrastructure.

If you provide the appropriate contextual OpenTracing tags, Instana analyzes and extracts information from that span to act on.

import opentracing
import opentracing.ext.tags as ext

with opentracing.tracer.start_active_span('webserver') as pscope:
    pscope.span.set_tag(ext.SPAN_KIND, "entry")
    pscope.span.set_tag(ext.PEER_HOSTNAME, "localhost")
    pscope.span.set_tag(ext.HTTP_URL, "/python/simple/two")
    pscope.span.set_tag(ext.HTTP_METHOD, "POST")
    pscope.span.log_kv({"foo": "bar"})

    # ...
    # work that took 50ms
    # ...

    pscope.span.set_tag(ext.HTTP_STATUS_CODE, 204)

This well annotated span tells Instana much more about what happened in the context of the span. From the tags provided, we know that this is an incoming webserver request to /python/simple/two and the resulting HTTP status code is 204.

A well annotated span, such as above, allows Instana to extract services, monitor connections and their health, which provides a richer experience in your dashboard.

For specifics on this example, see the OpenTracing specification which defines the authoritative list of all supported OpenTracing tags. A comprehensive list of tags supported by Instana (which is a superset of OpenTracing tags) is available at the end of this page.

Start new traces with entry spans

Within distributed tracing, there are three major types of spans:

• Entry: Annotates the reception of a request, the start of a new task or consuming a message
• Intermediate: Annotates internal work that doesn't make or receive calls (for example, view rendering)
• Exit: Annotates work that makes a call to a remote service or publishes a message to a queue

The span.kind tag marks the type of span being reported. See span.kind in the table at the end of this page or alternatively in the OpenTracing specification for more details.

When starting new traces, start with an Entry span. If you don't specify the span.kind tag, the span is considered an Entry type span. Setting this tag to the appropriate value allows for better processing, visualization backend call extraction and mapping in Instana.

See the following Go example:

// Start a new entry span and set 'kind' to Consumer (entry)
entrySpan := ot.StartSpan("RPCJobRunner")
entrySpan.SetTag(string(ext.SpanKind), string(ext.SpanKindConsumerEnum))

// Now the RPC exit span
spanName := fmt.Sprintf("%s %s", request.Service, request.Method)
clientSpan := ot.StartSpan(spanName, ot.ChildOf(entrySpan.Context()))
clientSpan.SetTag(string(ext.SpanKind), string(ext.SpanKindRPCClientEnum))
clientSpan.SetTag("rpc.call", request.Method)

// Make the RPC call

clientSpan.Finish()
entrySpan.Finish()

Marking a span with an error

Spans that represent work that contained an error, such as an exception, should include the error and message tags. For definitions of these tags, see the definition in the table as follows.

Passing context between microservices

To pass context across boundaries such as hosts, queues, and services, distributed tracing includes a method. This is usually done with carriers such as HTTP headers or message queue headers.

For more details, see our documentation on HTTP Headers.

Set a service name

This is optional. Service names are names applied to monitored application processes. In many languages, the best service name can be automatically detected based off of the Framework in use or the process command line.

If you want to override this, you can set a service name per Tracer. See the following example for the Go language.

serviceName := "default"
if *isServer {
    serviceName = "rpc-client"
} else {
    serviceName = "rpc-server"
}

opts := instana.Options{Service: serviceName})
opentracing.InitGlobalTracer(instana.NewTracerWithOptions(&opts)

All Instana OpenTracing Tracers also support this configuration via environment variable. If you set the environment variable INSTANA_SERVICE_NAME for the process, the value is used as the service name for the process. This overrides any code level service name configuration.

See also the documentation on Instana's overall service extraction techniques and rules.

Set an endpoint and a call name

This is optional. In many languages, endpoint and call names are automatically detected based on the service type.

If you want to override this, you can set endpoint and call names per Tracer. See the following example for the Java language.

SpanSupport.annotate(Span.Type.ENTRY, "my-custom-span", "endpoint", "endpoint-name");
SpanSupport.annotate(Span.Type.ENTRY, "my-custom-span", "call.name", "call-name");

For more information about Instana's overall endpoint extraction techniques and rules, see our endpoints docs.

Overwrite the span duration

The Tracing SDK automatically computes the span duration from the start and end of the span.

It usually represents what you want to measure, however when it does not, you can overwrite it by using the duration tag.

Expected format is in milliseconds.

// Overwrite the span duration with 15 milliseconds.
SpanSupport.annotate(Span.Type.ENTRY, "my-custom-span", "duration", "15");

Path templates: visual grouping of HTTP endpoints

Instana supports automatic grouping of endpoints with path templates. This is supported out of the box with Instana tracing for many frameworks. With OpenTracing, an additional step is required as described as follows.

Various frameworks usually have a REST path pattern similar to the following:

/api/query/1956-01-31/Guido-van-Rossum
/api/query/1976-04-18/Andrew-Ng
/api/query/1912-06-23/Alan-Turing

These similar endpoints can be grouped by reporting a http.path_tpl key in your HTTP spans with the value /api/query/{birthdate}/{name}, Instana uses this template to automatically group endpoints that match the supplied pattern. In your Instana dashboard, the HTTP endpoints will then be grouped as single endpoint:

/api/query/{birthdate}/{name}

See the following for an example in Go:

span := ot.GlobalTracer().StartSpan("myTemplatedSpan", ext.RPCServerOption(incomingContext))
span.SetTag("http.path_tpl", "/api/query/{birthdate}/{name}")
span.SetTag(string(ext.SpanKind), string(ext.SpanKindRPCServerEnum))
span.SetTag(string(ext.HTTPUrl), req.URL.Path)
span.SetTag(string(ext.HTTPMethod), req.Method)
span.SetTag(string(ext.HTTPStatusCode), 200)

Note that this feature is Instana specific and not OpenTracing compliant.

Processed tags

This section lists the specific tags that Instana looks for when identifying and processing spans. When a subset of the tags as follows is found for a span, Instana can then better process, analyze, visualize, and alert on the incoming spans.

Note that the tags listed as follows are a superset of the standard OpenTracing tags. Each tag is marked on whether it is OpenTracing compliant with either a ✓ or x if it is not.

Some tag descriptions as follows are taken directly from the OpenTracing semantic conventions document.

// NodeJS SDK => JSON
span.annotate('sdk.custom.tags.graphql.fields', {
    Account: [ 'id', 'name' ],
    User: [ 'id', 'name' ]
})
span.annotate('sdk.custom.tags.graphql.arguments', {
    User: [ 'where', 'orderBy' ]
})

// Java SDK => String
SpanSupport.annotate("graphql.fields", "{ \"Account\": [\"id\", \"name\"], \"User\": [\"id\", \"name\"] }");
SpanSupport.annotate("graphql.arguments", "{ \"User\": ["\where\", \"orderBy\"] }");

See also

• OpenTracing docs

• OpenCensus docs