Best practices for custom tracing

If you want to extend Instana AutoTrace™ or ingest manually instrumented spans, we recommend you follow the following best practices.

Use well annotated spans

To best monitor, learn from, and alert on your applications and infrastructure, Instana performs advanced processing and analysis on all incoming data. This includes all incoming spans from all sources.

To get the most benefit out Instana's automatic analysis and processing, adding the appropriate tags to your spans allows Instana to best analyze and act upon incoming spans.

For example, the following Python OpenTracing code provides very little information.

Note: The following example uses OpenTracing to illustrate but the intention applies to all custom tracing code.

import opentracing

with opentracing.tracer.start_active_span('vanilla') as pscope:
  # ...
  # do something that takes 50ms
  # ...
  pscope.span.log_kv({"foo": "bar"})

From this code, we only know that it's a span named vanilla that took 50 ms of time. We don't know the type of span it was; HTTP, RPC, or a Messaging span. We don't know what happened during that time, such as communicating with any other component in your infrastructure.

If you provide the appropriate contextual OpenTracing tags, Instana analyzes and extracts information from that span to act on.

import opentracing
import opentracing.ext.tags as ext

with opentracing.tracer.start_active_span('webserver') as pscope:
    pscope.span.set_tag(ext.SPAN_KIND, "entry")
    pscope.span.set_tag(ext.PEER_HOSTNAME, "localhost")
    pscope.span.set_tag(ext.HTTP_URL, "/python/simple/two")
    pscope.span.set_tag(ext.HTTP_METHOD, "POST")
    pscope.span.log_kv({"foo": "bar"})

    # ...
    # work that took 50ms
    # ...

    pscope.span.set_tag(ext.HTTP_STATUS_CODE, 204)

This well annotated span tells Instana much more about what happened in the context of the span. From the tags provided, we know that this is an incoming webserver request to /python/simple/two and the resulting HTTP status code is 204.

A well annotated span, such as above, allows Instana to extract services, monitor connections and their health, which provides a richer experience in your dashboard.

For specifics on this example, see the OpenTracing specification which defines the authoritative list of all supported OpenTracing tags. A comprehensive list of tags supported by Instana (which is a superset of OpenTracing tags) is available at the end of this page.

Annotating "Built-In" Spans

In some cases, it may make sense to add extra metadata to an existing span provided by an Instana provided library. For this data to display, it needs to be inside of the sdk.custom.tags object in the trace.

Each library handles this differently. To understand how to add this object, check the documentation for your language's SDK.

Start new traces with entry spans

Within distributed tracing, there are three major types of spans:

  • Entry: Annotates the reception of a request, the start of a new task or consuming a message
  • Intermediate: Annotates internal work that doesn't make or receive calls (for example, view rendering)
  • Exit: Annotates work that makes a call to a remote service or publishes a message to a queue

The span.kind tag marks the type of span being reported. See span.kind in the table at the end of this page or alternatively in the OpenTracing specification for more details.

When starting new traces, start with an Entry span. If you don't specify the span.kind tag, the span is considered an Entry type span. Setting this tag to the appropriate value allows for better processing, visualization backend call extraction and mapping in Instana.

See the following Go example:

// Start a new entry span and set 'kind' to Consumer (entry)
entrySpan := ot.StartSpan("RPCJobRunner")
entrySpan.SetTag(string(ext.SpanKind), string(ext.SpanKindConsumerEnum))

// Now the RPC exit span
spanName := fmt.Sprintf("%s %s", request.Service, request.Method)
clientSpan := ot.StartSpan(spanName, ot.ChildOf(entrySpan.Context()))
clientSpan.SetTag(string(ext.SpanKind), string(ext.SpanKindRPCClientEnum))
clientSpan.SetTag("", request.Method)

// Make the RPC call


Marking a span with an error

Spans that represent work that contained an error, such as an exception, should include the error and message tags. For definitions of these tags, see the definition in the table as follows.

Passing context between microservices

To pass context across boundaries such as hosts, queues, and services, distributed tracing includes a method. This is usually done with carriers such as HTTP headers or message queue headers.

For more details, see our documentation on HTTP Headers.

Set a service name

This is optional. Service names are names applied to monitored application processes. In many languages, the best service name can be automatically detected based off of the Framework in use or the process command line.

If you want to override this, you can set a service name per Tracer. See the following example for the Go language.

serviceName := "default"
if *isServer {
    serviceName = "rpc-client"
} else {
    serviceName = "rpc-server"

opts := instana.Options{Service: serviceName})

All Instana OpenTracing Tracers also support this configuration via environment variable. If you set the environment variable INSTANA_SERVICE_NAME for the process, the value is used as the service name for the process. This overrides any code level service name configuration.

See also the documentation on Instana's overall service extraction techniques and rules.

Set an endpoint and a call name

This is optional. In many languages, endpoint and call names are automatically detected based on the service type.

If you want to override this, you can set endpoint and call names per Tracer. See the following example for the Java language.

SpanSupport.annotate(Span.Type.ENTRY, "my-custom-span", "endpoint", "endpoint-name");
SpanSupport.annotate(Span.Type.ENTRY, "my-custom-span", "", "call-name");

For more information about Instana's overall endpoint extraction techniques and rules, see our endpoints docs.

Overwrite the span duration

The Tracing SDK automatically computes the span duration from the start and end of the span.

It usually represents what you want to measure, however when it does not, you can overwrite it by using the duration tag.

Expected format is in milliseconds.

// Overwrite the span duration with 15 milliseconds.
SpanSupport.annotate(Span.Type.ENTRY, "my-custom-span", "duration", "15");

Path templates: visual grouping of HTTP endpoints

Instana supports automatic grouping of endpoints with path templates. This is supported out of the box with Instana tracing for many frameworks. With OpenTracing, an additional step is required as described as follows.

Various frameworks usually have a REST path pattern similar to the following:


These similar endpoints can be grouped by reporting a http.path_tpl key in your HTTP spans with the value /api/query/{birthdate}/{name}, Instana uses this template to automatically group endpoints that match the supplied pattern. In your Instana dashboard, the HTTP endpoints will then be grouped as single endpoint:


See the following for an example in Go:

span := ot.GlobalTracer().StartSpan("myTemplatedSpan", ext.RPCServerOption(incomingContext))
span.SetTag("http.path_tpl", "/api/query/{birthdate}/{name}")
span.SetTag(string(ext.SpanKind), string(ext.SpanKindRPCServerEnum))
span.SetTag(string(ext.HTTPUrl), req.URL.Path)
span.SetTag(string(ext.HTTPMethod), req.Method)
span.SetTag(string(ext.HTTPStatusCode), 200)

Note that this feature is Instana specific and not OpenTracing compliant.

Processed tags

This section lists the specific tags that Instana looks for when identifying and processing spans. When a subset of the tags as follows is found for a span, Instana can then better process, analyze, visualize, and alert on the incoming spans.

Note that the tags listed as follows are a superset of the standard OpenTracing tags. Each tag is marked on whether it is OpenTracing compliant with either a or x if it is not.

Some tag descriptions as follows are taken directly from the OpenTracing semantic conventions document.


In the distributed tracing world, there are three major categories of spans:

  1. Entry spans: spans that receive requests (such as HTTP or RPC servers), consumes messages from a queue or runs a job
  2. Exit spans: spans that make client requests (for example, HTTP, RPC, or database), push messages onto a queue or makes client database calls
  3. Intermediate spans: spans that represent work that is done in the applications such as view rendering or action/controller processing.

The span.kind tag for a span identifies the type of span that is being reported. Reporting this tag allows Instana to extract, process, and map out the communications between spans. At Instana, we refer to this communication as a "call".

Having such information allows Instana to not only monitor spans, but also monitor the calls between spans giving greater insight into the connectivity between your systems. With this in place, Instana can then also monitor and alert on the health of these calls as spans are being reported.

Tag Type OpenTracing Compliant? Description
span.kind String Either client or server for the appropriate roles in an RPC or HTTP request, and producer or consumer for the appropriate roles in a messaging scenario. Other values accepted are: entry, exit or intermediate which are not OpenTracing compliant.


HTTP spans represent an HTTP client call or HTTP Server request processing.

Tag Type OpenTracing Compliant? Description
http.url String The fully qualified HTTP URL used in this HTTP client request or server processing.
http.method String The HTTP method used in this HTTP client request or server processing. Examples are "GET", "POST", "PUT" and so on.
http.status_code Integer The HTTP status code of this HTTP request.
http.status Integer x Alternative to http.status_code. Both are supported although only one should be sent.
http.path String x The HTTP path of the request. String x The remote host if a client request or the host handling an incoming HTTP request.
http.params String x The query parameters of the HTTP request
http.error String x In the case of an error, an error message associated with this span. For example, "Internal Server Error"
http.header String x Used to report custom headers in relation so this span such as "X-My-Custom-Header=afd812cab"
http.path_tpl String x Allows for visual grouping of endpoints. See the Path Templates documentation for details

|http.route_id|String|x|A unique identifier for your route such as; useful with frameworks where a distinct endpoint is referenced by ID


  1. An appropriate span.kind should always be sent with these tags.
  2., http.path and http.params should only be sent in place of http.url. Sending a mix of these tags with http.url is not supported and the results are undefined.


RPC spans are those that represent work done in an RPC call or RPC server call processing.

Tag Type OpenTracing Compliant? Description String The RPC call being called or serviced (depends on value of span.kind) String The RPC remote host for client calls or the RPC host handling the request if an RPC server.
rpc.params String x Parameters for the RPC call
rpc.port String x Port for the RPC call
rpc.flavor String x Flavor of the RPC library in use such as XMLRPC, GRPCIO, and so on
rpc.error String x Error message associated with the RPC call

Note: An appropriate span.kind should be sent with these tags.


Tag Type OpenTracing Compliant? Description
graphql.operationType String x Query or Mutation
graphql.operationName String x Query or mutation name
graphql.fields JSON/object or stringified JSON (see as follows) x Fields used as part of the query or mutation
graphql.arguments JSON/object or stringified JSON (see as follows) x Arguments used as part of the query

Regarding graphql.fields and graphql.arguments you should choose the appropriate data type (JSON or String) depending on what the tracer supports.


// NodeJS SDK => JSON
span.annotate('sdk.custom.tags.graphql.fields', {
    Account: [ 'id', 'name' ],
    User: [ 'id', 'name' ]
span.annotate('sdk.custom.tags.graphql.arguments', {
    User: [ 'where', 'orderBy' ]

// Java SDK => String
SpanSupport.annotate("graphql.fields", "{ \"Account\": [\"id\", \"name\"], \"User\": [\"id\", \"name\"] }");
SpanSupport.annotate("graphql.arguments", "{ \"User\": ["\where\", \"orderBy\"] }");


Tag Type OpenTracing Compliant? Description
db.instance String The database instance name. As an example in Java, if the jdbc.url="jdbc:mysql://", the instance name is "customers"
db.type String For any SQL database, "sql". For others, the lowercase database category, for example, "cassandra", "hbase", or "redis".
db.statement String A database statement for the given database type. For example, when db.type is "SQL" - SELECT * FROM user_table; when db.type is "redis" - SET mykey 'WuValue'.
db.user String Username for accessing database. For example, "readonly_user" or "reporting_user"
db.connection_string String
Connection string, for example, jdbc:mysql://


Tag Type OpenTracing Compliant? Description
message_bus.destination String An address at which messages can be exchanged. Can be a topic or queue and so on.


Tag Type OpenTracing Compliant? Description
batch.job String x The name of the job being executed.


Tag Type OpenTracing Compliant? Description
peer.hostname String The remote host of an exit span making an outgoing call.
peer.address String The remote address of an exit span making an outgoing call.
peer.service String The service name of the remote side of an exit span making an outgoing call. Note that this is optional and may be used when the remote side is not already instrumented.


Tag Type OpenTracing Compliant? Description
error Boolean Indicates if an error was encountered during the time this span represents.
message String x A message associated with the error

See also