Common Operations

How to initialize Go Collector?

To initialize Go Collector call instana.InitSensor() at the beginning of your main() function, providing an *instana.Options configuration object:

import instana "github.com/instana/go-sensor"

func main() {
    instana.InitSensor(&instana.Options{
        // ...
    })

    // ...
}

The Go Collector will use values of instana.DefaultOptions() for non-initialized fields of configuration object provided to the instana.InitSensor() function.

This would make your Go service appear in the Instana UI and report its runtime metrics. To see the trace information you'd need to instrument your app code first.

How to include Go Collector into a readiness probe?

To make sure that Go Collector has finished the announcement process and is ready to collect traces before sending the traffic to an instance, use the instana.Ready() method:

// Every 100ms whether Go Collector is ready and return on success
func awaitInstanaReady(ctx context.Context) error {
    ticker := time.NewTicker(100*time.Millisecond)
    defer ticker.Stop()

    for {
        select {
        case <-ticker.C:
            if instana.Ready() {
                return nil
            }
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    return nil
}

How to ensure that all trace data has been sent before shutdown?

To make sure that all collected traces have been sent to the host agent or serverless acceptor, the Go Collector API offers the instana.Flush() method. This method forces the agent client to send all buffered data immediately. It is a blocking operation, and it's recommended to define a timeout or a deadline for this operation, so your service can exit gracefully:

func main() {
    var srv http.Server

    // Initialize and use http.Server
    // ...

    // Server shutdown sequence:
    // 1. Finalize all pending requests and close the server
    shutdownCtx, _ := context.WithTimeout(ctx, 5 * time.Second)
    if err := srv.Shutdown(shutdownCtx); err != nil {
        // ...
    }
    // 2. Flush all buffered traces to the agent
    flushCtx, _ := context.WithTimeout(ctx, 5 * time.Second)
    if err := instana.Flush(flushCtx); err != nil {
        // ...
    }
    // ...
}

When a service is running on AWS Lambda instana.Flush() is called automatically by Go Collector once handler is done, so there is no need to do this manually.

instana.Flush() may interfere with Go Collector's data transmission cycle and it's not recommended to use it affect the trace data delivery during the service lifetime, e.g. upon finishing serving an HTTP request. There is no guarantee that Go Collector would stay operational after calling this method.

How to configure Go Collector to run in a serverless environment?

To use Go Collector for monitoring a service running in a serverless environment, such as AWS Fargate or Google Cloud Run, make sure that you have INSTANA_ENDPOINT_URL and INSTANA_AGENT_KEY environment variables set in your task definition. Please refer to the respective section of Instana documentation for detailed explanation on how to do this:

• Configuring AWS Fargate task definitions
• Configuring AWS Lambda functions
• Configuring Google Cloud Run services

The INSTANA_AGENT_HOST and INSTANA_AGENT_PORT environment variables will be ignored when an app is running in a serverless mode.

Services running in serverless environments don't use host agent to send metrics and trace data to Instana backend, therefore the usual way of configuring the in-app collector via configuration.yaml file is not applicable. Instead, there is a set of environment variables that can optionally be configured in service task definition:

Please refer to Instana documentation for more detailed description of these variables and their value format.

How to configure the Go Collector to redact sensitive data?

Certain instrumentation wrappers provided by Instana, e.g. the HTTP server and client wrappers, collect data that may contain sensitive information, such as passwords, keys and secrets. To avoid leaking these values the Go Collector replaces them with <redacted> before sending to the agent. The list of parameter name matchers is defined in com.instana.secrets section of the Host Agent Configuration file and will be sent to the in-app tracer during the announcement phase (requires agent Go trace plugin com.instana.sensor-golang-trace v1.3.0 and above).

The default setting for the secrets matcher is contains-ignore-case with following list of terms: key, password, secret. This would redact the value of a parameter which name contains any of these strings ignoring the case.

How to collect extra HTTP headers?

The HTTP instrumentation wrappers can collect HTTP headers, and send them along with the incoming or outgoing request spans. The list of case-insensitive header names can be provided both within (instana.Options).Tracer.CollectableHTTPHeaders field of the options object that is passed to instana.InitSensor() and in the Host Agent Configuration file. The latter setting takes precedence, and requires the agent Go trace plugin com.instana.sensor-golang-trace v1.3.0 or later versions:

instana.InitSensor(&instana.Options{
	// ...
	Tracer: instana.TracerOptions{
		// ...
		CollectableHTTPHeaders: []string{"x-request-id", "x-loadtest-id"},
	},
})

This configuration is an equivalent of following settings in the Host Agent Configuration file:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'

By default, the HTTP instrumentation does not collect any headers.

How to use a 3rd-party logger to print out Go Collector logs?

You can configure Go Collector to use a third-party logger conforming to the instana.LeveledLogger interface. For example, for github.com/sirupsen/logrus or go.uber.org/zap, use the [ instana.SetLogger()][instana.SetLogger] method:

import (
    "github.com/uber-go/zap"
    instana "github.com/instana/go-sensor"
)

func main() {
    logger, _ := zap.NewProduction()
    defer logger.Sync()

    instana.SetLogger(logger)
}

The INSTANA_DEBUG environment variable has no effect when using a 3rd-party logger. Please consult the documentation for your 3rd-party logger module to learn how to set the debug level via an environment variable or configuration file.

How to enable debug logs for Go Collector without changing the app code?

To enable the Go Collector writing debug logs without changing the app code, set the INSTANA_DEBUG environment variable to a non-empty value, for example:

export INSTANA_DEBUG=true
./your-app

The amount of debug logs that Go Collector produces depends on your application and can be significant. You are recommended to turn the debug logs on temporarily to gather information before you report an issue to Instana Support.

This setting works only if Go Collector uses logger.Logger to output internal log messages, and overrides any attempt to change the log level from within the app code.

How to collect logged errors and warnings associated with erroneous calls?

Instana Go Collector offers an instrumentation wrapper for github.com/sirupsen/logrus to automatically collect and send error and warning log records to the agent. github.com/instana/go-sensor/instrumentation/instalogrus provides an implementation of logrus.Hook that handles any error or warning records, and associates them with the span found in the record context.Context. Here is an example on how to instrument the global logrus.Logger, and use it inside of instrumented http.Handler:

func main() {
	// ...

	// Register instalogrus hook within the global logger
	logrus.AddHook(instalogrus.NewHook(sensor))
}

func myMethod(w http.ResponseWriter, req *http.Request) {
	// ...

    // When logging an error, make sure to provide the context.Context containing
    // an active span, so that instrumentation hook could associate this message
    // with a call.
    logrus.WithContext(req.Context()).Error("something went wrong")

    // ...
}

Refer to the package documentation for more detailed examples.

How to permanently enable continuous profiling from within the app code?

To permanently enable continuous profiling for your service provide EnableAutoProfile: true while initializing the sensor:

func main() {
    instana.InitSensor(&instana.Options{
		EnableAutoProfile: true,
		// ... other options
	})

	// ...
}

How to enable/disable continuous profiling temporarily from within the app code?

To temporarily turn AutoProfile™ on and off from your code, call autoprofile.Enable() and autoprofile.Disable():

func expensiveCalculation() {
    autoprofile.Enable()
    defer autoprofile.Disable()

    // ...
}

How to enable continuous profiling without changing the app code?

To enable AutoProfile™ for an app without code changes, set the INSTANA_AUTO_PROFILE=true environment variable:

export INSTANA_AUTO_PROFILE=true
./your-app

Similar to enabling debug logging with INSTANA_DEBUG that this value takes precedence and overrides any attempt to disable profiling from inside the application code.

How to instrument an HTTP server handler?

With support to wrap a http.HandlerFunc, Instana quickly adds the possibility to trace requests and collect child spans, executed in the context of the request span.

Minimal changes are required for Instana to be able to capture the necessary information. By simply wrapping the currently existing http.HandlerFunc Instana collects and injects necessary information automatically.

That said, a simple handler function like the following will simple be wrapped and registered like normal.

The following example code demonstrates how to instrument an HTTP handler using instana.TracingHandlerFunc():

sensor := instana.NewSensor("my-http-server")

http.HandleFunc("/", instana.TracingHandlerFunc(sensor, "/", func(w http.ResponseWriter, req *http.Request) {
	// ...
}))

In case your handler is implemented as a http.Handler, pass its ServeHTTP method instead:

h := http.FileServer(http.Dir("./"))
http.HandleFunc("/files", instana.TracingHandlerFunc(sensor, "index", h.ServeHTTP))

See the example/http-database-greeter for full example.

How to instrument an HTTP client?

Requesting data or information from other, often external systems, is commonly implemented through HTTP requests. To make sure traces contain all spans, especially over all the different systems, certain span information have to be injected into the HTTP request headers before sending it out. Instana's Go Collector provides support to automate this process as much as possible.

To have Instana inject information into the request headers, create the http.Client, wrap its Transport with instana.RoundTripper() and use it as in the following example.

req, err := http.NewRequest("GET", url, nil)
client := &http.Client{
	Transport: instana.RoundTripper(sensor, nil),
}

ctx := instana.ContextWithSpan(context.Background(), parentSpan)
resp, err := client.Do(req.WithContext(ctx))

The provided parentSpan is the incoming request from the request handler (see above) and provides the necessary tracing and span information to create a child span and inject it into the request.

How to instrument a database connection created with sql.Open()?

To instrument a database driver, register it using instana.InstrumentSQLDriver() first and replace the call to sql.Open() with instana.SQLOpen(). Here is an example on how to do this for github.com/lib/pq PostgreSQL driver:

// Create a new instana.Sensor instance
sensor := instana.NewSensor("my-database-app")

// Instrument the driver
instana.InstrumentSQLDriver(sensor, "postgres", &pq.Driver{})

// Create an instance of *sql.DB to use for database queries
db, err := instana.SQLOpen("postgres", "postgres://...")

The instrumented driver is registered with the name <original_name>_with_instana, e.g. in the example above the name would be postgres_with_instana.

See the example/http-database-greeter for full example.

How to instrument a database client created with sql.OpenDB()?

Starting from Go v1.10 database/sql provides a new way to initialize *sql.DB that does not require the use of global driver registry. If the database driver library provides a type that satisfies the database/sql/driver.Connector interface, it can be used to create a database connection.

To instrument a driver.Connector instance, wrap it using instana.WrapSQLConnector(). Here is an example on how this can be done for github.com/go-sql-driver/mysql/ MySQL driver:

// Create a new instana.Sensor instance
sensor := instana.NewSensor("my-database-app")

// Initialize a new connector
connector, err := mysql.NewConnector(cfg)
// ...

// Wrap the connector before passing it to sql.OpenDB()
db, err := sql.OpenDB(instana.WrapSQLConnector(sensor, "mysql://...", connector))

How to instrument a MongoDB client?

The github.com/instana/go-sensor/instrumentation/instamongo module provides constructor methods that create a new instance of go.mongodb.org/mongo-driver client and instrument it with Instana.

See the package documentation for detailed examples and usage instructions.

How to instrument a Redis client?

1. The github.com/instana/go-sensor/instrumentation/instaredis module offers function wrappers for go-redis that instrument an instance of redis.Client or redis.ClusterClient by adding hooks to the redis client.

   See the package documentation for detailed examples and usage instructions.

2. The github.com/instana/go-sensor/instrumentation/instaredigo module offers wrappers for redigo that instrument an instance of redis.Conn.

   See the package documentation for detailed examples and usage instructions.

How to instrument a gRPC server?

The github.com/instana/go-sensor/instrumentation/instagrpc module provides both stream and unary call interceptors. The module documentation provides detailed explanation on how to use them to instrument a gRPC server implemented with google.golang.org/grpc.

See the example/grpc-client-server for full example.

How to instrument a gRPC client?

The github.com/instana/go-sensor/instrumentation/instagrpc module provides both stream and unary call interceptors. The module documentation provides detailed explanation on how to use them to instrument a gRPC client implemented with google.golang.org/grpc.

See the example/grpc-client-server for full example.

How to instrument a Kafka producer/consumer?

The github.com/instana/go-sensor/instrumentation/instasarama module provides constructor methods that create a new instance of producer/consumer clients and instrument it with Instana.

See the example/kafka-producer-consumer for full example.

How to instrument a RabbitMQ producer/consumer?

The github.com/instana/go-sensor/instrumentation/instaamqp module offers a function wrapper around amqp.Channel that returns an instaamqp.AmqpChannel instance. This Instana object provides instrumentation for the amqp.Channel.Publish and amqp.Channel.Consume methods that are responsible for tracing data from messages sent and received.

See the full example at github.com/instana/go-sensor/blob/main/instrumentation/instaamqp.

How to instrument a method?

A minimal instrumentation of a method with Go Collector consists of two steps:

1. Start a new span with instana.Sensor.
2. Finalize the span before the method is returned.

Optional steps include collecting the details, relevant to this call to attach them to the active span, and injecting the active span into context.Context to ensure trace continuation.

func MyMethod(sensor *instana.Sensor) {
    // Start a new span associated with this method execution
    span := sensor.Tracer().StartSpan("my-method")
    // Schedule span finalization to send it to Instana agent
    defer span.Finish()

    // Optionally attach any relevant information that would help you to identify this call later
    span.SetTag("key1", "value1")
    span.SetTag("answer", 42)

    // ...
}

See the example/opentracing for more detailed example.

How to continue the trace inside of a subcall?

To associate a subcall with an active span, provide its context via the opentracing.ChildOf() start span option when starting the subcall span:

var spanOpts []opentracing.StartSpanOption

// Check whether there is an active trace by fetching the currently active span from context
if parent, ok := instana.SpanFromContext(ctx); ok {
    spanOpts = append(spanOpts, opentracing.ChildOf(parent.Context()))
}

// Start the subcall span
span := tracer.StartSpan("my-func", spanOpts...)

It's the caller's responsibility to ensure that an instance of context.Context provided to the callee contains the active span injected with instana.ContextWithSpan():

// And inject it into context, so any subcalls could use it as a parent
SubCall(instana.ContextWithSpan(ctx, span))

How to obtain the parent span inside of wrapped method?

The code wrappers that are provided by Instana utilize context.Context to inject the parent span into a subcall. That is, inside of an instrumented handler/callback function, you can assume that instana.SpanFromContext() returns the span that is started by the wrapper method:

parentSpan, ok := instana.SpanFromContext(ctx)

If a 3rd-party does not support passing context.Context to the handler method, the wrapper code may use alternative methods of injecting the active span context. Please consult the documentation for respective wrapper module to learn how to obtain the parent span context in this case.