Analyzing traces and calls: understanding the time spent within a specific call

Answer

Before going into details on this article topic it might be useful to provide some general background information on how tracing works in Instana. 

A trace represents a single request and its path through a system of services and a call represents communication between two services.
Each call is a set of data and time measurements and to capture this data  in distributed tracing both the caller and the callee side are measured for the same call, and these individual measurements are called spans. Spans represent the timing of code executions.

A Trace is comprised of one or more calls and each call is comprised of one or more spans:

• The root call in a trace has an entry span - this defines the entry to the call from an un-instrumented service/process.
• The outbound call out of a trace has an exit span - the exit of the call to an un-instrumented service/process.
• The intermediate calls have both an Exit Span (the exit from the caller) and an Entry Span (the entry into the call itself aka the callee) since both the caller and callee are instrumented. 

For intermediate calls we can calculate so-called Network Time as the Exit Span time minus the Entry Span time.

What Instana calls 'Network Time' is not necessarily network time - it is any time that was consumed between the two calls (both during the request and response). It could very well be network time, but it could be container network or serialization, e.g. overloaded network, following redirects, queueing, overloaded JVM.

Here's the summary of how the times are defined:

• Total Time: the total (clock) time of a call
• Wait Time:  the amount of time the call spends waiting on all downstream calls to complete = Sum of all downstream calls total times
• Self Time: the amount of time the call spends outside of downstream calls
• Network Time: The difference between the caller’s Exit Span Time and the call’s Entry Span Time