Tracing overview

Edit online

Tracing tracks requests as they flow through distributed systems, helping you identify performance bottlenecks, understand dependencies, and troubleshoot issues across service boundaries.

What is tracing

Tracing is the process of tracking a request as it moves through services and components in a distributed system. Each trace consists of spans that represent individual operations, creating a complete picture of the request's journey and timing.

Key features

The tracing framework provides the key capabilities such as:

External storage configuration
Configure external storage for the Tempo stack to store trace data. This provides flexibility to use your existing storage infrastructure and manage costs effectively.
AI-powered application observability instrumentation
Configure your AI-powered custom agents and applications using specialized frameworks that capture AI-specific metrics and traces. This helps you monitor AI-powered application performance, token usage, and response quality.

Trace storage

The tracing system uses Tempo for trace storage and management. You can configure external storage instead of using the default storage to integrate with your existing infrastructure. Tempo supports S3-compatible object storage systems for scalable and cost-effective trace retention.

AI-powered application observability

For applications using large language models (LLMs), specialized instrumentation frameworks provide enhanced observability. Some commonly used instrumentation which you can integrate for AI-powered application Observability include:

Langfuse
Provides detailed instrumentation for AI-powered applications, capturing prompts, completions, token usage, and performance metrics.
OpenLit
Offers comprehensive AI-powered application observability with support for AI frameworks and models, including cost tracking and quality metrics.
Traceloop (Gemini)
Specializes in tracing for Gemini and other Google AI models, providing detailed insights into model interactions and performance.

Key features

Implementing distributed tracing includes the benefits:

  • Identify performance bottlenecks across service boundaries
  • Understand service dependencies and call patterns
  • Troubleshoot issues in distributed systems more effectively
  • Monitor AI-powered application performance and costs
  • Optimize request flows and reduce latency
  • Gain visibility into complex microservices architectures

Integration with OpenTelemetry

The tracing system integrates with OpenTelemetry standards, ensuring compatibility with a wide range of instrumentation libraries and tools. This allows you to instrument applications in different languages and frameworks while maintaining consistent trace data collection and analysis.