AI Observability architecture

The architecture provides gen AI observability by integrating Instana, Traceloop, and OpenTelemetry (OTel DC) to monitor a gen AI application. The Traceloop instrumentation captures traces, logs, and metrics from the application, ensuring visibility into AI interactions.

The following architecture diagram illustrates how Instana monitors gen AI applications:

Instana supports two modes for sending traces, logs, and metrics:

  1. Agent mode
  2. Agentless mode

Agent mode (through Instana agent)

In agent mode, Traceloop sends traces and logs to the Instana agent first, which forwards them to the Instana backend through the agent acceptor. The Instana agent also collects metrics from LLM OTel DC, providing additional insights into AI model usage, token consumption, and costs.

Agentless mode (through Instana backend)

In agentless mode, Traceloop sends traces and logs directly to the Instana backend, bypassing the Instana agent. Also, LLM OTel DC collects metrics from the gen AI application through instrumentation and injects them into the OTLP acceptor in the Instana backend. This mode relies on the OTLP acceptor to ingest and process telemetry data, including traces, logs, and metrics collected from Traceloop's instrumentation.

The Instana UI visualizes the collected data, offering real-time insights into AI performance, cost analytics, and request trends. This comprehensive monitoring setup helps you to optimize AI resource utilization, detect anomalies, troubleshoot issues efficiently, enhancing AI system reliability, and performance.