Monitoring Generative AI applications

Observing LLM data

LLM observability is the practice of collecting and analyzing data to assess an LLM’s performance and behavior. This data helps to improve the LLM’s performance, detect biases, diagnose issues, and make sure reliable AI outcomes.

There are three main types of LLM observability data to monitor:

Metrics: Metrics are quantitative measures of the LLM’s performance, such as accuracy, latency, and throughput. Metrics can enable the customer to do some evaluation for different LLMs so as to select the best LLM for their use case.

Traces: Traces track the execution of individual LLM tasks. Traces help to identify performance bottlenecks and to diagnose issues.

Logs: Logs provide detailed information about the LLM’s input and output, such as the prompts that are given to the LLM and the responses that it generates.

Key benefits of LLM observability

Monitors model latency: Tracks how fast the AI responds to inputs to make sure efficient interactions.​

Tracks input/output token count: Measures data size and complexity to optimize resource use and performance.​

Identify bottlenecks: Detects delays or inefficiencies in processing.​

Tracks cost efficiency: Observes resource consumption to manage and reduce operational costs.​

Ensures scalability: Monitors performance under increasing workloads to make sure that the AI can handle growing demand.​

Relevant topics

For more information about Instana Observability and detailed insights into application and model usage, see the following topics:

• Architecture
• Getting started
  • Instana setup
  • Install OTel Data Collector for LLM (ODCL)
  • Traceloop instrumentation
• OpenTelemetry signals
  • Metrics
  • Traces
  • Logs
• AI model platforms
  • IBM WatsonX
  • OpenAI
  • Amazon Bedrock
  • Groq
  • Anthropic
• AI host platforms
  • vLLM (public preview)
  • GPU (public preview)
• AI Agent frameworks
  • LangChain
  • LangGraph
  • CrewAI