Getting Started with Generative AI Observability

Instana® uses OpenLLMetry to observe LLM-based and agentic applications.

Add two lines of code to instrument your application with OpenLLMetry, then configure environment variables to point telemetry to Instana. These steps are all you need to begin monitoring your AI applications.

By instrumenting your applications with OpenLLMetry, Instana collects detailed data on LLM interactions, API calls, and performance metrics. This guide walks you through setting up observability for your Generative AI applications, from understanding the architecture to seeing your first traces in Instana.

Understanding telemetry collection

Like any observability tool, monitoring your AI applications requires collecting telemetry data: metrics, logs, and traces that capture application activity. You do this by instrumenting your code with OpenLLMetry, a specialized package built on top of OpenTelemetry that follows GenAI semantic conventions. These conventions ensure that telemetry data includes the right metadata for Instana to process and display AI‑specific insights like token usage, model performance, and costs.

Important:

Even if you already use OpenTelemetry in your application, you must also install OpenLLMetry. OpenLLMetry extends OpenTelemetry with GenAI‑specific semantic conventions that Instana requires to properly process and display AI observability data.

Architecture

The diagram shows how Instana monitors Generative AI applications.

Instana supports two modes for sending traces, logs, and metrics from your AI applications.

Agent mode

In agent mode, OpenLLMetry sends traces, logs, and metrics to the Instana agent, which forwards them to the Instana backend through the agent acceptor. Use this mode when the Instana agent is deployed and you want to correlate AI telemetry with infrastructure metrics.

Agentless mode

In agentless mode, OpenLLMetry sends traces, logs, and metrics directly to the Instana backend, bypassing the agent and ingesting the data through the OTLP acceptor. Use this mode for cloud‑native deployments, serverless environments, or when you prefer a simpler setup without agents.

Choosing between agent and agentless mode

Use the following table: determine which mode best fits your deployment scenario.

Table 1. Comparison of agent and agentless mode
Use agent mode when	Use agentless mode when
You already have Instana agents deployed in your environment	You are running in serverless environments (AWS Lambda, Azure Functions, and other services)
You want to correlate AI telemetry with infrastructure metrics (CPU, memory, network)	You are doing a proof‑of‑concept or development setup and want the quickest path to get started
You are running in traditional VM or container environments	You are in a cloud‑native deployment without agent infrastructure
You need the agent to enrich telemetry with infrastructure context	You want a simpler setup without deploying and managing agents
	You are running in ephemeral or short‑lived environments

Prerequisites

Before configuring your AI applications for monitoring, ensure that you meet the following prerequisites.

Python 3.10 or later
An Instana account with appropriate access
For agent mode, make sure the Instana agent is installed and running
For agentless mode, Instana backend OTLP endpoint and API key
API key for your LLM provider (OpenAI, Anthropic, and other providers)

Configuring Instana for AI observability

To set up Instana for AI observability, complete the following steps:

Set up your Python environment.
1. Make sure Python 3.10 or later is installed.
```
python3 -V
```
2. Optional: Create a virtual environment to keep your dependencies isolated.
```
pip3 install virtualenv
virtualenv genai-env
source genai-env/bin/activate
```
Install and configure OpenLLMetry in the code or service that makes calls to LLMs. It can be any of the following items:
- Your AI application's main entry point, for example, app.py, main.py.
- The API server or backend service that handles AI requests
- Any Python module that directly interacts with LLM providers (OpenAI, Anthropic, watsonx®, and other providers).
Install the OpenLLMetry SDK:
```
pip3 install traceloop-sdk
```
Note:
Add traceloop-sdk to your requirements.txt file for easier dependency management in your project.
Configure environment variables. Environment variables tell OpenLLMetry where to send your telemetry data and how to authenticate with Instana. Choose the configuration that matches your deployment mode.
Agent mode configuration

Use this configuration if you have the Instana agent installed and running in your environment.
```
# Identifies telemetry as GenAI data for Instana processing
export OTEL_RESOURCE_ATTRIBUTES="INSTANA_PLUGIN=genai"

# Enables log collection from your AI application
export TRACELOOP_LOGGING_ENABLED=true

# Enables metrics collection (token usage, latency, cost)
export TRACELOOP_METRICS_ENABLED=true

# Points to your Instana agent endpoint (check your agent config for exact host and port)
export TRACELOOP_BASE_URL=<instana-agent-host>:<port>

# Allows unencrypted communication with local agent
export OTEL_EXPORTER_OTLP_INSECURE=true
```
The Instana agent automatically enriches telemetry with infrastructure context. For more information about Instana agent setup, see Instana agent.

Agentless mode configuration

Use this configuration to send telemetry directly to the Instana backend without an agent. This is ideal for serverless environments, cloud‑native deployments, proof‑of‑concept setups, or development environments where you want the quickest setup.
```
# Identifies telemetry as GenAI data for Instana processing
export OTEL_RESOURCE_ATTRIBUTES="INSTANA_PLUGIN=genai"

# Enables log collection from your AI application
export TRACELOOP_LOGGING_ENABLED=true

# Enables metrics collection (token usage, latency, cost)
export TRACELOOP_METRICS_ENABLED=true

# Points directly to Instana backend OTLP endpoint (check your Instana config for exact endpoint and port)
export TRACELOOP_BASE_URL=<instana-otlp-endpoint>:<port>

# Authentication headers for backend access (get these from your Instana tenant)
export TRACELOOP_HEADERS="x-instana-key=<agent-key>,x-instana-host=<instana-host>"

# Enforces secure HTTPS communication with backend
export OTEL_EXPORTER_OTLP_INSECURE=false
```
To find the <instana-otlp-endpoint>, <agent-key>, and <port> values for agentless mode:
1. Log in to your Instana tenant.
2. In the navigation menu, click Agents and Collectors.
3. Click the OpenTelemetry collectors tab.
4. Click Install a collector.
5. Select Linux - Automatic Installation (One‑liner).
6. In Step 2, find the command with flags -a, -e, and -H:
  - The value after -a is your <agent-key>.
  - The value after -e is your <instana-otlp-endpoint>:<port> (for gRPC).
  - The value after -H is your <instana-otlp-endpoint>:<port> (for HTTP).
  - <instana-host> is a string that you can set to identify your application in Instana.
Note:
The port varies by deployment and protocol (gRPC or HTTP). Check the flags in your specific Instana configuration command for the correct port number.

For more information about Instana backend OTLP endpoints for different SaaS environments, see Instana backend.
Instrument your application. Add OpenLLMetry instrumentation to your Python application. Place this code at the beginning of your main application file (for example, app.py, main.py) or in the module that initializes your AI service:
```
from traceloop.sdk import Traceloop

# Initialize OpenLLMetry
Traceloop.init(app_name="my_ai_app", disable_batch=True)
```
Optional: For complex workflows you can annotate them to get better visibility:
```
from traceloop.sdk.decorators import workflow

@workflow(name="my_workflow")
def my_ai_workflow():
    # Your AI logic here
    pass
```
Note:
If you use frameworks like LangChain, LangGraph, or CrewAI, no additional annotations are needed. For more information about decorators, see Annotations.

Run a simple example. A minimal working example uses Anthropic's Claude:

Install the Anthropic SDK:
```
pip3 install anthropic
```
Set your Anthropic API key as an environment variable:
```
ANTHROPIC_API_KEY=<your-anthropic-api-key>
```
To create an API key, see Anthropic API keys.

Create a file named hello_genai.py with the following code:

import os
import anthropic
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow

# Initialize OpenLLMetry
Traceloop.init(app_name="hello_genai", disable_batch=True)

# Create Anthropic client
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

@workflow(name="simple_chat")
def ask_question():
    message = client.messages.create(
        model="claude-3-sonnet-20240229",
        max_tokens=1024,
        messages=[
            {"role": "user", "content": "What is observability?"}
        ]
    )
    print(message.content[0].text)

# Run the workflow
ask_question()

Run the application:
```
python3 hello_genai.py
```

Verify your setup

After running your application, verify that data is flowing into Instana:

From the navigation menu in the Instana UI, click Gen AI Observability.
Instana displays your application data, including:
- LLM API calls and traces
- Token usage metrics
- Request latency
- Cost
- Error rates

Note:

You can also view metrics, logs, and traces in other areas of Instana (such as the Analytics section) by filtering for your service name. The app_name parameter that you set in Traceloop.init() (for example, hello_genai) becomes the service name in Instana.

If you encounter issues when you set up Instana for AI observability, see Troubleshooting for troubleshooting problems such as connection errors, missing data, and configuration issues.

Optional: configure LLM pricing

To see cost metrics in your dashboard, configure pricing for your LLM models:

Go to the Generative AI observability dashboard.
Go to the Configuration tab.
Configure pricing for your LLM models.

For more information, see LLM cost calculation.

Note:

Other metrics (latency, tokens, errors) are visible even without pricing configuration.

Next steps

Now that you have basic monitoring set up, explore framework-specific and provider-specific guides:

AI agent frameworks: For more information on LangChain, LangGraph, CrewAI, OpenAI Agents, and similar frameworks, see Monitoring AI agent frameworks.

LLM providers: For more information on OpenAI, Anthropic, Amazon Bedrock, IBM watsonx.ai, and others, see Monitoring LLMs from model providers.
Vector databases: See Monitoring vector databases.
Infrastructure and hosting: For more information on GPU and vLLM monitoring, see Monitoring infrastructure and hosting.
If you are upgrading from an older setup, see Migrating from OTel Data Collector.