Getting started

Architecture

How Instana monitors Generative AI applications is shown in the diagram.

Instana supports two modes for sending traces, logs, and metrics from your AI applications.

Agent mode

In agent mode, OpenLLMetry sends traces, logs, and metrics to the Instana agent first, which forwards them to the Instana backend through the agent acceptor. This mode is ideal when you have the Instana agent deployed and want to correlate AI telemetry with infrastructure metrics.

Agentless mode

In agentless mode, OpenLLMetry sends traces, logs, and metrics directly to the Instana backend, bypassing the Instana agent. The telemetry data is ingested through the OTLP acceptor in the Instana backend. This mode is ideal for cloud-native deployments, serverless environments, or when you want a simpler setup without deploying agents.

Prerequisites

Before you configure your AI applications for monitoring, make sure that the following prerequisites are met.

• Python 3.10 or later
• An Instana account with appropriate access
• For agent mode, Instana agent is installed and running
• For agentless mode, Instana backend OTLP endpoint and API key
• API key for your LLM provider (OpenAI, Anthropic, and so on)

Setting up Instana for AI observability

To set up Instana for AI observability, complete the following steps:

1. Set up your Python environment.
   1. Verify that Python 3.10 or later is installed:

      python3 -V

   2. Optional: Create a virtual environment to keep your dependencies isolated.

      pip3 install virtualenv
      virtualenv genai-env
      source genai-env/bin/activate

2. Install and configure OpenLLMetry in the code or service that makes calls to LLMs. It can be any of the following items:

   • Your AI application's main entry point, for example, app.py,main.py.
   • The API server or backend service that handles AI requests
   • Any Python module that directly interacts with LLM providers (OpenAI, Anthropic, watsonx, and so on.
   Install the OpenLLMetry SDK:

   pip3 install traceloop-sdk

   Note: Add traceloop-sdk to your requirements.txt file for easier dependency management in your project.
3. Set the following environment variables that are required for all monitoring modes.

   OTEL_RESOURCE_ATTRIBUTES="INSTANA_PLUGIN=genai"
   TRACELOOP_LOGGING_ENABLED=true
   TRACELOOP_METRICS_ENABLED=true

   Configure the mode-specific variables based on your deployment.

4. Complete one of the following steps:
   • Agent mode: Set these environment variables to send telemetry through the Instana agent:

     TRACELOOP_BASE_URL=<instana-agent-host>:4317
     OTEL_EXPORTER_OTLP_INSECURE=true

     The Instana agent automatically enriches telemetry with infrastructure context. For more information about Instana agent setup, see Instana agent.

   • Agentless mode: Set these environment variables to send telemetry directly to the Instana backend:

     TRACELOOP_BASE_URL=<instana-otlp-endpoint>:4317
     TRACELOOP_HEADERS="x-instana-key=<agent-key>,x-instana-host=<instana-host>"
     OTEL_EXPORTER_OTLP_INSECURE=false

     Find your Instana configuraion values. To find the <instana-otlp-endpoint> and <agent-key> values.

     1. Log in to your Instana tenant.
     2. In the navigation menu, click Agents and Collectors.
     3. Click the OpenTelemetry collectors tab.
     4. Click Install a collector.
     5. Select Linux - Automatic Installation (One-liner).
     6. In Step 2, you'll see a command similar to the following example:

        curl -Lo setup.sh https://github.com/instana/instana-otel-collector/releases/latest/download/instana-collector-installer-latest.sh && chmod +x setup.sh && sudo ./setup.sh -a <agent-key> -e <instana-otlp-endpoint>:4317 -H <instana-otlp-endpoint>:4318
         

     7. <instana-otlp-endpoint> is the endpoint shown before :4317 (gRPC port) and :4318 (HTTP port)
     8. <agent-key> is the key shown after -a
     9. <instana-host> can be any string of your choosing

   For more information about Instana backend OTLP endpoints for different SaaS environments, see Endpoints of the Instana backend OTLP acceptor.

5. Instrument your application. Add OpenLLMetry instrumentation to your Python application. Place this code at the beginning of your main application file (for example, app.py, main.py) or in the module that initializes your AI service:

   from traceloop.sdk import Traceloop
   
   # Initialize OpenLLMetry
   Traceloop.init(app_name="my_ai_app", disable_batch=True)
    

   Optional: For complex workflowsyou can annotate them to get better visibility:

   from traceloop.sdk.decorators import workflow
   
   @workflow(name="my_workflow")
   def my_ai_workflow():
       # Your AI logic here
       pass
    

   Note: If you use frameworks like LangChain, LangGraph, or CrewAI, no additional annotations are needed. For more information about decorators, see Annotations.
6. Run a simple example. The following example uses Anthropic's Claude:

   1. Install the Anthropic SDK:

      pip3 install anthropic
       

   2. Set your Anthropic API key as an environment variable:

      ANTHROPIC_API_KEY=<your-anthropic-api-key>
       

      To create an API key, see Anthropic API keys.

   3. Create a file named hello_genai.py with the following code:

      import os
      import anthropic
      from traceloop.sdk import Traceloop
      from traceloop.sdk.decorators import workflow
      
      # Initialize OpenLLMetry
      Traceloop.init(app_name="hello_genai", disable_batch=True)
      
      # Create Anthropic client
      client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
      
      @workflow(name="simple_chat")
      def ask_question():
          message = client.messages.create(
              model="claude-3-sonnet-20240229",
              max_tokens=1024,
              messages=[
                  {"role": "user", "content": "What is observability?"}
              ]
          )
          print(message.content[0].text)
      
      # Run the workflow
      ask_question()
       

   4. Run the application:

      python3 hello_genai.py
       

Verify your setup

After running your application, verify that data is flowing into Instana:

1. In the Instana UI, click Gen AI Observability.
2. The application data is displayed that includes the following items:
   • LLM API calls and traces
   • Token usage metrics
   • Request latency
   • Cost
   • Error rates

If you encounter issues when you set up Instana for AI observability, see Troubleshooting for troubleshooting problems such as connection errors, missing data, and configuration issues.