AI evaluation in Instana

Measure the quality of your generative AI (gen AI) applications by running evaluations against criteria that you define.

Generative AI (gen AI) evaluations provide a way to measure the quality of your AI systems. Using criteria that you define, such as factual accuracy or relevance, a large language model (LLM) acts as an automated judge, scoring your AI's outputs. These scores help you identify problems, track improvements over time, and ensure your AI system is performing effectively.

The AI evaluation feature in Instana provides a framework for assessing and monitoring the quality of your gen AI applications at scale. By combining pre-built evaluators, custom evaluation criteria, filter-based trace selection, sampling controls, and drill-down capabilities, you can:

  • Automate quality monitoring: Define filter criteria once and continuously evaluate new traces as they arrive.
  • Control evaluation costs: Use sampling to balance statistical confidence with LLM-judge expenses.
  • Ensure consistent quality: Monitor production LLM outputs across services, models, and time periods.
  • Identify root causes: Drill down from aggregate scores to individual traces and judge outputs.
  • Make data-driven decisions: Use detailed evaluation results to optimize prompts, models, and configurations.

To get started with AI evaluation:

  1. In AI gateway, configure and enable a gateway for the Gen AI evaluation capability.
  2. Create evaluators to select metrics and specify the criteria for assessing them. Then, create evaluation definitions that include the traces you want to evaluate and which evaluators to use.
  3. Review your scores to see if any adjustments are needed for your evaluation definitions. Scores are generated by an AI model that evaluates your model's output against your pre-defined criteria and are advisory only.

What is AI evaluation?

AI evaluation is a systematic approach to assess the quality, accuracy, and reliability of outputs generated by LLM models. As organizations increasingly deploy LLM-powered applications, measuring and monitoring performance is critical to ensure they meet business requirements and quality standards.

AI evaluation typically involves:

  • Quality assessment: Measuring accuracy, relevance, and completeness of model responses
  • Consistency checking: Ensuring logical coherence and clarity in outputs
  • Performance monitoring: Tracking evaluation metrics to identify issues

How Instana enables AI evaluation

The Generative AI Observability platform in Instana provides an LLM evaluation framework. You can:

  • Define custom evaluators: Create evaluation criteria tailored to your specific use cases.
  • Leverage pre-built evaluators: Use industry-standard evaluation metrics out of the box.
  • Dynamic trace selection: Use filter-based criteria to automatically select trace cohorts for evaluation.
  • Monitor results: Track pass and fail rates, scores, and detailed evaluation outcomes with drill-down capabilities.
  • Generative AI evaluation: Use advanced AI models to evaluate your LLM outputs objectively.

The evaluation system integrates with the Instana trace collection. You can evaluate real production data, define trace cohorts using filters, control evaluation costs with sampling, and drill down from aggregate scores to individual traces and judge outputs.

Prerequisites

To use the GenAI evaluation feature, your environment must meet the following prerequisites.

Permissions

To configure and run evaluations, in Gen AI observability select Create, configure, delete, and run generative AI (gen AI) evaluations. When you select the checkbox, Access gen AI observability is enabled automatically.

For more information on the permissions, see Managing user access.

Feature flags

The GenAI evaluation feature uses the following feature flags:

  • feature.genai.analytics.enabled
  • feature.ai.gateway.enabled
  • feature.gen.ai.evaluation.enabled

The flag feature.ai.gateway.enabled is enabled by default. You must enable other feature flags to use AI evaluation capabilities.

To configure the feature flags on Standard Edition, see GenAI evaluation.

To configure the feature flags on Custom Edition, see .

AI gateway configuration

To use GenAI evaluation, you must configure an AI gateway for GenAI evaluation capability. You can configure gateway using your own credentials, select the models you want to use, and customize the model configuration. For more information, see AI gateway.

Creating an evaluator

Create the evaluation criteria that Instana uses to assess your LLM outputs against your predefined criteria. You can use sample evaluation metric templates as a starting point or define fully custom evaluators.

About evaluators

Instana provides sample evaluation metric templates for common quality dimensions. These templates are provided for illustrative purposes only — modify them to define criteria that match your specific requirements before using them in production:

  • Accuracy: Measures whether the output meets your defined criteria for correctness.
  • Relevance: Measures whether the output meets your defined criteria for relevance to the input query.
  • Completeness: Measures whether the output meets your defined criteria for fully addressing the query.
  • Consistency: Measures whether the output meets your defined criteria for logical coherence.
  • Clarity: Measures whether the output meets your defined criteria for clarity and understandability.
Note:
The sample criteria are provided for illustrative purposes only. You are solely responsible for defining your own criteria that match your specific requirements before using them in production.

For specialized needs, you can create a fully custom evaluator with your own criteria and a passing score threshold.

Procedure

  1. In the Instana UI, select GenAI Observability.
  2. Click the Evaluations tab.
  3. Click the Evaluators tab to view and create evaluators.
  4. To use a sample evaluation metric template, select it from the list and modify the criteria to match your requirements.
  5. To create a custom evaluator:
    1. Click Create an evaluator.
    2. Enter a value in the Name field.
    3. In the Evaluation criteria field, define the criteria for the evaluation. Sample criteria are for illustrative purposes only. You are solely responsible for defining your own criteria.
    4. In the Passing score threshold field, set a threshold for the score. A passing score is greater than or equal to the threshold specified. Scores are generated by an AI model that evaluates your model's output against your pre-defined criteria. Scores are approximate indicators and are advisory only.
    5. Save the evaluator.

Results

Your evaluator is now available for selection when you create an evaluation definition.

Creating an evaluation definition

Create an evaluation definition that specifies the scope of traces to evaluate, the evaluators to apply, and the evaluation details such as name and description.

About filter-based trace selection

Evaluation definitions use filter criteria to dynamically select trace cohorts. The system automatically selects matching traces when the evaluation runs, which enables:

  • Continuous monitoring: Define once, evaluate repeatedly as new traces arrive.
  • Cohort analysis: Evaluate specific user segments or model configurations.

Before you begin

Ensure that you have created at least 1 evaluator. See Creating an evaluator.

Procedure

  1. Click the Evaluations tab.
  2. Click Create evaluation definition.
  3. On the Define scope step, filter the traces you want to evaluate:
    • Service: Filter by service name.
    • Model: Filter by LLM model used.
    • Time window: Specify the time range for trace selection (for example, last 1 hour, last 24 hours, custom range).

    Optionally, set a sample percentage to limit the number of traces evaluated. Sampling is applied randomly across the filtered trace set, ensuring representative coverage while controlling evaluation costs.

  4. On the Select evaluators step, select one or more evaluators from the list. Each evaluator runs independently on the sampled traces.
  5. On the Define details step, enter a value in the Name field and, optionally, enter a value in the Description field.
  6. Save the evaluation definition.

Results

Your evaluation definition is saved and ready to run. Each run automatically selects traces that match your filter criteria and applies the configured sampling.

Running an evaluation and viewing results

Run a saved evaluation definition and use the Evaluations tab to monitor run status, review scores, and drill down to individual trace results.

Before you begin

Ensure that you have created an evaluation definition. See Creating an evaluation definition.

Procedure

  1. Click the Evaluations tab.
  2. Locate the evaluation definition you want to run.
  3. Click the run button in the Actions column. The evaluation process begins and runs in the background.
  4. Review the run results directly in the Evaluations tab.

Results

The Evaluations tab displays a summary for each completed or in-progress run. For each evaluator in the run, the following details are available:

  • Evaluator: Name of the evaluator.
  • Score (mean): Mean score across all evaluated traces.
  • Threshold: Minimum score required for a passing evaluation, shown as a value such as ≥ 0.50.
  • Pass rate: Percentage of traces that passed the threshold.
  • Duration: Time taken by this evaluator.
  • Traces scored: Number of traces evaluated out of the total sample.

Drilling down into results

To investigate individual trace results, click any evaluator row to view the list of sampled traces evaluated, along with individual trace scores, pass/fail status, and reasoning provided by the AI judge.

Drilling down enables the following:

  • Root cause analysis: Identify why specific traces failed.
  • Pattern detection: Find common issues across failed traces.
  • Quality improvement: Use judge feedback to refine prompts or models.
  • Validation: Verify that evaluators are working as intended.

Best practices

Follow these guidelines to design effective evaluations, control costs, and maintain high-quality gen AI applications.

Start with sample evaluation metric templates

The standard evaluation metrics (Accuracy, Relevance, Completeness, Consistency, Clarity) are provided as sample templates for illustrative purposes only. Modify them to match your specific criteria before using them in production. Use them as a starting point to understand the evaluation framework before creating custom evaluators tailored to your requirements.

Design effective filter criteria

  • Start broad, then narrow: Begin with service-level filters, then add model or time constraints.
  • Use meaningful cohorts: Define filters that represent real user segments or deployment scenarios.
  • Combine filters strategically: Use multiple filter criteria to isolate specific scenarios (for example, "production service and gpt-4 model and last 24 hours").

Optimize sampling for cost and confidence

  • High-volume scenarios: Use 10–25% sampling for services generating thousands of traces per day.
  • Medium-volume scenarios: Use 50% sampling for moderate trace volumes.
  • Low-volume or critical scenarios: Use 100% sampling when trace count is low or quality is mission-critical.
  • Monitor and adjust: Review evaluation costs and adjust sampling rates based on budget and confidence requirements.
  • Statistical validity: Ensure sample size is large enough for meaningful results.

Set appropriate thresholds

  • Start with moderate thresholds (for example, 0.5) and adjust based on results.
  • Different use cases might require different thresholds.
  • Monitor pass rates and adjust thresholds to match your quality standards.

Combine evaluators for comprehensive assessment

You can select multiple evaluators for a single evaluation to assess different quality dimensions simultaneously. Each evaluator runs independently and produces its own score and pass/fail result. For example:

  • Accuracy and Completeness for factual Q&A systems
  • Relevance and Clarity for customer support chatbots
  • All five sample evaluation metric templates (modified to match your criteria) for critical production systems

Use drill-down capabilities

  • Investigate failures: When pass rates drop, drill down to individual traces to identify root causes.
  • Validate evaluators: Review evaluation outputs to ensure evaluators are working as intended.
  • Iterate on prompts: Use evaluation feedback to refine your LLM prompts and improve quality.

Enable continuous monitoring

  • Define reusable evaluations: Create evaluation definitions with filters that automatically capture new traces.
  • Schedule regular runs: Run evaluations periodically to track quality trends over time.

Custom evaluator guidelines

When you create custom evaluators:

  • Write clear, specific evaluation criteria.
  • Follow the placeholder text in the evaluator description.
  • Test with sample data before production use.
  • Document the purpose and expected behavior.

Troubleshooting AI evaluation

Diagnose and resolve common issues with AI evaluation runs, trace selection, and drill-down navigation.

Evaluation run fails to start

  • Check AI gateway configuration: Ensure watsonx credentials are valid and the gateway is enabled.
  • Verify filter criteria: Ensure filters are properly configured and match existing traces.
  • Verify evaluator selection: Evaluations require at least one evaluator to be selected.
  • Check time window: Ensure the time window contains traces matching your filter criteria.

No traces match filter criteria

  • Broaden filters: Remove some filter constraints to capture more traces.
  • Verify service names: Ensure service names match exactly (case-sensitive).
  • Check time window: Expand the time range to include more traces.

Low pass rates

  • Review threshold settings: Threshold might be too high for your use case.
  • Drill down to failed traces: Click through to individual traces to understand failure reasons.
  • Examine evaluation outputs: Review the reasoning provided by the AI evaluation model.
  • Validate evaluation criteria: Ensure criteria match your actual requirements.
  • Check for data quality issues: Failed traces might indicate problems with input prompts or model responses.

Slow evaluation runs

  • Reduce the sampling percentage: A lower sampling percentage reduces the number of traces evaluated.
  • Reduce evaluator count: Use fewer evaluators to speed up the evaluation process.
  • Narrow time window: Evaluate shorter time periods to reduce trace volume.
  • Optimize filters: Use more specific filters to reduce the matching trace set.

High evaluation costs

  • Reduce sampling percentage: Lower the sampling rate to evaluate fewer traces.
  • Narrow filter criteria: Use more specific filters to reduce the trace cohort size.
  • Reduce evaluator count: Run fewer evaluators per evaluation.
  • Optimize time windows: Evaluate shorter time periods or less frequently.

Missing results or incomplete drill-down

  • Wait for completion: Evaluations run asynchronously; check status before you drill down.
  • Review error messages: Check the run status for any errors.
  • Verify trace persistence: Ensure traces are still available in the system (not aged out).
  • Check permissions: Ensure you have access to view trace details.

Disclaimer

Note:
Scores are generated by an AI model that evaluates your model's output against your pre-defined criteria. Scores are approximate indicators and are advisory only. Sample criteria are for illustrative purposes only. You are solely responsible for defining your own criteria to ensure scores reflect your specific requirements.