Running evaluations

Edit online

After you create test cases, run evaluations where an LLM-based user simulator plays the role of a real user so you can measure how your agent behaves in realistic interactions.

Evaluate your test cases

To evaluate all test cases, go to the Test agent page and click Evaluate all. To run a single test case, click the Options icon next to the test case and select Run test.

Monitor progress in the Evaluation results section. Each test case runs and results display in the Evaluation results table.

Understanding pass or fail criteria

A test case passes when all configured conditions are met:

  • Tool calls: All expected tools are called with correct parameters (based on match type configuration)

  • Keywords: All specified keywords appear in the agent's response

  • Semantic match: The response is semantically similar to the expected answer

A test case fails if any configured condition is not met. Refer Understanding evaluation metrics to identify which specific conditions failed.

Troubleshooting test failures

If a test fails unexpectedly:

  • Tool call mismatches: Check if parameters need Fuzzy matching instead of Exact. Use the View traces option to see actual tool calls made.

  • Keyword mismatches: Verify keywords are spelled correctly and appear in the expected response format.

  • Semantic mismatches: Review the response summary to ensure the expected behavior is clearly defined.

  • Configuration changes: If you modified tools or knowledge, rerun evaluations to test against the current agent configuration.

Note:
  • You cannot edit test cases while an evaluation runs. The system re-enables editing after the evaluation completes.
  • Evaluations can take up to 10 minutes, depending on the number of test cases and system load.

Reviewing evaluation results

You see each evaluation as a row in the Evaluation results section, which provides key details to help you track and analyze your test outcomes. You can search for specific evaluations using the search box.

Table 1. Evaluation results

Property

Description

Date

When the evaluation started

Success rate

Percentage of tests that passed

Successful tests

Number of tests that passed

Total tests

Total number of tests run

Run by

User who started the evaluation

Download

Export your evaluation report in CSV format

You can select an evaluation result, then click the Download button to export the report in CSV format or click the Delete button to remove it.

To view detailed results:

  1. Click the date link in the Evaluation results table.

  2. Review the comprehensive metrics dashboard showing overall performance and individual test results.

The detailed results page displays the agent name and description at the top, followed by overall metrics cards and individual test results. Each test result row includes an overflow menu with options to Run test or View traces.

Use View traces to see the exact tool calls, parameters, and responses generated during test execution. This helps you debug tool call mismatches.

What to do next