Monitoring agents (legacy experience)

Edit online

Deprecated on AWS

Supported on IBM Cloud

Note:

Deprecated (AWS only): The legacy Analytics experience is deprecated and scheduled for removal in a future AWS release. Use the new experience for continued access and updates.

Deprecation applies only to AWS. The legacy Analytics experience remains supported on IBM Cloud.

Use Agent analytics in watsonx Orchestrate to understand how your AI agents perform in real time. When you monitor your agents, you can quickly spot issues, analyze message behavior, and confirm whether recent changes improved or harmed performance.

From the Analyze menu, you can:

  • Track how many messages your agents process and how often they fail.

  • Monitor response latency to identify slowdowns early.

  • Compare agents to find which ones need attention.

  • Open detailed Traces to see exactly how an agent handled a message, step by step.

These insights help you troubleshoot problems faster, validate your configurations, and continuously optimize your agent's behavior.

Important:

  • Monitoring agent analytics is currently not supported in On-premises deployment.

  • Agent monitoring is not supported under the Premium Agentic with Data isolation plan.

  • Builders and administrators on IBM watsonx Orchestrate can view metrics and traces for both draft and live agents.

  • When full redaction is enabled, Live monitoring metrics are impacted, especially those related to answer quality and tool quality.

Before you begin

  • Before you start monitoring your agents, make sure that you have Builder or Admin access, and that you can open the Analyze menu in watsonx Orchestrate. From this menu, you can access the Agent analytics page.

  • If you are monitoring a live agent, ensure that monitoring is enabled for that agent. Monitoring must be turned on before you can see metrics, failure counts, latency, or trace-level details. If it is disabled, the analytics page does not show any activity for that agent.

Accessing agent analytics

To access Agent analytics, open the main menu and click Analyze. It takes you to the analytics page, where you can monitor message activity, track failures and latency, compare agents, and dive into detailed traces to troubleshoot or validate how an agent behaved.

The sections that follow explain each part of the analytics page and help you interpret the performance data you see.

Dashboard overview

When you open Agent analytics, the dashboard gives you a quick, high‑level snapshot of how your agents are performing. Use this view to understand overall activity, detect unusual patterns, and decide which agents need deeper investigation.

To switch to the new experience, click Try the new version.

Table 1. Dashboard overview

Details

Description

Total messages

See how many messages your agents processed during the selected period, including successful and failed responses.

Failed messages

Check how many messages resulted in errors (for example, timeouts or unsuccessful responses). A sudden spike in failures usually signals misconfigurations, broken tool connections, or unexpected model behavior.

Latency average

Review how long your agents take to respond on average. Higher latency may indicate slow model calls, inefficient tool operations, or workflow bottlenecks.

Monitor

Shows whether monitoring is enabled for the selected agent. You can toggle monitoring on or off for live agents. For more details, see Monitoring an agent.

Metrics update automatically as your agents run. If any value looks unusual, such as a sharp increase in failure rate or a jump in latency, you can immediately drill down to individual agent records or open message-level traces to understand what happened. The dashboard reflects data from agents that processed messages in both draft and live environments.

Note:

Agent analytics data remains available for 30 days.

Viewing agent metrics

Use the Agent analytics table to review each agent's performance at a glance. It helps you understand how actively an agent is used, how often it encounters errors, and how quickly it responds.

The table shows:

Table 2. Agent metrics

Details

Description

Name

The agent name.

Description

A short description of the agent's purpose.

Messages

The total number of processed messages.

Failed messages

The number of messages that resulted in errors.

Latency avg

The average message processing time.

You can use this information to compare message volumes across agents, identify agents with higher failure rates or slower responses, and confirm whether recent changes improved performance or introduced new issues.

Viewing agent traces

You can examine the agent's behavior in depth by opening its message-level traces. Start by clicking the agent's name in the Agent analytics table. This opens the Traces view, where you can examine how the agent handled each message step by step.

Agent analytics trace table

Each row in the trace table shows:

Table 3. Agent traces

Details

Description

Timestamp

The time the message was processed.

Trace ID

A unique identifier for the message.

Status

Whether the response was a "Success" or "Error".

Model

The large language model used to generate the response.

Latency

The total time the agent took to process the message.

Use the trace view to spot patterns, investigate failures, or validate whether an agent responded as expected. Traces help you identify problematic messages, compare latency across models or prompts, and confirm whether your agent behaves consistently after updates.

For a deeper breakdown of how the agent processed a message including model decisions, tool executions, knowledge retrieval, and workflow actions, you can open the Trace details view. The Understanding trace details page provides a structured, step‑by‑step view of each span in the execution path and helps you diagnose complex issues more effectively.

Note:

As per the data retention policy, trace data in Elasticsearch is retained for 30 days and automatically deleted thereafter.

Organize agents with workspaces on IBM Cloud

  • You can now create workspaces to organize your agents and collaborate on IBM Cloud.
  • Workspaces give your team a private space where you can build, test, and manage agents together while keeping in‑progress work visible only to the people you choose.
  • You can assign members as owners or editors with role‑based permissions, move agents between workspaces while preserving dependencies, decide who can view or modify specific agents, and structure your work by project, team, or business unit.
  • Workspaces also include analytics that are specific to the workspace you are using, giving you focused insight into the activity and performance of the agents in that space.
  • For more information about creating and managing workspaces, see Overview of workspaces.

Monitoring deployed agents on watsonx.governance

Monitoring enables you to track the real-time performance and health of your agent in real time. When you enable monitoring, IBM watsonx Orchestrate sends message activity and evaluation signals to IBM watsonx.governance. This integration adds an extra layer of visibility and safety by assessing responses, flagging issues, and helping you maintain compliant interactions.

Monitoring is supported in the following IBM Cloud regions:

  • Dallas (us-south), Texas, US

  • Frankfurt (eu-de), Germany

  • London (eu-gb), England

  • Tokyo (jp-tok), Japan

  • Sydney (au-syd), Australia

  • Toronto (ca-tor), Canada

Important:

  • This feature is not supported in on-premises and AWS deployments.

  • Only live agents can be monitored on watsonx.governance. Detailed monitoring of deployed agents is available only for IBM Cloud deployments.

  • For tenants where Langfuse is enabled, the monitoring experience is not available.

Key purposes and benefits of monitoring

Agent analytics helps you view key metrics such as total messages, failed messages, average latency, and trace details. Monitoring extends this capability by tracking live messages that are sent between IBM watsonx Orchestrate and watsonx.governance. Together, these insights help you:

  • Monitor agent performance and usage trends

  • Detect errors and anomalies early

  • Improve speed and reduce latency

  • Strengthen observability and troubleshooting

  • Provide a data-driven foundation for tuning and evolving agents

Prerequisites for watsonx.governance monitoring

Before you enable monitoring, ensure that all requirements are met:

  • The agent must be deployed and active. Only deployed agents appear on the watsonx Orchestrate dashboard.

  • You must have Builder or Admin permissions in watsonx Orchestrate.

  • You need an active watsonx.governance instance in the same IBM Cloud region as your IBM watsonx Orchestrate instance.

  • Ensure your instance is provisioned in a supported region.

  • Admins and builders must configure an access group with:

    • Writer and Editor permissions for watsonx Orchestrate

    • Reader and Editor permissions for watsonx.governance. See Setting up access groups.

  • Users of watsonx.governance must have a database that is configured:

Enabling agent monitoring

When you enable monitoring, the system begins tracking and collecting analytics on top of agent traces, which provides insights into performance and helps with optimization. You can turn it on from either the Agent analytics page or the Build agents and tools page.

Enable monitoring from the Agent analytics page

  1. Open the Agent analytics page.

  2. Find the agent you want to monitor.

  3. In the Monitor column, toggle monitoring on.

You can turn off monitoring later, for example, to pause data collection or reduce resource usage; you can return to the same Monitor column and toggle it off.

Agent analytics page

Enable monitoring from the Build agents and tools page

  1. Go to the agent configuration page.

  2. Click the overflow menu beside Deploy.

  3. Select Activate monitoring.

To deactivate monitoring from this page, open the overflow menu again and select Deactivate monitoring.

Key considerations

  • After you activate the monitoring feature, the activation might take a few minutes before the dashboard becomes available.

  • The system automatically polls monitoring activation every 30 seconds for up to 5 minutes.

  • Polling stops automatically when activation succeeds, fails, times out, or when you exit the page.

  • Verify that the monitoring toggle remains “on”, which confirms activation.

  • If the toggle is off, the API activation failed. Retry or open a support ticket.

  • If your watsonx.governance instance is deleted, the dashboard will no longer load. Create a new instance and re‑enable monitoring.

Viewing monitoring results

The dashboard provides a centralized view of your agent’s performance and evaluation metrics. The dashboard icon is visible only when monitoring is enabled for your agent. The dashboard provides:

  • Usage metrics (number of users and number of messages)

  • Performance metrics (token count, latency, and failed messages)

  • Technical metrics (faithfulness, tool call relevance, and others)

  • Conversation-level and message-level logs

Click the dashboard icon dashboard icon to view the monitoring details.

Note:

Metrics on new incoming traces can take a few minutes to appear in the watsonx.governance monitoring dashboard.

For more information, see Evaluating AI agents and agentic applications.

What to do next

To continue exploring how to analyze agent behavior, proceed to Understanding trace details. This next section provides a structured overview of how to access and review execution traces, enabling you to analyze the steps an agent performed for any processed message and identify where issues may have occurred.

Known issues

For known issues related to monitoring and analytics, see Known issues for monitoring and analytics.