Configuring metrics synchronization for watsonx Orchestrate agents

You can configure Governance console to pull in evaluation metrics from watsonx Orchestrate and display them in task views.
Applies to
watsonx.governance as a Service on AWS
Required services
  • watsonx Orchestrate on AWS
  • watsonx.governance as a Service on AWS
Required access
  • Admin level access in watsonx Orchestrate
  • Administrator access in Governance console
To configure metrics synchronization, do the following steps:
  1. Make sure that you have the most recent solution updates.

    To review updates to the solutions, download the watsonx.governance solution change log.

  2. Set up a connection to one or more watsonx Orchestrate instances. For more information, see Configuring connections to agent platforms.
  3. Configure views to display the metrics.
    • If you are using the system task view for the Agent object type (watsonx-governance-Task-AIComponent), the view is preconfigured to display the metrics from watsonx Orchestrate.
    • If you are using custom views for the Agent object type, update the custom task views to display the metrics. You can open the system task view for the Agent object type in View Designer to see the field names.

Users can now import watsonx Orchestrate agents and view the evaluation metrics for these agents in Governance console.

Note: If you're also using the automatic AI asset discovery feature, watsonx Orchestrate agents might be listed on the AI Asset Discovery tab as Shadow agents. For information about discovered assets, see Governing discovered shadow AI assets.

Prerequisites for synchronization

Synchronization occurs when the following conditions are met:
  • The agent in Governance console is linked to an agent in watsonx Orchestrate.
  • A sync interval is set for the agent in Governance console.
A link to a watsonx Orchestrate agent is created when users do one of the following tasks:
  • Import the watsonx Orchestrate agent to a use case.
  • Associate the watsonx Orchestrate agent with an agent object in Governance console and set the sync interval.
  • By using the Discovered Assets tab, create an agent object to govern the watsonx Orchestrate agent in Governance console. Users must open the agent task view and set the sync interval.

To import a watsonx Orchestrate agent, users need the Admin or Agent Builder role on the agent in watsonx Orchestrate.

Synchronization

The synchronization process uses the Agent Metrics Collection - IBM watsonx Orchestrate job in the Scheduler. You do not need to configure the job – users set the sync interval for agents when they add them to Governance console. Users can also edit the sync interval for an agent and run the sync on demand from the agent task view.

For agents that are in production, the metrics are synchronized on a scheduled basis, based on the interval that the user configures (daily, weekly, or monthly).

When a user imports an agent and sets the interval to Daily, an initial sync runs and the next sync time is updated at the end of the run. The next synchronization time is computed by providing each agent with a specific time on the next day based on a calculation. Sometimes the next synchronization might occur less than 24 hours after the first run. Users might see the current day in the Next sync field. This behavior is expected and is not an issue.

To view the scheduled jobs and logs, do the following steps:
  1. Click Settings icon > Solution Configuration > Scheduler.
  2. Click the filter icon and select the Metrics Collection category.
  3. Click the Agent Metrics Collection - IBM watsonx Orchestrate job.
  4. Click the Executions tab to see the job runs.
  5. To see the log and other information about a job run, click a job ID.

Metrics for draft and production agents

The synchronization process pulls in metrics for both draft and production agents.

In the task view of an agent, the build metrics section is visible for agents with any status but the production metrics section is visible only if the agent is in Validation status or higher.

The Import button and workflows

If you're using the predefined workflows and system views, the Related Assets section in the use case task view is visible only when the use case workflow stage is Risk Assessment. As a result, the Import button for agents is visible only in the Risk Assessment stage.

Users can see the current workflow stage of a use case in the Workflow Information panel of the use case task view.