Adding custom AI models

Edit online

You can integrate AI models from various providers into watsonx Orchestrate to power your agents and other AI-driven features. Configure models as virtual models by using the Agent Development Kit (ADK), which registers them for use within your IBM watsonx Orchestrate tenant.

Prerequisites

Before you configure virtual models, ensure you have:

  • An active watsonx Orchestrate tenant with appropriate permissions

  • An account with at least one of the supported AI model providers

  • Valid API credentials (API keys, access tokens, or service credentials) from your chosen provider

  • Network access to the provider's API endpoints

Optimized models

watsonx Orchestrate provides optimized support for the following model and provider combinations:

  • gpt-oss-120b on AWS Bedrock

  • gpt-oss-120b on Groq

  • AWS GovCloud only: gpt-oss-120b on watsonx.ai

These models are evaluated and optimized for use with watsonx Orchestrate. They provide reliable performance for agent use cases and are the recommended choice for production deployments.

Virtual models

A virtual model is an external AI model from a supported provider that you register with watsonx Orchestrate. Virtual models extend the platform capabilities by enabling you to use models beyond the optimized options.

Review the following considerations before you register and use a virtual model:

Model and provider compatibility
Not every model and provider combination is supported. A list of tested providers is available in the ADK documentation, but not every model from those providers is tested or verified to work. New models are released frequently, and some introduce API changes that can cause runtime errors. For more information, see Managing virtual models: Supported providers.
Intermediate infrastructure
Components in the request path between watsonx Orchestrate and the provider such as a proxy, gateway, or adapter layer can introduce incompatibilities. These components can affect authentication, authorization, or API signatures in ways that watsonx Orchestrate does not support.
Model suitability for agents
Models vary in their suitability for agent use cases. A virtual model that registers successfully and runs without errors might still produce results that are not accurate or reliable enough for business-critical agents.
Testing requirements
Test your chosen model and provider combination to verify it works correctly for your use case. Testing can identify incompatibilities that prevent your agent from functioning as expected.
Important:

Third-party models operate under their own license terms and might impose use restrictions or other obligations. Review the applicable license terms before you use any external model. Read the terms.

Custom provider endpoints

You can register self-hosted AI models or models accessed through a proxy as virtual models if they provide OpenAI chat completion API compatibility.

For configuration instructions and examples, see Managing custom LLMs.

Configuring virtual models

To register and manage virtual models, use the Agent Development Kit (ADK). The ADK provides commands to add, list, and remove virtual models from your watsonx Orchestrate tenant.

For complete instructions, see the following topics:

Using AI models in agents

After you configure a virtual model, you can select it when you create or edit an agent:

  1. In watsonx Orchestrate, create or edit an agent.

  2. In the AI model field, select your configured model from the list.

  3. Save your agent configuration.

The selected model powers the agent's natural language understanding and response generation capabilities.