Registering external models through AI gateway

Registering external models in watsonx Orchestrate enables the integration of custom models into orchestration pipelines, allowing the domain-specific inference and decision logic.

Before you begin

Note: This page is required only if you have not configured the models using Inference Foundation Models (IFM).
  • Register your external models against each new service instance of watsonx Orchestrate as the LLM registration through AI gateway is specific to a service instance.
  • Register atleast one LLM as the default model in each service-instance of watsonx Orchestrate. To mark a model as the default model, use default tag, "tags": [ "default" ]. Recommended model is gpt-oss-120b that is hosted on watsonx.ai™.
  • Register atleast one embedding model with the default tag ("tags": ["default"]) if you intend to use agent knowledge file upload feature. This model becomes the default for all new knowledge bases in the tenant.
  • Register a model that supports Intelligent Document Processing (IDP) to use IDP features.
You can add, display, modify, or delete the external models in watsonx Orchestrate in the following ways:

Supported models for Intelligent Document Processing (IDP)

Important: gpt-oss-120b model is not recommended for using IDP features.
The watsonx Orchestrate supports the following models for IDP:
Structured Document Extractor and Unstructured Document Extractor
mistralai/mistral‑small‑3‑2‑24b‑instruct‑2506
Document Classifier
mistralai/mistral‑small‑3‑2‑24b‑instruct‑2506
Text Extractor with semantic KVP
mistralai/mistral‑small‑3‑2‑24b‑instruct‑2506

To enable the IDP feature in your cluster, see Enabling Intelligent Document Processing.

Important: To get service-instance-url that is required for the cURL commands, see Getting Service instance URL.

Registering the LLM models available in SaaS watsonx.ai

Run the following cURL command to register the models that are available in SaaS watsonx.ai. The example cURL command uses gpt-oss-120b model for illustration. Ensure to specify your preferred model name in the name and display_name field.
Note: In the given example cURL command, when you enter a value in the "name" or "display_name" field, use the format: virtual-model/watsonx/<model_name>. Here <model_name> is the name of the model that you would use to call watsonx.ai directly. For example, to use the gpt-oss-120b model, specify the value as virtual-model/watsonx/openai/gpt-oss-120b.

curl -X POST \
  "https://<service-instance-url>/v1/orchestrate/models" \
  -H "accept: application/json" \
  -H "Authorization: ZenApiKey <ZenApiKey>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "virtual-model/watsonx/openai/gpt-oss-120b",
    "display_name": "virtual-model/watsonx/openai/gpt-oss-120b",
    "provider_config": {
      "api_key": "<wx.ai-api-key>",
      "watsonxSpaceId": "<wx.ai-spaceid>",
      "customHost": "<wx.ai-hostname>"
    },
    "description": "string",
    "tags": [
      "default"
    ],
    "model_type": "chat"
  }' \
  --insecure -v

Registering the LLM models available in external IBM Cloud Pak for Data watsonx.ai cluster

To register the models, you must use API key authentication or Basic authentication.

Required credentials for API key authentication:
Required credentials for Basic authentication:
  • watsonxCpdUsername
  • watsonxCpdPassword
  • watsonx_cpd_url - Must be IBM Cloud Pak for Data instance URL
  • watsonxSpaceId - to get the space id for watsonx.ai deployment space, refer to Viewing and managing deployment spaces.
Run the following cURL command to register the models that are available in SaaS watsonx.ai. The example cURL command uses gpt-oss-120b model for illustration. Ensure to specify your preferred model name in the name and display_name field.
Note: In the given example cURL command, when you enter a value in the "name" or "display_name" field, use the format: virtual-model/watsonx/<model_name>. Here <model_name> is the name of the model that you would use to call watsonx.ai directly. For example, to use the gpt-oss-120b model, specify the value as virtual-model/watsonx/openai/gpt-oss-120b.

curl -X POST \
  "https://<service-instance-url>/v1/orchestrate/models" \
  -H "accept: application/json" \
  -H "Authorization: ZenApiKey <ZenApiKey>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "virtual-model/watsonx/openai/gpt-oss-120b",
    "display_name": "virtual-model/watsonx/openai/gpt-oss-120b",
    "provider_config": {
      "watsonx_cpd_url": "<cpd-wx.ai-instance-url>",
      "watsonxCpdUsername": "<username>",
      "watsonxCpdPassword": "<password>",
      "watsonxSpaceId": "<wx.ai-spaceid>
      
    },
    "description": "string",
    "tags": [
      "default"
    ],
    "model_type": "chat"
  }' \
  --insecure -v

Registering third-party models

Note: When you are using third-party models, ensure to test your models for accuracy that aligns with your specific use case requirements.
You can register third-party models through API or ADK. A few examples are given for illustration. Follow the same procedure to register the models of your choice.
Registering the models through API
Example 1: virtual-model/bedrock/us.anthropic.claude-4-5-sonnet-20250929-v1:0. 

curl -X POST \
  "https://<service-instance-url>/v1/orchestrate/models" \
  -H "accept: application/json" \
  -H "Authorization: ZenApiKey <ZenApiKey>" \
  -d '{
    "name": "virtual-model/bedrock/us.anthropic.claude-4-5-sonnet-20250929-v1:0",
    "display_name": "test",
    "provider_config": {
      "awsSecretAccessKey": "<awsSecretAccessKey>",
      "awsAccessKeyId": "<awsAccessKeyId>"
    },
    "description": "string",
    "tags": [
      "default"
    ],
    "model_type": "chat"
  }'
Example 2: google/gemini-2.5-flash
Note: Generate yourApiKey from https://aistudio.google.com/prompts/new_chat.

curl -X POST \
  "https://<service-instance-url>/v1/orchestrate/models" \
  -H "accept: application/json" \
  -H "Authorization: ZenApiKey <ZenApiKey>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "virtual-model/google/gemini-2.5-flash",
    "display_name": "virtual-model/google/gemini-2.5-flash",
    "description": "None",
    "provider_config": {
      "provider": "google",
      "api_key": "<yourApiKey>"
    },
    "tags": [
      "default"
    ],
    "model_type": "chat"
  }'

Registering the models through ADK

To register the models through ADK, see https://developer.watson-orchestrate.ibm.com/llm/managing_llm.

Registering the embedding model through AI Gateway

Run the following cURL command to register the embedding model through AI Gateway. In this example, the command demonstrates the registration process using the ibm-slate-30m-english-rtrvr model. 

curl -X POST \
  "https://<service-instance-url>/v1/orchestrate/models" \
  -H "accept: application/json" \
  -H "Authorization: ZenApiKey <ZenApiKey>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "virtual-model/watsonx/ibm/slate-30m-english-rtrvr",
    "display_name": "virtual-model/watsonx/ibm/slate-30m-english-rtrvr",
    "provider_config": {
      "api_key": "<wx.ai-api-key>",
      "watsonxSpaceId": "<wx.ai-spaceid>",
      "customHost": "<wx.ai-hostname>"
    },
    "description": "string",
    "tags": [
      "default"
    ],
    "model_type": "embedding"
  }' \
  --insecure -v

Displaying the added models

Use the following example command to get the list of models that are added to your watsonx Orchestrate instance.
curl -X GET "https://<service-instance-url>/v1/orchestrate/models" \
 -H 'accept: application/json' \
 -H "Authorization: ZenApiKey <ZenApiKey>"

Modifying the added models

Use the following example command to modify the models that are added to your watsonx Orchestrate instance. 
curl -X PUT "https://<service-instance-url>/v1/orchestrate/models" \
 -H 'Content-Type: application/json' \
 -H "Authorization: ZenApiKey <ZenApiKey>" \
 -d '{
  "name": "virtual-model/openai/gpt-5-mini",
  "display_name": "new-display-name",
  "provider_config": {
  "customHost": "https://some-url1.com"
  },
  "description": "new-description",
  "tags": [
    "new tag"
  ],
  "model_type": "chat"
}'

Deleting the added models

Use the following example command to delete the models that are added to your watsonx Orchestrate instance. 
curl -X DELETE "https://<service-instance-url>/v1/orchestrate/models" \
 -H 'accept: application/json' \
 -H "Authorization: ZenApiKey <ZenApiKey>"

What to do next

  1. Optional: To store your AI gateway model configuration, see [Optional] Storing your AI gateway model configuration.