Creating a deployment for a custom foundation model

Creating a deployment from the watsonx.ai user interface

Follow these steps to create a deployment for a custom foundation model:

1. In your deployment space or your project, go to the Assets tab.

2. Find your model in the asset list, click the Menu icon , and select Deploy.

3. Adjust model details, as needed.

   Note:

   • Use the Serving name field to specify a name for your deployment instead of deployment ID.
   • The serving name must be unique within the namespace.
   • The serving name must contain only these characters: [a-z,0-9,_] and must be a maximum 36 characters long.
   • In workflows where your custom foundation model is used periodically, consider assigning your model the same serving name each time you deploy it. This way, after you delete and then re-deploy the model, you can keep using the same endpoint in your code.

4. Optional: If you want to override some of the base model parameters, click Model deployment parameters and then enter new parameter values. For information on available model parameters, see Global parameters for custom foundation models.

5. Click Create.

Creating a deployment programmatically

To use the watsonx.ai API, you need a bearer token. For more information, see Credentials for programmatic access.

To deploy a custom foundation model programmatically:

1. Initiate model deployment. See this code for an example deployment to space:

   curl -X POST "https://<your cloud hostname>/ml/v4/deployments?version=2024-01-29" \
   -H "Authorization: Bearer $TOKEN" \
   -H "content-type: application/json" \
   --data '{
     "asset": {
       "id": "b89ff4c6-9264-4e87-9b04-c98e973f9f67"
     },
     "hardware_spec": {
       "name": "1l40s-48g",
       "num_nodes": 1
     },
     "description": "Description",
     "name": "granite-speech",
     "space_id": "<space id>",
     "online": {
       "parameters": {
         "foundation_model": {
           "tool_call_parser": "granite",
           "chat_template": "granite-3.3.j2",
           "functions": [
               "text_generation",
               "text_chat",
               "image_chat"
           ],
         }
       }
     }
   }'
   

   For hardware_spec use:

   • General-purpose models: the name of the GPU-based configuration that you want to assign to your model
   • Time-series models: The size of the CPU-based configuration that you want to assign to your model
   
   

   For project deployments, instead of space_id use project_id.

   If needed, for models that use chat API, use the chat_template field to provide the name of the template that overrides the model's standard chat template.

   For functions:

   • If you don't select any functions:
     • If the model does not include a chat template, the default task is text generation.
     • If the model includes a chat template, the default functions are: text_generation and text_chat.
   • The functions that you select when deploying your model override the functions that you set at the model creation stage.

   The deployment ID is returned in the API response, in the metadata.id field.

2. Use the deployment ID to poll for the deployment status. See this code for an example of how to poll for the status of a model that is deployed to a project.

   curl -X GET "https://<your cloud hostname>/ml/v4/deployments/<your deployment ID>?version=2024-01-29&project_id=<your project ID>" \
   -H "Authorization: Bearer $TOKEN"
   

   The deployed_asset_type is returned as custom_foundation_model. Wait until the status changes from initializing to ready.

Next steps

Prompting a custom foundation model