Creating a model asset
After you upload your custom foundation model to cloud object storage, create a connection to the model and a corresponding model asset. Use the connection to create a model asset in a project or space.
To create a model asset, add a connection to the model and then create a model asset. If you want to first test your custom foundation model in a project (for example by evaluating it in a Jupyter notebook), add your custom foundation model asset to a project and then promote it to a space.
After you add the model asset, you can deploy it and use Prompt Lab to inference it.
If you upload your model to remote cloud storage, you must create a connection that is based on your personal credentials. Only connections that use personal credentials are allowed with remote cloud storage. As a result, other users of the same deployment space do not get access to the model content but are allowed to do inference on the model deployments. Create the connection by using your access key and your secret access key. For information on how to enable personal credentials for your account, see Account settings.
Before you begin
You must enable task credentials to be able to deploy a custom foundation model. For more information, see Adding task credentials.
Adding a connection to your model
You can add a connection to your model from either a deployment space or a project. To add a connection programmatically, see Creating a connection by using the Python client or Creating a connection by using the API.
To use the watsonx.ai API, you need a bearer token. For more information, see Credentials for programmatic access.
Adding a connection to the model from a project or a deployment space
To add a connection to the model from your project or your deployment space, follow the steps that are listed in Adding connections to data sources in a project.
-
For Credentials, select
Access key and Secret key. If you select any other option your deployment won't work. -
For connection endpoint you must use one of these endpoint types:
- Public
- Public (Tethered)
- Direct
- Direct (Tethered)
-
When you're prompted for the Credential setting, make sure that you select
Personal credentials. If you do not see the credential setting option, you must enable it in your account. For more information, see Account settings.
Creating a model asset
To create a custom foundation model asset:
-
In your deployment space or your project, go to Assets and then click Import assets.
-
Select Custom foundation model.
-
Select the connection to the cloud storage where the model is located.
-
Select the folder that contains your model.
-
Adjust model details, as needed. If you don't submit any entries in Model deployment parameters, default values are used. For information on available model parameters, see Global parameters for custom foundation models.
-
Click Import.
If you don't select any tasks under Model tasks:
- If the model does not include a chat template, the default task is text generation.
- If the model includes a chat template, the default tasks are: text generation and text chat.
Creating a custom foundation model asset programmatically
To use the watsonx.ai API, you need a bearer token. For more information, see Credentials for programmatic access.
For information on available model parameters, see Global parameters for custom foundation models. If you don't submit any entries for model parameters, default values are used.
See example code for adding a model asset to a deployment space:
curl -X POST "https://<your cloud hostname>/ml/v4/models?version=2024-01-29" \
-H "Authorization: Bearer $TOKEN" \
-H "content-type: application/json" \
--data '{
"type": "custom_foundation_model_1.0",
"framework": "custom_foundation_model",
"version": "1.0",
"name": "<asset name>",
"software_spec": {
"name": "<name of software specification>"
},
"space_id": "<your space ID>",
"foundation_model": {
"model_id": "<model ID>",
"functions": [
"text_generation"
"text_chat"
],
"parameters": [
{
"name": "dtype",
"default": "float16",
"type": "string",
"display_name": "Data Type",
"options": ["float16","bfloat16"]
},
{
"name": "max_batch_size",
"default": 256,
"type": "number",
"display_name": "Max Batch Size"
}],
"model_location": {
"type": "connection_asset",
"connection": {
"id": "<your connection ID>"
},
"location": {
"bucket": "<bucket where the model is located>",
"file_path": "<subpath to model files, if required>"
}
}
}
}'
See example code for adding a model asset to a project:
curl -X POST "https://<your cloud hostname>/ml/v4/models?version=2024-01-29" \
-H "Authorization: Bearer $TOKEN" \
-H "content-type: application/json" \
--data '{
"type": "custom_foundation_model_1.0",
"framework": "custom_foundation_model",
"version": "1.0",
"name": "<asset name>",
"software_spec": {
"name": "<name of software specification>"
},
"project_id": "<your project ID>",
"foundation_model": {
"model_id": "<model ID>",
"functions": [
"text_generation"
"text_chat"
],
"parameters": [
{
"name": "dtype",
"default": "float16",
"type": "string",
"display_name": "Data Type",
"options": ["float16","bfloat16"]
},
{
"name": "max_batch_size",
"default": 256,
"type": "number",
"display_name": "Max Batch Size"
}],
"model_location": {
"type": "connection_asset",
"connection": {
"id": "<your connection ID>"
},
"location": {
"bucket": "<bucket where the model is located>",
"file_path": "<subpath to model files, if required>"
}
}
}
}'
If you don't add any model functions:
- If the model does not include a chat template, the default task is text generation.
- If the model includes a chat template, the default tasks are: text generation and text chat.