Properties and parameters for custom foundation models
You can set and adjust the parameters of your custom foundation model to define its behavior.
Global parameters for custom foundation models
You can use global parameters to deploy your custom foundation models. You must set the value of your base model parameter within the range specified in the following table. If you don't do that, your deployment might fail and inferencing will not be possible. If the default values for your model parameters result in an error, contact your administrator to modify the model's registry in the watsonxaiifm CR.
You can use the following global parameters within the specified range of values to deploy your custom foundation models:
| Parameter | Type | Range of values | Default value | Description |
|---|---|---|---|---|
dtype |
String | float16, bfloat16 |
float16 |
Use this parameter to specify the data type for your model. |
max_batch_size |
Number | max_batch_size >= 1 |
256 | Use this parameter to specify the maximum batch size for your model. |
max_concurrent_requests |
Number | max_concurrent_requests >= 1 and max_concurrent_requests >= max_batch_size |
1024 | Use this parameter to specify the maximum number of concurrent requests that can be made to your model. |
max_new_tokens |
Number | max_new_tokens >= 20 |
2048 | Use this parameter to specify the maximum number of tokens that can be generated by your model for an inference request. |
max_sequence_length |
Number | max_sequence_length >= 20 and max_sequence_length > max_new_tokens |
2048 | Use this parameter to specify the maximum sequence length for your model. |
If you use the watsonx-cfm-caikit-1.1 software specification to deploy your model, the value of the max_concurrent_requests parameter is not used.
For detailed parameter descriptions, see Properties for global parameters for custom foundation models.
Properties for global parameters for custom foundation models
You can use the following properties for the global parameters for custom foundation models:
| Property | Type | Required or optional | Description |
|---|---|---|---|
name |
String | Required | Use this property to specify the name of the parameter. |
default |
String, number, boolean | Required | Use this property to specify the default value of the parameter. |
min |
Number | Optional | Use this property to specify the minimum value of the paratemer. The min value must be less than or equal to the entered value. |
max |
Number | Optional | Use this property to specify the maximum value of the parameter. The max value must be greater than or equal to the entered value. |
options |
String, number | Optional | Use this property to specify a list of options to choose for the parameter. The type of options value must be the same as parameter value. The selected value must be from within the options list. |
- If the system administrator sets default model parameters at the model registration phase, you can override them at the creation phase and during an update.
- If the system administrator does not set default parameters during the model registration phase, watsonx sets the default parameters at the creation phase. You can then override them during an update.
Next steps
Creating a deployment for a custom foundation model
Parent topic: Deploying custom foundation models