Enabling custom parameters for model tuning

When custom parameters for model tuning are enabled, users can pass custom parameters directly to the underlying trainer (fms-hf-tuning) when they're fine-tuning foundation models.

Before you begin

You must be a cluster administrator.

Attention: Enabling custom parameters for model tuning allows users to pass arbitrary parameters to the trainer without validation, which can cause training jobs to fail if incompatible parameters are provided.

About custom parameters

Enabling custom parameters allows experienced users to configure training parameters beyond the standard options that are provided by the fine-tuning API. When enabled, users can include a custom.parameters object in their training payload that passes parameters directly to the underlying trainer.

Key characteristics:

Custom parameters are merged into the generated trainer configuration
Custom parameters override standard parameters when conflicts occur
No validation is performed on custom parameters
Incompatible parameters cause training to fail at runtime, not at submission time

Procedure

To enable custom parameters for model tuning:

Set the wml-cr to maintenance mode with the following command:

oc patch wmlbase wml-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{"spec":{"ignoreForMaintenance": true}}'

Update the training configuration to enable custom parameters:

oc patch cm wmltrainingconfigmap \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{"data":{"service.fine_tuning.allow_custom_parameters": "true"}}'

To disable the feature, set the value to false.

Note the names of training pods after running the following command:
```
oc get pods | grep wmltraining
```
Restart the training pods by using the pod names:
```
oc delete pod <training-pod-name>
```
Note the names of Watson Studio pods after running the following command:
```
oc get pods | grep portal-ml-dl
```
Restart the Watson Studio pods by using the pod names:
```
oc delete pod <studio-pod-name>
```

Take the wml-cr out of maintenance mode:

oc patch wmlbase wml-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{"spec":{"ignoreForMaintenance": false}}'

Behavior when enabled or disabled

When enabled: Values in custom.parameters are merged into the generated trainer configuration. If a key conflicts with a standard parameter, the custom value takes precedence. The API returns a warning message: "Custom training parameters are used at your own risk. Custom parameters will override conflicting standard training parameters and may be incompatible, potentially causing training to fail."
When disabled (default): The custom object is persisted and returned in API responses, but custom.parameters have no effect on training. The API returns a warning message: "Custom training parameters are not supported and will be ignored."

Important considerations

Important: Enable this feature only if your users require advanced training configuration options and understand the risks. Custom parameters are passed directly to the underlying trainer without validation and can cause training to fail if incompatible values are provided.