You can modify a foundation model's parameters to best fit your generative AI use case. The way you modify the settings depends on the type of model you use.
Before you begin
You must be a system user with access to the credentials to log in to the management cluster where IBM® Sovereign Core is installed.
Procedure
-
Identify the deployment for the foundation model you want to modify:
-
Edit the model deployment and set the parameters you want to configure:
oc edit mdeploy -n aiiaas <model-deployment-name>
The parameters in the
llmInferenceServiceSpec key in the deployment configuration are equivalent to the attributes in the
LLMInferenceServiceSpec class in the KServe reference documentation.
-
You can modify model configuration, such as tool calling capabilities, or update model parameters such as the model context length.
- Tool calling
- To configure your model to use one or more external tools to process the input submitted to the model and generate a response, add
--enable-auto-tool-choice --tool-call-parser <parser name> to the VLLM_ADDITIONAL_ARGS environment variable in the model deployment configuration. For details about the various model-specific parsers you can specify in the <parser> argument, see Tool calling in the vLLM reference documentation.
- Model context length
- The context length model parameter is the total number of tokens in the model's input prompt and the output the model generates. To change the upper limit to the context length, add
--max-model-len <length> to the VLLM_ADDITIONAL_ARGS environment variable in the model deployment and specify the new limit in the <length> argument.