Configuring GPU features and models

Use GPU features to boost your watsonx Orchestrate's efficiency through accelerated processing, handling complex models, real-time inference, and large-scale data handling. This improves the efficiency and accuracy of AI models. This topic guides you through updating your preferred models, disabling unused ones, or enabling models after the installation or upgrade process.

Permissions that you need for these tasks:: You must be an administrator of the Red Hat® OpenShift® project to manage the cluster.

Restriction: This topic is applicable only if you have not set any user-defined models by using custom installation option.

Complete the following tasks to configure the GPU features and supported models:

Supported foundation models for GPU features
Enabling a specific model
Disabling the Out of the Box model
Disabling the specialized model
Disabling GPU features

Supported foundation models for GPU features

Note: Supported foundation models are available only when the GPU feature is enabled.

GPU features support the following foundation models during installation:

Specialized model in watsonx Orchestrate: ibm-granite-8b-unified-api-model-v2

Out of the Box models

llama-3-2-90b-vision-instruct
llama-3-1-70b-instruct
granite-3-8b-instruct

For more information, see GPU requirements for models.

Enabling a specific model

Note: If the available GPU memory is inadequate for the installation of a new model, ensure to disable any unused models to clear the space. For more information, see Disabling the Out of the Box model or Disabling the specialized model.

When you enable GPU features, ibm-slate-30m-english-rtrvr model is installed automatically.

If you want to install the model of your choice, use the following commands:

To enable the Out of the Box model:

oc patch wo wo \ --namespace=${PROJECT_CPD_INST_OPERANDS} \ --type=merge \ --patch='{ "spec": { "watsonAssistants": { "config": { "configOverrides": { "enabled_components": { "store": { "ifm": true } }, "watsonx_enabled": true, "ifm": { "model_config": { "ootb": { "<model-name>": {} } } } } } } } }'

To enable the Specialized model:

oc patch wo wo \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{
  "spec": {
    "watsonAssistants": {
      "config": {
        "configOverrides": {
          "enabled_components": {
            "store": {
              "ifm": true
            }
          },
          "watsonx_enabled": true,
          "ifm": {
            "model_config": {
              "syom": {
                "ibm-granite-8b-unified-api-model-v2"
              }
            }
          }
        }
      }
    }
  }
}'

If you want to change the model after installation, you must restart the store deployment to pick up the newly installed models.
```
oc rollout restart deployment wo-wa-store
```

Disabling the `Out of the Box` model

To disable the Out of the Box model, do the following steps:

Set the model's name that you want to remove in MODEL_NAME.
```
export MODEL_NAME="<model-name>"
```

Remove the entry from watsonx™ Orchestrate custom resource.

oc patch wo wo \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=json \
--patch "[{
  \"op\": \"remove\",
  \"path\": \"/spec/watsonAssistants/config/configOverrides/ifm/model_config/ootb/${MODEL_NAME}\"
}]"

Verify that the disabled model is removed from the watsonx Assistant custom resource:

oc get watsonassistantstores -n "${PROJECT_CPD_INST_OPERANDS}" -o json | jq '.items[0].configOverrides.ifm.model_config.ootb'

Remove the entry from watsonx.ai™ IFM custom resource.

oc get watsonxaiifm watsonxaiifm-cr -o json | jq ".spec.install_model_list -= [\"${MODEL_NAME}\"]" | oc apply -f -

Remove the InferenceService resource for the model.
```
oc delete isvc ${MODEL_NAME}
```

Disabling the specialized model

To disable the specialized model, run the following command:

oc patch wo wo \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=json \
--patch "[{
  \"op\": \"remove\",
  \"path\": \"/spec/watsonAssistants/config/configOverrides/ifm/model_config/syom\"
}]"

To verify that the disabled model is removed from the system, do the following steps:

Check and verify whether the disabled model is listed:
```
oc get isvc -n ${PROJECT_CPD_INST_OPERANDS}
```

If the disabled model is listed, delete the same by using the following command:

oc delete isvc ibm-granite-8b-unified-api-model-v2 -n ${PROJECT_CPD_INST_OPERANDS}

If the disabled model is still listed, use the following command to delete the same:
```
oc delete cm ibm-granite-8b-unified-api-model-v2 -n {PROJECT_CPD_INST_OPERANDS}
```

Disabling GPU features

To disable the GPU features, use the following command:

oc patch wo wo --type=merge -p '{"spec": {"watsonAssistants": {"config": {"configOverrides": {"enabled_components": {"store": {"ifm": false}}, "watsonx_enabled": false}}}}}'