Configuring GPU features and models

Use GPU features to boost your watsonx Assistant's efficiency through accelerated processing, handling complex models, real-time inference, and large-scale data handling. This improves the efficiency and accuracy of AI models. Usage of the GPU features is optional.

Permissions you need for these tasks:
You must be an administrator of the Red Hat® OpenShift® project to manage the cluster.
Complete the following tasks to configure the GPU features and supported models:

Enabling or disabling GPU features

To enable the GPU features, use the following command:
oc patch wa wa --type=merge -p="{\"configOverrides\": {\"enabled_components\": {\"store\": {\"ifm\": true}}, \"watsonx_enabled\": true }}"
To disable the GPU features, use the following command:
oc patch wa wa --type=merge -p="{\"configOverrides\": {\"enabled_components\": {\"store\": {\"ifm\": false}}, \"watsonx_enabled\": false }}"

Supported foundation models for GPU features

Note: Supported foundation models are available only when the GPU feature is enabled.
GPU features support the following foundation models during installation:
Specialized model in watsonx Assistant
ibm-granite-8b-unified-api-model-v2
Out of the Box models
  • granite-3-8b-instruct
  • llama-3-1-70b-instruct
For more information, see GPU requirements for models.

Enabling a specific model

Note: If the available GPU memory is inadequate for the installation of a new model, ensure to disable any unused models to clear the space. For more information, see Disabling the Out of the Box model or Disabling the specialized model.

When you enable GPU features, ibm-granite-8b-unified-api-model-v2 and granite-3-8b-instruct models are installed automatically. If you want to install the model of your choice, use the following commands.

To enable the Out of the Box model without modifying replicas or shards:
oc patch wa wa --type='merge' -p='{"configOverrides":{"ifm":{"model_config":{"ootb":{"<model-name>":{}}}}}}'
To enable the Specialized model without modifying replicas or shards: 
oc patch wa wa --type='merge' -p='{"configOverrides":{"ifm":{"model_config":{"syom":{"ibm-granite-8b-unified-api-model-v2":{}}}}}}'
If you want to change the model after installation, you must restart the store deployment to pick up the newly installed models.
oc rollout restart deployment wa-store

Disabling the Out of the Box model

To disable the Out of the Box model, do the following steps:
  1. Set the model's name that you want to remove in model_name.
    export MODEL_NAME="<model-name>"
  2. Remove the entry from watsonx Assistant custom resource.
    oc patch wa wa --type json --patch "[{ "op": "remove", "path": "/configOverrides/ifm/model_config/ootb/$MODEL_NAME" }]"
  3. Remove the entry from watsonx.ai™ IFM custom resource.
    oc get watsonxaiifm watsonxaiifm-cr -o json | jq ".spec.install_model_list -= [\"${MODEL_NAME}\"]" | oc apply -f -
  4. Remove the InferenceService resource for the model.
    oc delete isvc ${MODEL_NAME}

Disabling the specialized model

To disable the specialized model, run the following steps:
  1. Clean up the configuration overrides in custom resource.
    oc patch wa wa --type json --patch '[{ "op": "remove", "path": "/configOverrides/ifm/model_config/syom" }]'
  2. Remove the syom configuration from the watsonassistantstore resource.
    
oc patch watsonassistantstore wa --type json --patch '[{ "op": "remove", "path": "/configOverrides/ifm/model_config/syom" }]'
  3. Delete the configmap that is associated with ibm-granite-8b-unified-api-model-v2.
    oc delete configmap ibm-granite-8b-unified-api-model-v2
  4. Wait for approximately 5 to 10 minutes for the changes to reconcile.

Adjusting Replicas for a model

You can start extra model replicas to handle the increased load.

To enable and adjust replicas for the Specialized model, use the following command:
oc patch wa wa --type='merge' -p='{"configOverrides":{"ifm":{"model_config":{"syom":{"ibm-granite-8b-unified-api-model-v2":{"replicas": <replica-value>}}}}}}'
To enable and adjust replicas for the Out of the Box model, use the following command:
oc patch wa wa --type='merge' -p='{"configOverrides":{"ifm":{"model_config":{"ootb":{"<model-name>":{"replicas": <replica-value>}}}}}}'

Adjusting Shards for Out of the Box model

To enable and adjust shards for Out of the Box model, use the following command:
oc patch wa wa --type='merge' -p='{"configOverrides":{"ifm":{"model_config":{"ootb":{"<model-name>":{"shards": <shard-value>}}}}}}'