Manually scaling resources for foundation models

Scale a foundation model deployment by increasing the number of replicas of the model to improve throughput and increase processing capacity during peak usage times.

Before you begin

  • To complete this task the first time, you must be an instance administrator of the IBM watsonx.ai.
  • Review the system requirements for the foundation models that you want to scale to ensure that you have the necessary resources to support more copies of the foundation models. See System requirements for foundation models in IBM watsonx.ai. Make a note of the model IDs for the foundation models.

Procedure

To scale foundation model resources:

Use the following command to change the number of foundation model replicas: 
oc patch watsonxaiifm watsonxaiifm-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch '{"spec": { "<model_id_with_underscores>_replicas": "2"}}'
When you refer to the foundation model, change the model ID to include underscores instead of hyphens. For example, use the following command to change the number of replicas of the ibm/granite-3-8b-instruct model:
oc patch watsonxaiifm watsonxaiifm-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch '{"spec": { "ibm_granite_3_8b_instruct_replicas": "2"}}'