Adding foundation models to IBM watsonx.ai

Install with default configuration

Simplest installation option where models are installed on available resources. Some foundation models are sharded as part of their standard configuration. The number of GPUs specified per model in the table in Foundation models in IBM watsonx.ai indicates the number of shards used.

See Installing models with the default configuration.

Note: Use this method for adding embedding and reranker models. You cannot shard embedding or reranker models. Use this method for adding the models that support text extraction also.

Shard the foundation models during installation

You can use sharding to partition large models into smaller units, known as shards, that can be processed across multiple GPU processors in parallel. Sharding foundation models can improve performance and also reduces the amount of memory needed for each GPU.

See Sharding the foundation models.

Install foundation models on preconfigured GPU partitions

Partitioning GPU processors lets you install more than one smaller model on a single GPU to use resources more efficiently.

Important: Not all foundation models can be sharded or installed on partitioned GPU processors. See Foundation models in IBM watsonx.ai for details.

See Installing models on GPU partitions.

Install foundation models on Intel Gaudi 3 AI Accelerator GPUs

A subset of foundation models provided in IBM watsonx.ai can be installed on Intel Gaudi 3 AI Accelerator GPUs.

See Installing models on Intel Gaudi 3 AI Accelerator GPUs.

Install embedding models on GPUs

For better performance while vectorizing text, you can configure embedding models to run on GPUs for better performance.

See Installing embedding models on GPUs.

oc get watsonxaiifm -n ${PROJECT_CPD_INST_OPERANDS}

• To check whether the model predictor pod is running:

  oc get po -n ${PROJECT_CPD_INST_OPERANDS} | grep predictor

• To check whether the predictor pod shows the correct number of GPU processors. If you specified a node selector, you can also check the value of the Node-Selectors field.

  oc describe po <model-predictor-pod> -n ${PROJECT_CPD_INST_OPERANDS}

• To check which node a foundation model is using:

  oc get po -n ${PROJECT_CPD_INST_OPERANDS} -o wide | grep predictor

Shard the foundation models

oc patch watsonxaiifm watsonxaiifm-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type merge \
--patch '{"spec": {"install_model_list":["<model-id>"], "model_install_parameters": {"<model_id_with_underscore>":{"shard": <shard-value>}}}}'

Make the following edits:

• Specify the models that you want to use in a comma-separated list in install_model_list. For example, ["llama-3-1-8b-instruct","ibm-granite-13b-chat-v2"].
• Specify each model that you want to shard by specifying the model ID, but use underscores instead of hyphens in the model ID in model_install_parameters. For example, for the llama-3-1-8b-instruct model, specify "llama_3_1_8b_instruct".
• Assign a shard value to each model in model_install_parameters. The shard value specifies the number of units in which to split the model. Accepted shard values are 2, 4, or 8 only. If you specify a value other than one of these accepted values, the default shard value (number of GPUs) for the model is used. No message is shown to inform you that your configuration change is not applied.

For example, the following patch adds the llama-3-1-8b-instruct and granite-13b-chat-v2 foundation models along with the models that are required for the text extraction feature:

oc patch watsonxaiifm watsonxaiifm-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type merge \
--patch '{"spec": {"install_model_list":["llama-3-1-8b-instruct","ibm-granite-13b-chat-v2","wdu"], "model_install_parameters": {"ibm_granite_13b_chat_v2":{"shard": 2}}}}'

Important: Do not edit the custom resource directly because it is easy to introduce errors.

Shard the foundation models on specific nodes

You can specify the node where you want the shards to run by specifying the hostname of the node in the nodeSelector object.

1. Get a list of the nodes:

   oc get nodes

2. Check the label for the node that you want to use:

   oc describe no <node-name> | grep kubernetes.io/hostname

3. Verify that the node has a GPU by using the following command. Foundation models can be sharded on GPU nodes only.

   oc describe node <node-hostname> | grep nvidia.com/gpu.product

   A GPU node returns nvidia.com/gpu.product=true.
4. Run the following command to patch the custom resource:

   oc patch watsonxaiifm watsonxaiifm-cr \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   --type merge \
   --patch '{"spec": {"install_model_list":["<model-id>"], "model_install_parameters": {"<model_id_with_underscore>":{"shard": <shard-value>, "nodeSelector":{"kubernetes.io/hostname": "<hostname-value>"}}}}}'

   Make the following edits:
   • Specify the models that you want to use in a comma-separated list in install_model_list. For example, ["llama-3-1-8b-instruct","ibm-granite-13b-chat-v2"].
   • Specify each model that you want to shard by specifying the model ID, but use underscores instead of hyphens in the model ID in model_install_parameters. For example, for the llama-3-1-8b-instruct model, specify "llama_3_1_8b_instruct".
   • Assign a shard value to each model in model_install_parameters. The shard value specifies the number of units in which to split the model. Accepted shard values are 2, 4, or 8 only. If you specify a value other than one of these accepted values, the default shard value (number of GPUs) for the model is used. No message is shown to inform you that your configuration change is not applied.
   • For each shard that you want to host on a specific node, specify the node hostname value in the nodeSelector object. For example, "nodeSelector":{"kubernetes.io/hostname":"worker0.example.com"}

For example:

oc patch watsonxaiifm watsonxaiifm-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type merge \
--patch '{"spec": {"install_model_list":["meta-llama-llama-3-8b-instruct","ibm-granite-13b-chat-v2"], "model_install_parameters": {"ibm_granite_13b_chat_v2":{"shard": 2, "nodeSelector":{"kubernetes.io/hostname": "worker0.example.com"}}}}}'

Use the MIG profile label

1. Get a list of the nodes:

   oc get nodes

2. Check the MIG profile label for the NVIDIA Multi-Instance GPU node that you want to use with the following command:

   oc describe node <node-name> | grep nvidia.com/mig.config

   For example:

   oc describe node worker10.wxai.example.com | grep nvidia.com/mig.config=
   nvidia.com/mig.config=all-3g.40gb

3. Run the following command to patch the custom resource:

   oc patch watsonxaiifm watsonxaiifm-cr \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   --type merge \
   --patch '{"spec": {"install_model_list":["<model-id>"], "model_install_parameters": {"<model_id_with_underscore>":{"nodeSelector":{"nvidia.com/mig.config": "<mig-label>"}}}}}'

   Make the following edits:
   • Specify the models that you want to use in a comma-separated list in install_model_list. For example, ["llama-3-1-8b-instruct","ibm-granite-13b-chat-v2"].
   • Specify the node label in the nodeSelector object. For example, "nodeSelector":{"nvidia.com/mig.config":"all-3g.40gb"}

Use the GPU type label

1. Get a list of product labels for the different types of NVIDIA Multi-Instance GPU nodes in your cluster:

   oc get nodes -L nvidia.com/gpu.product

2. Run the following command to patch the custom resource:

   oc patch watsonxaiifm watsonxaiifm-cr \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   --type merge \
   --patch '{"spec": {"install_model_list":["<model-id>"], "model_install_parameters": {"<model_id_with_underscore>":{"nodeSelector":{"nvidia.com/gpu.product": "<product-label>"}}}}}'

   Make the following edits:
   • Specify the models that you want to use in a comma-separated list in install_model_list. For example, ["llama-3-1-8b-instruct","ibm-granite-13b-chat-v2"].
   • Specify the node label in the nodeSelector object. For example, "nodeSelector":{"nvidia.com/gpu.product":"NVIDIA-A100-SXM4-80GB"}

• Set the SERVICE_VERSION environment variable to the watsonx.ai™ service operand version.

  export SERVICE_VERSION=<watsonxai-service-operand-version>

• Use the following command to patch the custom resource:

  cat <<EOF | oc apply -f -
  apiVersion: watsonxaiifm.cpd.ibm.com/v1beta1
  kind: Watsonxaiifm
  metadata:
    name: watsonxaiifm-cr
    namespace: ${PROJECT_CPD_INST_OPERANDS}
  spec:
    blockStorageClass: ocs-storagecluster-ceph-rbd
    install_model_list:
      - codestral-2501
    fileStorageClass: ocs-storagecluster-cephfs
    license:
      accept: true
    version: ${SERVICE_VERSION}
    model_install_parameters:
      codestral_2501:
        env:
          - name: MODEL_NAME
            value: /mnt/models/codestral-2501
          - name: SERVED_MODEL_NAME
            value: mistralai/codestral-2501
          - name: MAX_NUM_SEQS
            value: "8"
          - name: MAX_NEW_TOKENS
            value: "8192"
          - name: MAX_SEQUENCE_LENGTH
            value: "28672"
          - name: DISABLE_PROMPT_LOGPROBS
            value: "true"
          - name: TOKENIZER_MODE
            value: mistral
          - name: LOAD_FORMAT
            value: mistral
          - name: CONFIG_FORMAT
            value: mistral
          - name: ENABLE_PREFIX_CACHING
            value: "1"
          - name: NUM_GPUS
            value: "1"
          - name: CUDA_VISIBLE_DEVICES
            value: "0"
          - name: HUGGINGFACE_HUB_CACHE
            value: /mnt/models/
          - name: HF_MODULES_CACHE
            value: /tmp/huggingface/modules
          - name: PORT
            value: "3000"
          - name: MAX_LOG_LEN
            value: "100"
          - name: GPRC_PORT
            value: "8033"
          - name: ENABLE_VLLM_LOG_REQUESTS
            value: "TRUE"
  EOF

oc get watsonxaiifm watsonxaiifm-cr -n ${PROJECT_CPD_INST_OPERANDS}

• google-flan-t5-xxl
• meta-llama-llama-2-13b-chat
• mncai-llama2-13b-dpo-v7
• bigscience-mt0-xxl

• ESTIMATE_MEMORY_BATCH_SIZE=4
• ESTIMATE_MEMORY=off

#!/bin/bash

export model_id=<replace with model ID>

export PROJECT_CPD_INST_OPERANDS=<replace with instance-namespace>

export current_envs=$(oc get isvc $model_id -n $PROJECT_CPD_INST_OPERANDS -ojsonpath='{.spec.predictor.model.env}')

memory_batch_check=$(echo $current_envs | jq 'map(select(.name == "ESTIMATE_MEMORY_BATCH_SIZE" and .value == "4")) | length > 0')

estimate_memory_check=$(echo $current_envs | jq 'map(select(.name == "ESTIMATE_MEMORY" and .value == "off")) | length > 0')

if [ "$memory_batch_check" = "false" ]; then oc patch InferenceService  $model_id -n $PROJECT_CPD_INST_OPERANDS \
  --type=json \
  -p='[{"op": "add", "path": "/spec/predictor/model/env/-", "value": {"name":"ESTIMATE_MEMORY_BATCH_SIZE","value":"4"}}]' ; fi

if [ "$estimate_memory_check" = "false" ]; then oc patch InferenceService $model_id -n $PROJECT_CPD_INST_OPERANDS \
  --type=json \
  -p='[{"op": "add", "path": "/spec/predictor/model/env/-", "value": {"name":"ESTIMATE_MEMORY","value":"off"}}]' ; fi