You can split graphics processing units (GPUs) into multiple GPU instances on the shared AI inference service cluster. Partitioning GPUs is useful when a foundation model does not require all of the resources that are available on the GPU. Each partition is treated as an independent GPU.
Before you begin
You must have access to the credentials to log in to the AI inference service cluster where IBM® Sovereign Core is installed.
Procedure
-
Set up the AI inference service as follows:
- Create the shared cluster for the service with the GPU operator installed.
- Remove all foundation model pods from the cluster. You cannot partition GPUs when model deployments are active.
For details, see
Installing AI inference service.
-
Follow the instructions to partition GPUs for the specific GPU type in the service cluster.
For details about how to partition
NVIDIA GPUs, see the
Configuring NVIDIA Multi-Instance GPUs sample runbook in the
IBM Sovereign Core GitHub repository.
Note: Resource notation for mixed strategy partitioning with NVIDIA GPUs is not visible on the user interface (UI). For mixed strategy, you can deploy models by using the command line interface (CLI). For details, see
Managing foundation models.
What to do next
Redeploy any foundation models that you removed by using the one of the following methods: