Deploying or undeploying LLMs in AI Optimizer for IBM Z and IBM LinuxONE

AI Optimizer for Z and LinuxONE version 3.1.2 supports multiple large language models for local deployment, including Granite™ inference models for instruction-following and reasoning, and embedding models for semantic search and retrieval tasks. You can deploy one or more model instances by using the inference router UI.

1. From your web browser, log in to the AI Optimizer for Z and LinuxONE UI (https://<ssc-lpar-ip>) with a valid username and password.

   By default, the UI opens to the Inference router page.

2. On the Inference router page, start the LLM deployment by clicking the Deploy LLM button.
3. On the Specify an LLM page, select an LLM and provide the required details.
   1. From the Type option menu, select the model you want to deploy.

      Version 3.1.2 supports the following models:

      • Inference models (require 4 Spyre Accelerator cards):
        • ibm/granite-3.3-8b-instruct
        • ibm/granite-3.3-8b-instruct-FP8
        • ibm/granite-4.1-8b
        • ibm/granite-4.1-8b-fp8
        • mistralai/Ministral-3-14B-Instruct-2512-BF16
      • Embedding models (require 1 Spyre Accelerator card):
        • ibm/granite-embedding-125m-english
        • ibm/granite-embedding-278m-multilingual

      See Supported models and resource requirements for AI Optimizer for IBM Z and IBM LinuxONE for detailed specifications and resource requirements for each model.

   2. Enter a descriptive display name that can uniquely identify the deployment.

      For example, use granite-338b-inst-1 for the first instance of the granite-3.3-8b-instruct model, or embedding-english-1 for an embedding model deployment.

   3. Enter a brief description of the deployment in the optional Description field.
   4. From the list of available Spyre Accelerator cards, select the required number of GPU cards for the model.

      The number of Spyre cards required depends on the model type:

      • Inference models: Select 4 Spyre cards
      • Embedding models: Select 1 Spyre card

      If a Spyre card is currently in use or selected for another model deployment, the card will no longer be available for new selection. See Supported models and resource requirements for AI Optimizer for IBM Z and IBM LinuxONE for detailed Spyre card requirements.

   5. Continue the deployment by clicking Next.
4. On the Assign tags page, assign one or more tags to the specified LLM.

   Tags are labels or markers used to describe, distinguish, and categorize LLMs. When applied effectively, they help applications quickly route inference requests to the best matching LLMs.

   Assign one or more tags to the specified LLM. If the model comes with predefined tags, you can change them or add more based on your actual AI inference need.

   When needed, create custom tags. When customizing tags, make sure that they are short, precise, purposeful, and meaningful. Name the tags based on the intent, context, capabilities, application domains, use cases, and other characteristics of the LLMs. Use consistent naming conventions across the tags.

   The AI Optimizer for Z and LinuxONE inference router uses tag-based routing and routes your inference requests to the most appropriate model by using the tags that you specify.

   Click Next to continue.

5. On the Review deployment details page, verify that the LLM information, assigned tags, and selected GPU cards are correct.
6. On the Review deployment details page, complete the LLM deployment by clicking the Deploy button.

   The UI returns to the Inference router page, and the new deployment is added to the LLMs list with the Deploying status. The deployment process runs in the background, and it may take a few moments to complete.

   If needed, you can monitor the deployment by going to the Logs page of the Appliance Manager UI. Once there, navigate to the Service logs tab and select RHAIIS (All services) from the service selection menu. You can watch the deployment activities live or download the log when the process completes.

   The LLM deployment completes successfully when the status changes to Enabled. If the status changes to Failed, you will need to identify and fix any issues and then click the Restart deployment () icon to repeat the deployment process.

7. If needed, remove an LLM deployment that you no longer need.
   1. From the LLMs list, select a deployment that you want to remove.
   2. Click the Delete () icon to open the Confirm delete dialogue box.
   3. Confirm the deployment removal by clicking Delete.

      Remember that deleting an LLM deployment will permanently remove the deployment configuration. You must create a new deployment when you need to use the LLM again.

   4. Verify that the removed deployment is no longer on the LLMs list on the Inference router page.