Known issues and limitations for IBM watsonx.ai

Applies to:

• IBM® Software Hub version 5.1.0
• IBM watsonx.ai version 10.0.0

Fixed in:

• IBM Software Hub version 5.1.1
• IBM watsonx.ai version: 10.1.0

Problem

Serviceability tools like diagnostic log collection and OOTB monitor don't work if the watsonxaiifm add-on ID is missing when you run the following command:

cpd-cli diag list-services --profile=${CPD_PROFILE_NAME}

 ID                       Version             Name                                                                  Type                Scripts  
 ---------                ---------           ------------                                                          ------------        ------------  
 zenbase                  -                   IBM Cloud Pak for Data                                                Platform            - 
 spark                    5.1.1               Analytics Engine powered by Apache Spark                              Service             - 
 ccs                      10.1.0              Common Core Services                                                  Service             Catalog API serviceability logs, Runtime Manager serviceability logs, Connection serviceability logs, Connector serviceability logs, Flight serviceability logs 
 rshaper                  10.1.0              Data Refinery                                                         Service             - 
 databases                5.1.0               Common Database Services                                              Service             - 
 ikc_premium              5.1.1               IBM Knowledge Catalog Premium                                         Service             - 
 watsonx-data             2.1.1               watsonx.data                                                          Service             Console API Logs, Console UI Logs, Ingest API Logs, Data Access Service (DAS) Logs, Common Policy Gateway (CPG) Logs, Metadata Service (MDS) Rest Logs, Metadata Service (MDS) Thrift Logs, Presto Single Node Logs, Presto Coordinator Logs, Presto (Java) Worker Logs, Presto (C++) Worker Logs, Presto Single Node Files (Property and Config), Presto Coordinator Files (Property and Config), Presto (Java) Worker Files (Property and Config), Engine Query Metrics, Query History Monitoring and Metrics (QHMM) Logs 
 wkc                      5.1.1               IBM Knowledge Catalog                                                 Service             - 
 ibm_neo4j                1.1.1               IBM Neo4j                                                             Service             Neo4j Instance Report 
 

Cause

The output is missing the watsonxaiifm add-on in the installed service list .

Solution

The output now shows the watsonxaiifm add-on in the installed service list when you run the following command:

cpd-cli diag list-services --profile=${CPD_PROFILE_NAME}

 ID                       Version             Name                                                                  Type                Scripts  
 ---------                ---------           ------------                                                          ------------        ------------  
 zenbase                  -                   IBM Cloud Pak for Data                                                Platform            - 
 spark                    5.1.1               Analytics Engine powered by Apache Spark                              Service             - 
 ccs                      10.1.0              Common Core Services                                                  Service             Catalog API serviceability logs, Runtime Manager serviceability logs, Connection serviceability logs, Connector serviceability logs, Flight serviceability logs 
 rshaper                  10.1.0              Data Refinery                                                         Service             - 
 databases                5.1.0               Common Database Services                                              Service             - 
 ikc_premium              5.1.1               IBM Knowledge Catalog Premium                                         Service             - 
 watsonx-data             2.1.1               watsonx.data                                                          Service             Console API Logs, Console UI Logs, Ingest API Logs, Data Access Service (DAS) Logs, Common Policy Gateway (CPG) Logs, Metadata Service (MDS) Rest Logs, Metadata Service (MDS) Thrift Logs, Presto Single Node Logs, Presto Coordinator Logs, Presto (Java) Worker Logs, Presto (C++) Worker Logs, Presto Single Node Files (Property and Config), Presto Coordinator Files (Property and Config), Presto (Java) Worker Files (Property and Config), Engine Query Metrics, Query History Monitoring and Metrics (QHMM) Logs 
 wkc                      5.1.1               IBM Knowledge Catalog                                                 Service             - 
 ibm_neo4j                1.1.1               IBM Neo4j                                                             Service             Neo4j Instance Report 
 watsonx_ai_ifm           10.1.0              watsonx.ai inference foundation models                                Service             - 
 

Applies to:

• IBM Software Hub version 5.1.0
• IBM watsonx.ai version 10.0.0

Fixed in:

• IBM Software Hub version 5.1.1
• IBM watsonx.ai version: 10.1.0

Problem

The following message is generated during training: AutoAI RAG experiment failed with: Not unique document file names passed in connections.

Cause

Adding files with duplicate names as part of the document collection for training a RAG experiment is not supported.

Solution

Remove any duplicate files or rename files with different content but the same file name before you run the experiment.

Applies to:

• IBM Software Hub version 5.1.0
• IBM watsonx.ai version 10.0.0

Fixed in:

• IBM Software Hub version 5.1.1
• IBM watsonx.ai version: 10.1.0

Problem

If you are running an AutoAI experiment for retrieval-augmented generation (RAG), the experiment might fail with this error:

Failure during generate. (POST [Internal URL]\nStatus code: 400, body: 
{"errors":[{"code":"invalid_input_argument","message":"Invalid input argument for Model 
\'google/flan-ul2\': the number of input tokens 5601 cannot exceed the total tokens limit 
4096 for this model

Cause

This failure can happen when the tokens used exceed the capacity of one of the foundation models used in the experiment. For example, if you use HTML files for your document collection, the number of tokens submitted might be much higher than you expect because the HTML tagging uses up tokens in addition to the text.

Solution

To resolve the issue, open the Experiment settings page for the experiment and try these configuration changes:

• Deselect the smallest foundation model available for the experiment.
• Disable the window retrieval method.

Run the experiment again.

Applies to:

• IBM Software Hub version 5.1.2 and later
• IBM watsonx.ai version 10.2.0 and later

Problem

When running an inference notebook to test and evaluate a pattern generated in an AutoAI RAG experiment, the following error is generated: No module named 'unitxt'.

Cause

The unitxt==1.14.0 library is needed to evaluate the patterns but it is not installed on the runtime by default.

Solution

Install the unitxt==1.14.0 library.

Applies to: 5.0.0 and later

Problem

When you check the status of the pod that hosts a provided foundation model, a CrashLoopBackOff message is displayed. The logs mention, Error while deserializing header.

Cause

The provided foundation model was not downloaded successfully by the install job images that download the model content to the cluster.

Solution

Download the model again. To do so, delete the model's persistent volume claim (PVC), which is basically a mechanism for reserving storage for the model, and then restart the service operator by using the following commands:

1. Find and delete the pod where the model is being deployed.

   oc get pods | grep <start-of-model-id> | awk '{print $1}' | xargs oc delete pod

   For example, for the mixtral-8x7b-instruct-v01 foundation model, the command includes the first part of the model's model id, which is mistralai-mixtral-8x7b-instruct-v01:

   oc get pods | grep mistralai-mixtral-8x7b | awk '{print $1}' | xargs oc delete pod

2. Delete the inference service for the model.

   oc delete inferenceservice <model-id>

   For example:

   oc delete inferenceservice mistralai-mixtral-8x7b-instruct-v01

3. Delete the persistent volume claim for the model.

   oc delete pvc <model-id>-pvc

   For example:

   oc delete pvc mistralai-mixtral-8x7b-instruct-v01-pvc

4. Delete the watsonx-ai-ifm-operator pod in the operator namespace.

   oc delete pod <watsonx-ai-ifm-operator-pod> -n ${PROJECT_CPD_INST_OPERANDS}

Now, you can try to install the model again. For more information, see Adding foundation models.

Applies to: 5.0.0 and later

Problem

To add a custom foundation model, you created a custom deployment by using a predefined hardware specification. However, after you deploy the custom foundation model, the following error is displayed: Failed to deploy the custom foundation model. The runtime failed to start.

Cause

The predefined hardware specification does not allocate enough resources to the custom foundation model.

Solution

Define a custom hardware specification that has enough resources to support your custom foundation model. For more information, see Creating custom hardware specifications.

Applies to: 5.1.0 and later

Problem

When you try to deploy the llama-3-1-70b-instruct foundation model on four L40S GPUs, the watsonxaiifm-cr shows the following error message:

Reconcile History: The failed task is : unknown task and the error message is: \
No message available

The llama-3-1-70b-instruct-predictor pod shows the following message:

Back-off restarting failed container kserve-container in pod
      llama-3-1-70b-instruct-predictor-etc_ibm-cpd-operands

The logs of the kserve-container in the predictor pod show the error message:

ValueError: The model's max seq len (131072) is larger than the maximum number \
of tokens that can be stored in KV cache (81840). Try increasing `gpu_memory_utilization` \
or decreasing `max_model_len` when initializing the engine.

Cause

The maximum sequence length of 131,072 that is specified for the foundation model is too long for the provided resources to handle.

Solution

Set a lower maximum sequence length value for the llama-3-1-70b-instruct foundation model. Run a patch command to set the value of MAX_SEQUENCE_LENGTH to 80,000. Because the maximum sequence length parameter setting is an environment parameter, other values also need to be specified in the command. Complete the following steps:

1. Before you change any values, you can check the current values for the configuration by using the following command:

   oc get deploy llama-3-1-70b-instruct-predictor -o yaml

2. Set the value of ${PROJECT_CPD_INST_OPERANDS} to the namespace where the model is installed, and then run this command:

   oc patch --namespace ${PROJECT_CPD_INST_OPERANDS} watsonxaiifm watsonxaiifm-cr --type merge --patch '{"spec": { "model_install_parameters": {"llama_3_1_70b_instruct": {"env": \
   [{ "name": "MODEL_NAME", "value": "/mnt/models/models--meta-llama--llama-3-1-70b-instruct" },{ "name": "SERVED_MODEL_NAME", "value": "meta-llama/llama-3-1-70b-instruct" },{ "name": "MAX_SEQUENCE_LENGTH", "value": "80000" },{ "name": "MAX_NUM_SEQS", "value": "8" },
   { "name": "MAX_NEW_TOKENS", "value": "8192" },{ "name": "DISABLE_PROMPT_LOGPROBS", "value": "true" },{ "name": "ENABLE_AUTO_TOOL_CHOICE", "value": "true" },{ "name": "TOOL_CALL_PARSER", "value": "llama3_json" },{ "name": "CHAT_TEMPLATE", "value": "/mnt/models/models--meta-llama--llama-3-1-70b-instruct/tool_chat_template_llama3.1_json.jinja" },
   { "name": "VLLM_ATTENTION_BACKEND", "value": "XFORMERS" },{ "name": "NUM_GPUS", "value": "4" },{ "name": "CUDA_VISIBLE_DEVICES", "value": "0,1,2,3" },{ "name": "HUGGINGFACE_HUB_CACHE", "value": "/mnt/models/" },{ "name": "HF_MODULES_CACHE", "value": "/tmp/huggingface/modules" },{ "name": "PORT", "value": "3000" },
   { "name": "MAX_LOG_LEN", "value": "100" },{ "name": "GPRC_PORT", "value": "8033" },{ "name": "NCCL_NVLS_ENABLE", "value": "0" } \]}}}}'

3. After you run the patch command, the watsonxaiifm-cr switches to the InProgress state, and then reaches the Completed state. You can use the following command to verify that the predictor pod is running:

   # oc get po | grep predictor

Applies to: 5.1.2 and later

Problem

When you try to deploy the llama-3-3-70b-instruct foundation model on four L40S GPUs, the watsonxaiifm-cr shows the following error message:

Reconcile History: The failed task is : unknown task and the error message is: \
No message available

The llama-3-3-70b-instruct-predictor pod shows the following message:

Back-off restarting failed container kserve-container in pod
      llama-3-3-70b-instruct-predictor-etc_ibm-cpd-operands

The logs of the kserve-container in the predictor pod show the error message:

ValueError: The model's max seq len (131072) is larger than the maximum number \
of tokens that can be stored in KV cache (81840). Try increasing `gpu_memory_utilization` \
or decreasing `max_model_len` when initializing the engine.

Cause

The maximum sequence length of 131,072 that is specified for the foundation model is too long for the provided resources to handle.

Solution

Set a lower maximum sequence length value for the llama-3-3-70b-instruct foundation model. Run a patch command to set the value of MAX_SEQUENCE_LENGTH to 80,000. Because the maximum sequence length parameter setting is an environment parameter, other values also need to be specified in the command. Complete the following steps:

1. Before you change any values, you can check the current values for the configuration by using the following command:

   oc get deploy llama-3-3-70b-instruct-predictor -o yaml

2. Set the value of ${PROJECT_CPD_INST_OPERANDS} to the namespace where the model is installed, and then run this command:

   oc patch --namespace ${PROJECT_CPD_INST_OPERANDS} watsonxaiifm watsonxaiifm-cr --type merge --patch '{"spec": { "llama_3_3_70b_instruct_replicas": "1","llama_3_3_70b_instruct_resources": {"limits": {"cpu": "3", "ephemeral-storage": "1Gi", "memory": "96Gi", "nvidia.com/gpu": "4"},
   "requests": {"cpu": "2", "ephemeral-storage": "10Mi", "memory": "85Gi", "nvidia.com/gpu": "4"}},"model_install_parameters": {"llama_3_3_70b_instruct": {"env": \[{ "name": "MODEL_NAME", "value": "/mnt/models/llama-3-3-70b-instruct" },{ "name": "SERVED_MODEL_NAME", "value": "meta-llama/llama-3-3-70b-instruct" },\
   { "name": "MAX_SEQUENCE_LENGTH", "value": "80000" },{ "name": "MAX_NUM_SEQS", "value": "8" },{ "name": "MAX_NEW_TOKENS", "value": "8192" },{ "name": "DISABLE_PROMPT_LOGPROBS", "value": "true" },{ "name": "ENABLE_AUTO_TOOL_CHOICE", "value": "true" },{ "name": "TOOL_CALL_PARSER", "value": "llama3_json" },
   { "name": "CHAT_TEMPLATE", "value": "/mnt/models/llama-3-3-70b-instruct/tool_chat_template_llama3.1_json.jinja" },{ "name": "VLLM_ATTENTION_BACKEND", "value": "XFORMERS" },{ "name": "NUM_GPUS", "value": "4" },{ "name": "CUDA_VISIBLE_DEVICES", "value": "0,1,2,3" },{ "name": "HUGGINGFACE_HUB_CACHE", "value": "/mnt/models/" },
   { "name": "HF_MODULES_CACHE", "value": "/tmp/huggingface/modules" },{ "name": "PORT", "value": "3000" },{ "name": "MAX_LOG_LEN", "value": "100" },{ "name": "GPRC_PORT", "value": "8033" },{ "name": "NCCL_NVLS_ENABLE", "value": "0" } \], "shard": "4"}}}}'

3. After you run the patch command, the watsonxaiifm-cr switches to the InProgress state, and then reaches the Completed state. You can use the following command to verify that the predictor pod is running:

   # oc get po | grep predictor

Applies to: 5.0.0 and later

Problem

After upgrading from a 4.8.x release to a 5.0.x release or 5.1.0 release, some deprecated or withdrawn models incorrectly return an <|endoftext|> token at the end of the response.

Cause

A mismatch between upgraded services and the deprecated or withdrawn model versions causes the deprecated model to generate incorrect tokens.

Solution

1. Run the following command to find the deployment for the deprecated or withdrawn model that generates the incorrect <|endoftext|> tokens:

   oc get deploy | grep inference-server | grep -v predictor

2. Edit the deployment to remove the OUTPUT_SPECIAL_TOKENS configuration from the env section of the deployment:

   oc set env deployment/<deployment_name_for_deprecated_model> OUTPUT_SPECIAL_TOKENS-

3. Wait for the model to start a new pod. If you do not have enough GPU space, you may have to delete the older pod in the deployment so that a new pod can start.

Applies to: 5.1.1

Fixed in: 5.1.2

Problem

After you follow the instructions to remove the codestral-2501 foundation model from your deployment, the model remains on your cluster.

Cause

Due to a mismatch in references, the foundation model image is not removed by the procedure.

Solution

To remove the model, use the following command:

oc get isvc -n ${PROJECT_CPD_INST_OPERANDS} \
oc delete isvc codestral-2501

Applies to: 5.0.0 and later

Problem

Inference requests that are sent to prompt-tuned foundation models with AI guardrails turned on (from Prompt Lab or by enabling the HAP or PII moderations in API) fail with the message Downstream Caikit request with Model failed.

Cause

Under heavy load, the readiness probe fails causing the service/fmaas-caikit-inf-prompt-tunes to stop sending network traffic to the pod. Because no more pods are available for the service, requests are refused.

Solution

Increase the number of replicas of the fmaas-caikit-inf-prompt-tunes pod so that a healthy pod is available to handle requests. Use the following command to increase the number of replicas to 2:

oc patch cais fmaas \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{"spec":{"inference":{"caikitDeployment":{"deployments":{"promptTunes":{"replicas": 2}}}}}}'

Applies to: 5.1.1

Fixed in: 5.1.2

Problem

The persistent volume claim (PVC) size in the watsonxaiifm-cr operator is 6Gi for the granite-3-2b-instruct model.

Solution

To fix the PVC size for the model, use the following procedure:

1. Delete the PVC and reinstall the model. Update the watsonxaiifm custom resource when you install the granite-3-2b-instruct model:

    oc patch watsonxaiifm watsonxaiifm-cr \
   --namespace=${PROJECT_CPD_INST_OPERANDS} --type merge \
   --patch '{"spec": { "model_install_parameters": {"granite_3_2_2b_instruct": {"pvc_size": "20Gi" }}}}'

2. Restart the operator pod:

   oc delete po <operator pod for ifm> -n ${PROJECT_CPD_INST_OPERATORS}

3. Optional: If the PVC size is still 6Gi, delete the PVC, reinstall the model, and restart the watsonxaiifm operator pod. After the PVC is created from the operator and before it starts downloading the model content, update the PVC size to 20Gi from the UI.

Applies to: 5.1.1 and later

Problem

The persistent volume claim (PVC) size in the watsonxaiifm-cr operator is 20Gi for the llama-3-1-8b model.

Solution

To fix the PVC size for the model, use the following procedure:

1. Delete the PVC and reinstall the model. Update the watsonxaiifm custom resource when you install the llama-3-1-8b model:

    oc patch watsonxaiifm watsonxaiifm-cr \
   --namespace=${PROJECT_CPD_INST_OPERANDS} --type merge \
   --patch '{"spec": { "model_install_parameters": {"llama_3_1_8b": {"pvc_size": "40Gi" }}}}'

2. Restart the operator pod:

   oc delete po <operator pod for ifm> -n ${PROJECT_CPD_INST_OPERATORS}

3. Optional: If the PVC size is still 20Gi, delete the PVC, reinstall the model, and restart the watsonxaiifm operator pod. After the PVC is created from the operator and before it starts downloading the model content, update the PVC size to 40Gi from the UI.

Problem

If you deploy a custom foundation model on a cluster with MIG enablement and use multiple GPUs, your deployment might fail. You might receive the following error message:

Custom foundation model deployment failed with architecture 'mpt' because this architecture does not support parallel tensors. Provide the predefined hardware specification 'WX-S' or specify only one GPU in a custom hardware specification.

Cause

Deploying custom foundation models on MIG-enabled clusters with multiple GPUs is not supported for TGIS and vLLM runtimes.

Solution

For clusters that are enabled to use MIG, you must deploy your custom foundation models by using a single MIG partition only and use the vLLM runtime for deployment. You can configure the size of the MIG partition based on the GPU requirement of your custom foundation model.

You might receive the following response when you trying to prompt a globally deployed custom foundation model in the Prompt Lab:

As no document has been uploaded, there is nothing to summarize.