Known issues and limitations for Watson Machine Learning

Known issues for Federated Learning

Authentication failures for Federated Learning training jobs when allowed IPs are specified in the Remote Training System

Applies to: 5.1.0 and later

Currently, the Red Hat OpenShift Ingress controller is not setting the X-Forwarded-For header with the client's IP address regardless of the forwardedHeaderPolicy setting. This causes authentication failures for Federated Learning training jobs when allowed_ips are specified in the Remote Training System even though the client IP address is correct.

To use the Federated Learning Remote Training System IP restriction feature in Cloud Pak for Data 4.0.3, configure an external proxy to inject the X-Forwarded-For header. For more information, see the article on configuring ingress.

Known issues for AutoAI

Importing an AutoAI notebook from a catalog can result in runtime error

Applies to: 5.1.0 and later (watsonx.ai 2.0 and later)

If you save an AutoAI notebook to an IBM Knowledge Catalog, and then you import it into a project and run it, you might get this error: Library not compatible or missing.

This error results from a mismatch between the runtime environment saved in the catalog and the runtime environment required to run the notebook in the project. To resolve, update the runtime environment to the latest supported version. For example, if the imported notebook uses Runtime 23.1 in the catalog version, update to Runtime 24.1 and run the notebook job again.

Known issues for Deep Learning

GPU profiles not displayed in hardware specification list for TensorFlow models with GPU inferencing enabled

Applied to: 5.1.2 and later

When you deploy a TensorFlow model for GPU inferencing, the GPU profiles are not included in the dropdown list of hardware specifications.

The root cause of the issue is that the software specification is not automatically updated when the model is promoted to the deployment space.

To work around this issue, you can modify the code to remove the ID when overriding the software specification for GPU, as shown here:

if (options.modelAttributes.useOnGpu && payloadFromFile.software_spec?.name) {
	delete payloadFromFile.software_spec.id
	payloadFromFile.software_spec.name = constants.SOFTWARE_SPEC_NAME_RT24_1_PY3_11_CUDA;
}

Known issues for Watson Machine Learning

Unusable deployments after an upgrade or restoring from backup

Applies to: 5.1.0 and later

For deployments created on Cloud Pak for Data, generating predictions with a deployment might fail after an upgrade. The error message for this problem is:

Deployment: <deployment-ID> has been suspended due to the deployment owner either not being a member of the deployment space: <space-ID> any more or removed from the system.

These errors can also occur following a restore from backup.

The resolution is to update the deployments by using the following steps. You must use alternative steps that are specific to R Shiny deployments.

To update deployments, except for R Shiny deployments:

1. For HOST="CP4D_HOSTNAME", replace "CPD_HOSTNAME" with the Cloud Pak for Data hostname.

2. For SPACE_ID="WML_SPACE_ID", replace "WML_SPACE_ID" with the space ID of the deployment that is failing.

3. For DEPLOYMENT_ID="WML_DEPLOYMENT_ID" replace "WML_DEPLOYMENT_ID" with the deployment ID of the broken deployment.

4. Use "Authorization: ZenApiKey <token>" and supply a valid token. If you export the environment variable use ${TOKEN} instead of <token>.

5. Use this CURL command to replace the "OWNER_ID" with actual owner ID on this cluster in the PATCH payload.

   curl  -k -X PATCH "$HOST/ml/v4/deployments/$DEPLOYMENT_ID?version=2020-04-20&space_id=$SPACE_ID" -H "content-type: application/json" -H "Authorization: ZenApiKey <token>" --data '[{ "op": "replace", "path": "/metadata/owner", "value": "OWNER_ID" }]'
   

To update R-Shiny deployments:

1. Use oc get pods -n NAMESPACE | grep "wml-deployment-manager" and replace the NAMESPACE with WML Namespace.

2. For oc exec -it WML_DEPLOYMENT_MANAGER_POD_NAME bash -n NAMESPACE, replace the WML_DEPLOYMENT_MANAGER_POD_NAME with the Deployment manager pod name displayed in the previous step and replace the NAMESPACE with the Watson Machine Learning namespace`.

3. For deployment_id="DEPLOYMENT_ID", replace the DEPLOYMENT_ID with the deployment ID.

4. For space_id="SPACE_ID", replace the SPACE_ID with the space ID for the deployment.

5. For HOST="https://wml-deployment-manager-svc.NAMESPACE.svc:16500", replace the NAMESPACE with the Watson Machine Learning namespace`.

6. Use "Authorization: ZenApiKey <token>" and supply a valid token. If you export the environment variable use ${TOKEN} instead of <token>.

7. Re-create the R Shiny deployment using the following CURL command:

   curl -k -X PUT "$HOST/ml/v4_private/recreate_deployment/$deployment_id?version=2020-06-12&space_id=$space_id" -H "Authorization: ZenApiKey <token>"
   

8. Verify the status of R Shiny deployment and wait for the deployment to become "Ready" before proceeding to the next step.

   curl -k -X GET "$HOST/ml/v4/deployments/$deployment_id?version=2020-06-12&space_id=$space_id" -H "Authorization: ZenApiKey ${MY_TOKEN}"
   

9. If you are upgrading to Cloud Pak for Data 4.8.0 or restoring from backup, scale up the number of copies by 1 from the deployment space UI.

   

The deployment state will be changed from "Unusable" to "Deployed" state.

Decision Optimization deployment job fails with error: "Add deployment failed with deployment not finished within time"

Applies to: 5.1.0 and later

If your decision optimization deployment job fails with the following error, complete the steps to extend the timeout window.

"status": {
     "completed_at": "2022-09-02T02:35:31.711Z",
     "failure": {
         "trace": "0c4c4308935a3c4f2d9987b22139c61c",
         "errors": [{
              "code": "add_deployment_failed_in_runtime",
              "message": "Add deployment failed with deployment not finished within time"
         }]
     },
     "state": "failed"
   }

To update the deployment timeout in the deployment manager:

1. Edit the wmlbase wml-cr and add this line: ignoreForMaintenance: true. This sets the WML operator into maintenance mode, which stops automatic reconciliation. The automatic reconciliation will undo any configmap changes applied otherwise.

   oc patch wmlbase wml-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' -n <namespace>
   

   For example:

   oc patch wmlbase wml-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' -n zen
   

2. Capture the contents of the wmlruntimemanager configmap in a YAML file.

   oc get cm wmlruntimemanager -n <namespace> -o yaml > wmlruntimemanager.yaml
   

   For example:

   oc get cm wmlruntimemanager -n zen -o yaml > wmlruntimemanager.yaml
   

3. Create a backup of the wmlruntimemanager YAML file.

   cp wmlruntimemanager.yaml wmlruntimemanager.yaml.bkp
   

4. Open the wmlruntimemanager.yaml.

   vi wmlruntimemanager.yaml
   

5. Navigate to file runtimeManager.conf and search for property service.

6. Increase the number of retries in the retry_count field to extend the timeout window:

   service {
   
        jobs {
   
            do {
                check_deployment_status {
                    retry_count = 420   // Increase the number of retries to extend the timeout window }
                    retry_delay = 1000
                }
            }
        }
   

   Where:

   • Field retry_count = Number of retries
   • Field retry_delay = Delay between each retry in milliseconds

   In the example, the timeout is configured as 7 minutes (retry_count * retry_delay = 420 * 1000 = 7 minutes). If you want to increase the timeout further, you can increase the number of retries in the retry_count field.

7. Apply the deployment manager configmap changes:

   oc delete -f wmlruntimemanager.yaml
   oc create -f wmlruntimemanager.yaml
   

8. Restart the deployment manager pods:

   oc get pods -n <namespace> | grep wml-deployment-manager
   
   oc delete pod <podname> -n <namespace>
   

9. Wait for the deployment manager pod to come up:

   oc get pods -n <namespace> | grep wml-deployment-manager
   

Configuring runtime definition for a specific GPU node fails

Applies to: 5.1.0 and later

When you configure the runtime definition to use a specific GPU node with the nodeaffinity property, the runtime definition fails.

As a workaround, you must enable the MIG configuration for all GPU nodes if MIG is enabled for even a single GPU node. You must also use the Single profile type for all the GPU nodes. Mixed profiling is not supported. To learn more about single and mixed profiling strategies, see NVIDIA documentation.

Online Backup with Netapp causes Watson Machine Learning to enter InMaintenance Mode

Applies to: 5.1.1

Problem description: After performing a Netapp backup, Watson Machine Learning enter the InMaintenance mode. You might receive the following message:

wml WmlBase wml-cr 2025-02-08T02:15:55Z zen 5.1.1 5.1.1 5.1.1-1625 100% Completed wml install/upgrade/restart The last reconciliation was completed successfully. InMaintenance

Root cause: The issue is caused by the pre-hooks and post-hooks configuration in the backup-meta, which puts the Watson Machine Learning CR into maintenance mode during the backup process. However, the Watson Machine Learning CR eventually reconciles and reaches the Completed state, but this may take longer than the default timeout value of 1800s.

Workaround: No changes are required to the configmap. If you encounter this issue, please wait a little longer (more than 1800s) for the Watson Machine Learning CR to reconcile and reach the Completed state. The Watson Machine Learning CR will automatically transition to the Completed state once the reconciliation is complete.

Hyperparameter tuning fails when using 2 parallel jobs

Applies to: 5.1.1

When running a hyperparameter tuning workload using 2 parallel jobs, the workload may fail.

Try running your hyperparameter tuning workload using a single job.

Limitations for AutoAI experiments

AutoAI file gets pushed to the Git repository in default Git projects

After you create an AutoAI experiment in a default Git project, you create a commit and see a file that includes your experiment name in the list of files that can be committed. There are no consequences to including this file in your commit. The AutoAI experiment will not appear in the asset list for any other user who pulls the file into their local clone using Git. Additionally, other users won’t be prevented from creating an AutoAI experiment with the same name.

Maximum number of feature columns in AutoAI experiments

The maximum number of feature columns for a classification or regression experiment is 5000.

No support for Cloud Pak for Data authentication with storage volume connection

You cannot use a storage volume connection with the 'Cloud Pak for Data authentication' option enabled as a data source in an AutoAI experiment. AutoAI does not currently support the user authentication token. Instead, disable the 'Cloud Pak for Data authentication' option in the storage volume connection to use the connection as a data source in your AutoAI experiment.

Limitations for Watson Machine Learning

Deep Learning experiments with storage volumes in a Git enterprise project are not supported

If you create a Git project with assets in storage volumes, then create a Deep Learning experiment, running the experiment fails. This use case is not currently supported.

Deep Learning jobs are not supported on IBM Power (ppc64le) or Z (s390x) platforms

If you submit a Deep Learning training job on IBM Power (ppc64le) or Z (s390x) platform, the job fails with an InvalidImageName error. This is the expected behavior as Deep Learning jobs are not supported on IBM Power (ppc64le) or Z (s390x) platforms.

Deploying a model on an s390x cluster might require retraining

Training an AI model on a different platform such as x86/ppc and deploying the AI model on s390x using Watson Machine Learning might fail because of an endianness issue. In such cases, retrain and deploy the existing AI model on the s390x platform to resolve the problem.

Limits on size of model deployments

Limits on the size of models you deploy with Watson Machine Learning depend on factors such as the model framework and type. In some instances, when you exceed a threshold, you will be notified with an error when you try to store a model in the Watson Machine Learning repository, for example: OverflowError: string longer than 2147483647 bytes. In other cases, the failure might be indicated by a more general error message, such as The service is experiencing some downstream errors, please re-try the request or There's no available attachment for the targeted asset. Any of these results indicate that you have exceeded the allowable size limits for that type of deployment.

Automatic mounting of storage volumes is not supported by online and batch deployments

You cannot use automatic mounts for storage volumes with Watson Machine Learning online and batch deployments. Watson Machine Learning does not support this feature for Python-based runtimes, including R-script, SPSS Modeler, Spark, and Decision Optimization. You can use only automatic mounts for storage volumes with Watson Machine Learning shiny app deployments and notebook runtimes.

As a workaround, you can use the download method from the Data assets library, which is a part of the ibm-watson-machine-learning python client.

Batch deployment jobs that use large inline payload might get stuck in starting or running state

If you provide a large asynchronous payload for your inline batch deployment, it can result in the runtime manager process to go out of heap memory.

In the following example, 92 MB of payload was passed inline to the batch deployment which resulted in the heap to go out of memory.

Uncaught error from thread [scoring-runtime-manager-akka.scoring-jobs-dispatcher-35] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[scoring-runtime-manager]
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
	at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
	at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:174)
   ...

This could result in concurrent jobs getting stuck in starting or running state. The starting state can only be cleared once the deployment is deleted and a new deployement is created. The running state can be cleared without deleting the deployment.

As a workaround, use data references instead of inline for huge payloads that are provided to batch deployments.

Setting environment variables in a conda yaml file does not work for deployments

Setting environment variables in a conda yaml file does not work for deployments. This means that you cannot override existing environment variables, for example LD_LIBRARY_PATH, when deploying assets in Watson Machine Learning.

As a workaround, if you're using a Python function, consider setting default parameters. For details, see Deploying Python functions.

Deploying assets on IBM Z and LinuxONE fails

Deploying assets on IBM Z and LinuxONE fails because Watson Machine Learning for Cloud Pak for Data version 5.1.1 does not support deployments on the s390x architecture.

Hyperparameter tuning runs with 2 parallel jobs

A hyperparameter (HPO) tuning job will run with a maximum of 2 parallel jobs, even if you set the max_parallel_job_num parameter of hyper_parameters_optimization in training_reference to a value larger than 2.

Parent topic: Service issues