Known issues and limitations for Watson Machine Learning

The following known issues and limitations apply to Watson Machine Learning.

Known issues

Limitations

Known issues for AutoAI

Importing an AutoAI notebook from a catalog can result in runtime error

Applies to: 5.2.0 (watsonx.ai 2.0 and later)

If you save an AutoAI notebook to an IBM Knowledge Catalog, and then you import it into a project and run it, you might get this error: Library not compatible or missing.

This error results from a mismatch between the runtime environment saved in the catalog and the runtime environment required to run the notebook in the project. To resolve, update the runtime environment to the latest supported version. For example, if the imported notebook uses Runtime 23.1 in the catalog version, update to Runtime 24.1 and run the notebook job again.

Tip: When you update your runtime environment, check that you have adequate computing resources. The recommended configuration is at least 2 vCPU and 8GB RAM for an experiment notebook, and at least 4 vCPU and 16GB RAM for a pipeline notebook.

Known issues for Watson Machine Learning

Upgrading Watson Machine Learning might fail because of runtime errors

Applies to: 5.2.0

When upgrading Watson Machine Learning from version 5.0.1 or 5.1.x to version 5.2.0 while on the s390x architecture, the upgrade might fail. This is because of a runtime error returning as undefined.

The following is an example of the error that can be seen in the Watson Machine Learning custom resource (CR):

Message: AnsibleUndefinedVariable: 'onnxruntime_opset_19_server_json' is undefined
The playbook has failed. See earlier output for exact error

Confirm the failure in the Watson Machine Learning CR
Run the following command to see the status of the wml-cr:

oc describe wmlbase wml-cr -n zen

The following is an exmaple output:

   Message:               AnsibleUndefinedVariable: 'onnxruntime_opset_19_multi_server_json' is undefined
The playbook has failed. See earlier output for exact error
    Reason:                Failed
    Status:                True
    Type:                  Failure
    Last Transition Time:  2025-06-04T06:12:13Z
    Message:               Running reconciliation
    Reason:                Running
    Status:                True
    Type:                  Running
  Progress:                5%
  Progress Message:        Finished Pre-Configuration
  Reconcile History:
    The last reconciliation was completed successfully.
  Versions:
    Reconciled:  5.1.2
  Wml Status:    InProgress
Events:          <none>

Workaround: To fix this issue, follow these steps:

Note: This workaround patch ensures the playbook includes all runtime JSON definitions needed during upgrade.

  1. Enable maintenance mode on the Watson Machine Learning CR:

    oc patch wmlbase wml-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' -n <cpd_instance_ns>

  2. Identify the Watson Machine Learning operator pod:

    oc get pods -n <cpd-operator-ns> | grep ibm-cpd-wml-operator

  3. Take note of the playbook upgrade file:

    • If you are upgrading from version 5.0.x, the upgrade playbook file willbe called upgrade50-s390x.yml.
    • If you are upgrading from version 5.1.x, the upgrade playbook file willbe called upgrade51-s390x.yml.

  4. Copy and edit the playbook upgrade file by entering the pod:

    oc rsh -n <cpd-operator-ns> <wml-operator-pod-name>

  5. While inside the pod, change directories to get to the playbook upgrade files:

    cd /opt/ansible/5.2.0/roles/wml-base/tasks

  6. Copy the playbook upgrade file to a location outside the pod:

    oc cp -n <cpd-operator-ns> <wml-operator-pod-name>:/opt/ansible/5.2.0/roles/wml-base/tasks/upgrade50-s390x.yml /tmp/upgrade50-s390x.yml

  7. Keep a backup of the playbook upgrade file:

    mv upgrade50-s390x.yml upgrade50-s390x.yml.org

  8. Edit the upgrade playbook file and update the list of json files under the section:

    name: Load runtime definitions for 
include_role:
name: "common"
tasks_from: load_runtime_definition.yml
loop:

    To add the following json files to the list:

    - { file_name: "files/onnxruntime_opset_19-server.json", var_name: "onnxruntime_opset_19_server_json" }
- { file_name: "files/onnxruntime_opset_19-multi-server.json", var_name: "onnxruntime_opset_19_multi_server_json" }

    Here is an example of the complete code block with the updated json list:

    - name: Load runtime definitions for 
include_role:
  name: "common"
  tasks_from: load_runtime_definition.yml
loop:
  - { file_name: "files/auto_ai.kb-server.json", var_name: "auto_ai_kb_server_json" }
  - { file_name: "files/auto_ai.ts-server.json", var_name: "auto_ai_ts_server_json" }
  - { file_name: "files/auto_rag-server.json", var_name: "auto_rag_server_json" }
  - { file_name: "files/training-job-server.json", var_name: "training_job_server_json" }
  - { file_name: "files/spss-modeler_batch-server.json", var_name: "spss_modeler_batch_server_json" }
  - { file_name: "files/spss-modeler_online-server.json", var_name: "spss_modeler_online_server_json" }
  - { file_name: "files/do-server.json", var_name: "do_server_json" }
  - { file_name: "files/wml-rshiny-server.json", var_name: "wml_rshiny_server_json" }
  - { file_name: "files/wml-rshiny-rstudio-24.1-r4.3-server.json", var_name: "wml_rshiny_rstudio_241_r43_server_json" }
  - { file_name: "files/spark-mllib_3.4-multi-server.json", var_name: "spark_mllib_34_multi_server_json" }
  - { file_name: "files/spark-mllib_3.5-multi-server.json", var_name: "spark_mllib_35_multi_server_json" }
  - { file_name: "files/pmml-3.0_4.3-multi-server.json", var_name: "pmml_30_43_multi_server_json" }
  - { file_name: "files/wx-cfm-job-0.1-server.json", var_name: "wx_cfm_job_01_server_json" }
  - { file_name: "files/wx-fm-deployment-job-1.0-server.json", var_name: "wx_fm_deployment_job_10_server_json" }
  - { file_name: "files/runtime-24.1-py3.11-server.json", var_name: "runtime_241_py311_server_json" }
  - { file_name: "files/runtime-24.1-py3.11-multi-server.json", var_name: "runtime_241_py311_multi_server_json" }
  - { file_name: "files/runtime-24.1-py3.11-cuda-server.json", var_name: "runtime_241_py311_cuda_server_json" }
  - { file_name: "files/tensorflow_rt24.1-py3.11-server.json", var_name: "tensorflow_rt241_py311_server_json" }
  - { file_name: "files/tensorflow_rt24.1-py3.11-dist-server.json", var_name: "tensorflow_rt241_py311_dist_server_json" }
  - { file_name: "files/tensorflow_rt24.1-py3.11-edt-server.json", var_name: "tensorflow_rt241_py311_edt_server_json" }
  - { file_name: "files/pytorch-onnx_rt24.1-py3.11-edt-multi-server.json", var_name: "pytorch_onnx_rt241_py311_edt_multi_server_json" }
  - { file_name: "files/pytorch-onnx_rt24.1-py3.11-edt-server.json", var_name: "pytorch_onnx_rt241_py311_edt_server_json" }
  - { file_name: "files/pytorch-onnx_rt24.1-py3.11-multi-server.json", var_name: "pytorch_onnx_rt241_py311_multi_server_json" }
  - { file_name: "files/pytorch-onnx_rt24.1-py3.11-server.json", var_name: "pytorch_onnx_rt241_py311_server_json" }
  - { file_name: "files/pytorch-onnx_rt24.1-py3.11-dist-multi-server.json", var_name: "pytorch_onnx_rt241_py311_dist_multi_server_json" }
  - { file_name: "files/pytorch-onnx_rt24.1-py3.11-dist-server.json", var_name: "pytorch_onnx_rt241_py311_dist_server_json" }
  - { file_name: "files/onnxruntime_opset_19-server.json", var_name: "onnxruntime_opset_19_server_json" }
  - { file_name: "files/onnxruntime_opset_19-multi-server.json", var_name: "onnxruntime_opset_19_multi_server_json" }
  - { file_name: "files/autoai-kb_rt24.1-py3.11-server.json", var_name: "autoai_kb_rt241_py311_server_json" }
  - { file_name: "files/autoai-ts_rt24.1-py3.11-server.json", var_name: "autoai_ts_rt241_py311_server_json" }
  - { file_name: "files/runtime-24.1-r4.3-server.json", var_name: "runtime_241_r43_server_json" }
  - { file_name: "files/dl-training-workload_rt24.1-py3.11-cuda-server.json", var_name: "dl_training_workload_rt241_py311_cuda_server_json" }
  - { file_name: "files/dl-training-workload_rt24.1-py3.11-server.json", var_name: "dl_training_workload_rt241_py311_server_json" }
  - { file_name: "files/training-job-go-server.json", var_name: "training_job_go_server_json" }
  - { file_name: "files/wml-hpo-job-server.json", var_name: "wml_hpo_job_server_json" }
  - { file_name: "files/pytorch-onnx_rt25.1-py3.12-edt-multi-server.json", var_name: "pytorch_onnx_rt251_py312_edt_multi_server_json" }
  - { file_name: "files/pytorch-onnx_rt25.1-py3.12-edt-server.json", var_name: "pytorch_onnx_rt251_py312_edt_server_json" }
  - { file_name: "files/pytorch-onnx_rt25.1-py3.12-multi-server.json", var_name: "pytorch_onnx_rt251_py312_multi_server_json" }
  - { file_name: "files/pytorch-onnx_rt25.1-py3.12-server.json", var_name: "pytorch_onnx_rt251_py312_server_json" }
  - { file_name: "files/pytorch-onnx_rt25.1-py3.12-dist-multi-server.json", var_name: "pytorch_onnx_rt251_py312_dist_multi_server_json" }
  - { file_name: "files/pytorch-onnx_rt25.1-py3.12-dist-server.json", var_name: "pytorch_onnx_rt251_py312_dist_server_json" }
  - { file_name: "files/tensorflow_rt25.1-py3.12-server.json", var_name: "tensorflow_rt251_py312_server_json" }
  - { file_name: "files/tensorflow_rt25.1-py3.12-dist-server.json", var_name: "tensorflow_rt251_py312_dist_server_json" }
  - { file_name: "files/tensorflow_rt25.1-py3.12-edt-server.json", var_name: "tensorflow_rt251_py312_edt_server_json" }
  - { file_name: "files/runtime-25.1-py3.12-server.json", var_name: "runtime_251_py312_server_json" }
  - { file_name: "files/runtime-25.1-py3.12-multi-server.json", var_name: "runtime_251_py312_multi_server_json" }
  - { file_name: "files/autoai-kb_rt25.1-py3.12-server.json", var_name: "autoai_kb_rt251_py312_server_json" }
  - { file_name: "files/autoai-ts_rt25.1-py3.12-server.json", var_name: "autoai_ts_rt251_py312_server_json" }
  - { file_name: "files/runtime-25.1-r4.4-server.json", var_name: "runtime_251_r44_server_json" }

  9. Push the updated file back into the pod:

    oc cp /tmp/upgrade50-s390x.yml <wml-operator-pod-name>:/opt/ansible/5.2.0/roles/wml-base/tasks/upgrade50-s390x_new.yml -n <cpd-operator-ns>

oc rsh -n <cpd-operator-ns> <wml-operator-pod-name>

cd /opt/ansible/5.2.0/roles/wml-base/tasks

  10. Rename the new modified file to the original:

    mv upgrade50-s390x_new.yml upgrade50-s390x.yml
chmod 777 upgrade50-s390x.yml
exit

  11. Turn off maintenance mode:

    oc patch wmlbase wml-cr --type merge --patch '{"spec": {"ignoreForMaintenance": false}}' -n <cpd_instance_ns>

Upgrading Watson Machine Learning might fail because of orphaned objects

Applies to: 5.2.0

Upgrading Watson Machine Learning, might fail because of Watson Machine Learning deployment objects which are orphaned.

If this issue is detected early on during the upgrade process, the pre-upgrade job will fail.

Confirm upgrade failure is due to orphaned objects
If you want to confirm that the upgrade failure is due to the orphaned objects, run the following steps:

  1. Fetch the pod to confirm the job run has failed:
    oc get pods -n <namespace> | grep wml-pre-upgrade-check
  2. Check the pod output for indications of orphaned deployments:
    oc logs <pod-name> | grep "Orphaned deployments found! Exiting with failure."
  3. If orphaned objects are detected, you will receive a detailed report outlining the orphaned objects found in the Watson Machine Learning environment. For example:
    2025/05/03 11:36:27,837|INFO|check_dep_orphans.py:250: Generating report.....
2025/05/03 11:36:27,841|INFO|check_dep_orphans.py:252: Below are the WML deployments orphaned objects found.
2025/05/03 11:36:27,843|INFO|check_dep_orphans.py:255: ------------------------------------------------
2025/05/03 11:36:27,844|INFO|check_dep_orphans.py:256:    space_id: 350d356e-fdf7-42da-9f6d-71a72ee77221
2025/05/03 11:36:27,845|INFO|check_dep_orphans.py:257: ------------------------------------------------
2025/05/03 11:36:27,846|INFO|check_dep_orphans.py:260: Deployments:
2025/05/03 11:36:27,849|INFO|check_dep_orphans.py:264:  e854d101-1c55-46c1-bb89-3589c27395ac {'name': 'base', 'sw_spec_id': '121c6a74-4ed4-5828-8b81-56ef47f3bc2f', 'type': 'base', 'model_id': 'fcb4c4d0-3587-4069-b58c-6fcec002384d'}
2025/05/03 11:36:27,850|INFO|check_dep_orphans.py:266: Models:
2025/05/03 11:36:27,851|INFO|check_dep_orphans.py:271:  f5742786-4546-43c5-853f-13758d04ee0a
2025/05/03 11:36:27,854|INFO|check_dep_orphans.py:273: Derived software specification:
2025/05/03 11:36:27,855|INFO|check_dep_orphans.py:278:  7274f418-19c9-4d9c-b13f-ba9b1411fc79 {'name': 'derived1_cw_spec-upgrade', 'base_sw_spec_id': '121c6a74-4ed4-5828-8b81-56ef47f3bc2f'}
2025/05/03 11:36:27,856|INFO|check_dep_orphans.py:280: Missing software specification:
2025/05/03 11:36:27,858|INFO|check_dep_orphans.py:285:  121c6a74-4ed4-5828-8b81-56ef47f3bc2f
27,896|ERROR|check_dep_orphans.py:448: Orphaned deployments found! Exiting with failure.
Error: /opt/ibm/scripts/check_dep_orphans.pyc rc=1
    Review the list of Watson Machine Learning deployment objects in the log report.

Workaround to clear orphaned objects
To clear the orphaned objects, run the following steps:

  1. You can delete orphaned objects directly by patching the Watson Machine Learning custom resource (CR) with the delete_orphan field:

    oc patch wmlbase wml-cr -n <namespace> --type=merge -p '{"spec":{"delete_orphan":true}}'

    After running the patch command, the Watson Machine Learning operator will reconcile and the job will re-run.

  2. When the job run is complete, you will see a message:

    oc logs <pod-name> | grep "Orphaned deployments deleted successfully. Exiting with success."

    For example:

    [root@api.xxxxxx.cp.fyre.ibm.com ~]# oc logs wml-pre-upgrade-check-kqpqs | grep "Orphaned deployments deleted successfully. Exiting with success."
2025/05/03 11:49:33,713|INFO|check_dep_orphans.py:445: Orphaned deployments deleted successfully. Exiting with success.

Unusable deployments after an upgrade or restoring from backup

Applies to: 5.2.0

For deployments created on Cloud Pak for Data, generating predictions with a deployment might fail after an upgrade. The error message for this problem is:

Deployment: <deployment-ID> has been suspended due to the deployment owner either not being a member of the deployment space: <space-ID> any more or removed from the system.

These errors can also occur following a restore from backup.

The resolution is to update the deployments by using the following steps. You must use alternative steps that are specific to R Shiny deployments.

To update deployments, except for R Shiny deployments:

  1. For HOST="CP4D_HOSTNAME", replace "CPD_HOSTNAME" with the Cloud Pak for Data hostname.

  2. For SPACE_ID="WML_SPACE_ID", replace "WML_SPACE_ID" with the space ID of the deployment that is failing.

  3. For DEPLOYMENT_ID="WML_DEPLOYMENT_ID" replace "WML_DEPLOYMENT_ID" with the deployment ID of the broken deployment.

  4. Use "Authorization: ZenApiKey <token>" and supply a valid token. If you export the environment variable use ${TOKEN} instead of <token>.

  5. Use this CURL command to replace the "OWNER_ID" with actual owner ID on this cluster in the PATCH payload.

    curl  -k -X PATCH "$HOST/ml/v4/deployments/$DEPLOYMENT_ID?version=2020-04-20&space_id=$SPACE_ID" -H "content-type: application/json" -H "Authorization: ZenApiKey <token>" --data '[{ "op": "replace", "path": "/metadata/owner", "value": "OWNER_ID" }]'
Note:

To run this script, you must generate and export the token as the ${TOKEN} environment variable. For details, see Generating an API authorization token.

To update R-Shiny deployments:

  1. Use oc get pods -n NAMESPACE | grep "wml-deployment-manager" and replace the NAMESPACE with WML Namespace.

  2. For oc exec -it WML_DEPLOYMENT_MANAGER_POD_NAME bash -n NAMESPACE, replace the WML_DEPLOYMENT_MANAGER_POD_NAME with the Deployment manager pod name displayed in the previous step and replace the NAMESPACE with the Watson Machine Learning namespace`.

  3. For deployment_id="DEPLOYMENT_ID", replace the DEPLOYMENT_ID with the deployment ID.

  4. For space_id="SPACE_ID", replace the SPACE_ID with the space ID for the deployment.

  5. For HOST="https://wml-deployment-manager-svc.NAMESPACE.svc:16500", replace the NAMESPACE with the Watson Machine Learning namespace`.

  6. Use "Authorization: ZenApiKey <token>" and supply a valid token. If you export the environment variable use ${TOKEN} instead of <token>.

  7. Re-create the R Shiny deployment using the following CURL command:

    curl -k -X PUT "$HOST/ml/v4_private/recreate_deployment/$deployment_id?version=2020-06-12&space_id=$space_id" -H "Authorization: ZenApiKey <token>"

  8. Verify the status of R Shiny deployment and wait for the deployment to become "Ready" before proceeding to the next step.

    curl -k -X GET "$HOST/ml/v4/deployments/$deployment_id?version=2020-06-12&space_id=$space_id" -H "Authorization: ZenApiKey ${TOKEN}"

  9. If you are upgrading to Cloud Pak for Data 4.8.0 or restoring from backup, scale up the number of copies by 1 from the deployment space UI.

    Scaling deployment copies

The deployment state will be changed from "Unusable" to "Deployed" state.

Restoring R Shiny deployment

Note:
  1. You can optionally scale the number of copies back to 1 or the original setting when the deployment is working as expected.
    2.To run this script, you must generate and export the token as the ${TOKEN} environment variable. For details, see Generating an API authorization token.

Decision Optimization deployment job fails with error: "Add deployment failed with deployment not finished within time"

Applies to: 5.2.0

If your decision optimization deployment job fails with the following error, complete the steps to extend the timeout window.

"status": {
     "completed_at": "2022-09-02T02:35:31.711Z",
     "failure": {
         "trace": "0c4c4308935a3c4f2d9987b22139c61c",
         "errors": [{
              "code": "add_deployment_failed_in_runtime",
              "message": "Add deployment failed with deployment not finished within time"
         }]
     },
     "state": "failed"
   }

To update the deployment timeout in the deployment manager:

  1. Edit the wmlbase wml-cr and add this line: ignoreForMaintenance: true. This sets the WML operator into maintenance mode, which stops automatic reconciliation. The automatic reconciliation will undo any configmap changes applied otherwise.

    oc patch wmlbase wml-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' -n <namespace>

    For example:

    oc patch wmlbase wml-cr --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' -n zen

  2. Capture the contents of the wmlruntimemanager configmap in a YAML file.

    oc get cm wmlruntimemanager -n <namespace> -o yaml > wmlruntimemanager.yaml

    For example:

    oc get cm wmlruntimemanager -n zen -o yaml > wmlruntimemanager.yaml

  3. Create a backup of the wmlruntimemanager YAML file.

    cp wmlruntimemanager.yaml wmlruntimemanager.yaml.bkp

  4. Open the wmlruntimemanager.yaml.

    vi wmlruntimemanager.yaml

  5. Navigate to file runtimeManager.conf and search for property service.

  6. Increase the number of retries in the retry_count field to extend the timeout window:

    service {

     jobs {

         do {
             check_deployment_status {
                 retry_count = 420   // Increase the number of retries to extend the timeout window }
                 retry_delay = 1000
             }
         }
     }

    Where:

    • Field retry_count = Number of retries
    • Field retry_delay = Delay between each retry in milliseconds

    In the example, the timeout is configured as 7 minutes (retry_count * retry_delay = 420 * 1000 = 7 minutes). If you want to increase the timeout further, you can increase the number of retries in the retry_count field.

  7. Apply the deployment manager configmap changes:

    oc delete -f wmlruntimemanager.yaml
oc create -f wmlruntimemanager.yaml

  8. Restart the deployment manager pods:

    oc get pods -n <namespace> | grep wml-deployment-manager

oc delete pod <podname> -n <namespace>

  9. Wait for the deployment manager pod to come up:

    oc get pods -n <namespace> | grep wml-deployment-manager
Note:

If you plan to upgrade the Cloud Pak for Data cluster, you must bring the WML operator out of maintenance mode by setting the field ignoreForMaintenance to false in wml-cr.

Configuring runtime definition for a specific GPU node fails

Applies to: 5.2.0

When you configure the runtime definition to use a specific GPU node with the nodeaffinity property, the runtime definition fails.

As a workaround, you must enable the MIG configuration for all GPU nodes if MIG is enabled for even a single GPU node. You must also use the Single profile type for all the GPU nodes. Mixed profiling is not supported. To learn more about single and mixed profiling strategies, see NVIDIA documentation.

A certification path error appears when users inference online deployments by using Java

Applies to: 5.2 and later

A certification path error might appear when you inference online deployments by using Java. See an example error:

The URL is not valid.
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Workaround:

  1. Open a web browser, paste the URL of the prediction endpoint into the address bar and then press Enter.

  2. Access security information from the address bar. In most browsers, you must click on the padlock icon. Then open certificate details.

  3. Export the certificate (for example, as cpd-cert.crt).

  4. Open your system's terminal or command-line tool and then set these environment variables:

    • $FILE_WITH_CERTS: use name of the file that contains the exported certificates
    • $CERT_ALIAS: create an alias for the new certificate

  5. Import the certificates by using keytool. See example code for Linux:

    keytool -import -trustcacerts -alias $CERT_ALIAS -file $FILE_WITH_CERTS -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit

  6. Verify that the certificate is properly added. See example Linux code:

    keytool -list -v -keystore $JAVA_HOME/lib/security/cacerts

  7. Compile and execute your code.

Inferencing of existing PMML and Tensorflow deployments fails after upgrade

Applies to: 5.3

Inferencing of existing PMML and Tensorflow deployments might fail after upgrading Watson Machine Learning from version 5.1 or 5.2. This issue is caused by a defect in the migration flow.

Workaround:

  1. Update the Watson Machine Learning custom resource:

    oc patch wmlbase wml-cr -n <PROJECT_CPD_INST_OPERANDS> --type=merge -p '{
  "spec": {
    "image_digests": {
      "wml_deployment_manager_image": "sha256:5fa3b2292616682f9720f2dcd75fbf36a61d8921c5f2209d7d8cbcabdfe3794b"
    }
  }
}'

  2. Run this code to check whether the Watson Machine Learning operator is reconciled:

    oc get wmlbase

    Expected output:

    NAME     VERSION   BUILD   STATUS      PERCENT   RECONCILED   AGE
wml-cr   5.3.0     330     Completed   100%      5.3.0        <time>

    The value of STATUS must be Completed and the value of PERCENT must be 100%.

  3. Get the name of the namespace where Watson Machine Learning is installed and export it as an environment variable:

    List custom Watson Machine Learning resources in all namespaces:

    oc get wmlbase --all-namespaces

    In the output, look for NAMESPACE value next to wml-cr. Example output:

    NAMESPACE        NAME       AGE
cpd-instance     wml-cr     10d

    Export the namespace as an environment variable

    export INSTANCE_NAMESPACE=<instance-namespace>

  4. If you are using a SOD configuration, get the name of your dataplane namespace and export it as an environment variable:

    Print the dataplane namespace:

    oc get cpdinstance --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{" -> "}{.spec.operand-deployment.namespace}{"\n"}{end}'

    Export the dataplane namespace as an environment variable:

    export DATAPLANE_NAMESPACE=<dataplane-namespace>

  5. Recreate the upgade job to use the updated image by applying this job template:

    Note:

    By default, the job template is configured with ARCH=x86_64. If you are running on ppc64le or s390x architecture, update the ARCH environment variable accordingly before applying the job. If you're not using a SOD configuration, remove the DATAPLANE_NAMESPACE from the job template.

     

    oc apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: wml-post-upgrade-and-rollback-hotfix
namespace: ${INSTANCE_NAMESPACE}
annotations:
   helm.sh/hook: pre-upgrade
   helm.sh/hook-delete-policy: before-hook-creation
   helm.sh/hook-weight: "1"
labels:
   app: wml-post-upgrade-and-rollback
   app.kubernetes.io/instance: ibm-wml-cpd
   app.kubernetes.io/managed-by: ansible
   app.kubernetes.io/name: ibm-wml-cpd
   chart: wml
   icpdsupport/addOnId: wml
   release: ibm-wml
spec:
backoffLimit: 5
completions: 1
parallelism: 1
completionMode: NonIndexed
podReplacementPolicy: TerminatingOrFailed
template:
   metadata:
      annotations:
      cloudpakId: eb9998dcc5d24e3eb5b6fb488f750fe2
      cloudpakInstanceId: 808f64fa-0fbc-471e-96ec-72d54e627126
      cloudpakName: IBM Cloud Pak for Data
      productChargedContainers: All
      productCloudpakRatio: "1:1"
      productID: eb9998dcc5d24e3eb5b6fb488f750fe2
      productName: IBM Watson Machine Learning
      productVersion: 5.3.0
      labels:
      app: wml-post-upgrade-and-rollback
      app.kubernetes.io/instance: ibm-wml-cpd
      app.kubernetes.io/managed-by: ansible
      app.kubernetes.io/name: ibm-wml-cpd
      batch.kubernetes.io/job-name: wml-post-upgrade-and-rollback-hotfix
      icpdsupport/addOnId: wml
      icpdsupport/app: api
      icpdsupport/module: wml-scoring
      job-name: wml-post-upgrade-and-rollback-hotfix
   spec:
      restartPolicy: Never
      serviceAccountName: zen-editor-sa
      imagePullSecrets:
      - name: ibm-entitlement-key
      terminationGracePeriodSeconds: 30
      affinity:
      nodeAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
                  - key: kubernetes.io/arch
                  operator: In
                  values:
                     - amd64
                     - ppc64le
                     - s390x
      securityContext:
      runAsNonRoot: true
      containers:
      - name: cleanup-hibernate
         image: cp.icr.io/cp/cpd/wml-post-upgrade-cleanup-deployments@sha256:755c278b896ac153067f4cb9706f1b177b81f5a03151c18e8b15a36e88a8fec7
         imagePullPolicy: Always
         command:
            - /bin/sh
            - /opt/ibm/scripts/run_upgrade_or_rollback.sh
         env:
            - name: DSX_CLIENT_DSI_MOUNT_PATH
            value: /opt/ibm/dsi/
            - name: VERSION
            value: "5.3.0"
            - name: WDP_BASE_URL
            valueFrom:
               configMapKeyRef:
                  name: wdp-config
                  key: host-url
            - name: ETCD_RETRY
            value: "3"
            - name: ETCD_RETRY_INTERVAL
            value: "5"
            - name: ONLINE_DEP_LIST_PATH
            value: /tmp/online_dep_list.txt
            - name: SCRIPT_ERR_PATH
            value: /tmp/errors.out
            - name: ARCH
            value: x86_64
            - name: SCC_SOD_COMPLIANT
            value: "true"
            - name: DATAPLANE_NAMESPACE
            value: ${DATAPLANE_NAMESPACE}
         resources:
            limits:
            cpu: 250m
            memory: 350Mi
            ephemeral-storage: 200Mi
            requests:
            cpu: 250m
            memory: 350Mi
            ephemeral-storage: 20Mi
         securityContext:
            runAsNonRoot: true
            capabilities:
            drop:
               - ALL
         volumeMounts:
            - name: tmp-volume
            mountPath: /tmp
            - name: wml-secrets
            mountPath: /opt/ibm/dsi
            - name: wml-manager-pvc-mount
            mountPath: /new-manager-pvc
            - name: old-manager-pvc-mount
            mountPath: /old-manager-pvc
      volumes:
      - name: tmp-volume
         emptyDir: {}
      - name: wml-secrets
         projected:
            defaultMode: 420
            sources:
            - secret:
                  name: internal-tls
                  items:
                  - key: ca.crt
                     path: server.cacrt
            - secret:
                  name: zen-service-broker-secret
            - secret:
                  name: wdp-service-id
            - secret:
                  name: wml-redis-cp-cr
      - name: wml-manager-pvc-mount
         persistentVolumeClaim:
            claimName: wml-manager-pvc
      - name: old-manager-pvc-mount
         emptyDir: {}
EOF

  6. Run this code to verify that the job completed successfully:

    oc get job wml-post-upgrade-and-rollback-hotfix -n ${INSTANCE_NAMESPACE}

    Make sure that the job status shows as Completed.

  7. Try inferencing the PMML and Tensorflow deployments that were failing before.

  8. Delete the hotfix job:

    oc delete job wml-post-upgrade-and-rollback-hotfix -n ${INSTANCE_NAMESPACE}

Limitations for AutoAI experiments

AutoAI file gets pushed to the Git repository in default Git projects

After you create an AutoAI experiment in a default Git project, you create a commit and see a file that includes your experiment name in the list of files that can be committed. There are no consequences to including this file in your commit. The AutoAI experiment will not appear in the asset list for any other user who pulls the file into their local clone using Git. Additionally, other users won’t be prevented from creating an AutoAI experiment with the same name.

Maximum number of feature columns in AutoAI experiments

The maximum number of feature columns for a classification or regression experiment is 5000.

Limitations for Watson Machine Learning

Deep Learning experiments with storage volumes in a Git enterprise project are not supported

If you create a Git project with assets in storage volumes, then create a Deep Learning experiment, running the experiment fails. This use case is not currently supported.

Deep Learning jobs are not supported on IBM Power (ppc64le) or Z (s390x) platforms

If you submit a Deep Learning training job on IBM Power (ppc64le) or Z (s390x) platform, the job fails with an InvalidImageName error. This is the expected behavior as Deep Learning jobs are not supported on IBM Power (ppc64le) or Z (s390x) platforms.

Deploying a model on an s390x cluster might require retraining

If you trained an AI model on a platform different than s390x, such as x86/ppc, and then you try to deploy the model on the s390x platform, such a deployment might fail and report an endianness issue: Argument shape does not agree with the input data. This happens if an older version of Pytorch (older than 2.1.2) was used to train the model (runtimes older than 24.1). To resolve the problem:

  • Retrain the model by using a runtime that contains a newer version of Pytorch on the x86/ppc platform and then deploy the model on the s390x platform
  • Retrain the AI model on the s390x platform and then deploy the model on the s390x platform

Limits on size of model deployments

Limits on the size of models you deploy with Watson Machine Learning depend on factors such as the model framework and type. In some instances, when you exceed a threshold, you will be notified with an error when you try to store a model in the Watson Machine Learning repository, for example: OverflowError: string longer than 2147483647 bytes. In other cases, the failure might be indicated by a more general error message, such as The service is experiencing some downstream errors, please re-try the request or There's no available attachment for the targeted asset. Any of these results indicate that you have exceeded the allowable size limits for that type of deployment.

Automatic mounting of storage volumes is not supported by online and batch deployments

You cannot use automatic mounts for storage volumes with Watson Machine Learning online and batch deployments. Watson Machine Learning does not support this feature for Python-based runtimes, including R-script, SPSS Modeler, Spark, and Decision Optimization. You can use only automatic mounts for storage volumes with Watson Machine Learning shiny app deployments and notebook runtimes.

As a workaround, you can use the download method from the Data assets library, which is a part of the ibm-watson-machine-learning python client.

Batch deployment jobs that use large inline payload might get stuck in starting or running state

If you provide a large asynchronous payload for your inline batch deployment, it can result in the runtime manager process to go out of heap memory.

In the following example, 92 MB of payload was passed inline to the batch deployment which resulted in the heap to go out of memory.

Uncaught error from thread [scoring-runtime-manager-akka.scoring-jobs-dispatcher-35] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[scoring-runtime-manager]
java.lang.OutOfMemoryError: Java heap space
   at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
   at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)
   at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)
   at java.base/java.lang.StringBuilder.append(StringBuilder.java:174)
   ...

This could result in concurrent jobs getting stuck in starting or running state. The starting state can only be cleared once the deployment is deleted and a new deployement is created. The running state can be cleared without deleting the deployment.

As a workaround, use data references instead of inline for huge payloads that are provided to batch deployments.

Setting environment variables in a conda yaml file does not work for deployments

Setting environment variables in a conda yaml file does not work for deployments. This means that you cannot override existing environment variables, for example LD_LIBRARY_PATH, when deploying assets in Watson Machine Learning.

Example:

variables:
  my_var: my_value

In this code, my_value will not be effective in a deployment's environment.

Workaround for online deployments:

If you're using a Python function, consider setting default parameters. For details, see Deploying Python functions.

Workaround for batch jobs:

For Python functions and Python scripts, if you're running batch jobs, use scoring.environment_variables in job's payload.

Example code that creates a batch deployment for a Python function by using the ibm-watsonx-ai SDK:

scoring_payload = {
    'input_data': [{
        'values': [[0]]
    }],
    "environment_variables" : {
        "my_var": "my_value",
    }
}
client.deployments.create_job(deployment_id, scoring_payload)

Jobs for batch deployments that use package extensions might fail

Jobs for batch deployments that use package extensions might fail with this error: WMLClientError: The product version <version> is not supported yet. This happens when a package extension downgrades ibm-watsonx-ai.