Known issues and limitations for watsonx.governance

Error when tracking assets in use cases

Applies to: 5.4.0 and later

When tracking an asset in a use case, a user gets the following error:

ASTSV3108E: User '<ID>' with catalog role '[editor]' must be an editor or owner of the asset to perform the operation 'ASSET_ATTACHMENT_ATTRIBUTE_CREATE_UPDATE_DELETE'.

This error can occur when:

• The asset is in a project or a deployment space
• The user has the Editor role on the inventory

To resolve this issue, give the user Admin access to the inventory.

Complete setup fails with a 403 error

Applies to: 5.4.0 and later

When you click Complete setup, the process fails with a 403 error.

Alert notification emails contain broken links

Applies to: 5.4.0 and later

Alert notification emails contain links that point to Watson OpenScale instead of watsonx.governance. As a result, the links do not work.

Assets untracked when promoted or imported to a deployment space

Applies to: 5.4.0 and later

When you promote or import a tracked asset to a deployment space, the promoted asset is no longer tracked.

To resolve this issue, restart the wkc-factsheet-service pods by doing the following steps:

1. In IBM Software Hub, click Monitoring from the left navigation menu.
2. Select Pods.
3. Search for wkc-factsheet-service.
4. Restart the pods.
5. For assets that were untracked due to this known issue, manually retrack the assets.

Debiasing results show 0 for the corrected record count for SPSS Modeler subscriptions

Applies to: 5.4.0 and later

The debiasing measurement validation on the cluster differs from the reference value when tested by using the API. Also, the corrected record count is zero.

This issue occurs only in a binary classification model that is hosted in SPSS Modeler. All other validations for fairness measurements, both for Watson Machine Learning and non-Watson Machine Learning models, are populated as expected.

SHAP global explanations limitations

Applies to: 5.4.0 and later

When you configure SHAP global explanations, Watson OpenScale has the following limitations:

• Large sample sizes can cause failures when generating multiple explanations within short time periods.
• If you configure explanations for multiple subscriptions and your deployment contains 20 features or fewer, you must specify default values for the sample size and number of perturbations.

Binary prediction data types not supported

Applies to: 5.4.0 and later

Models with binary prediction data types are not supported. Change the prediction output to a string or integer data type.

XGBoost framework limitations

Applies to: 5.4.0 and later

Watson OpenScale supports the following XGBoost functions:

• For binary classification: binary:logistic with output as a probability of True.
• For multiclass classification: multi:softprob with output as predicted probabilities for each class.

Column names with equals sign (=) not supported

Applies to: 5.4.0 and later

Using an equals sign (=) in a column name causes explainability errors. Avoid using = in column names.

Test data upload size limit

Applies to: 5.4.0 and later

If your test data exceeds the default maximum size of 10,485,760 bytes, the upload might fail. To avoid this issue, increase the value of the -Dservice.defaults.import.max_csv_line_length option in the ADDITIONAL_JVM_OPTIONS environment variable.

Double quotation marks in column names not supported

Applies to: 5.4.0 and later

Watson OpenScale does not support column names with double quotation marks (") in scoring payloads or feedback data in CSV or JSON format.

Image classification payload size limit

Applies to: 5.4.0 and later

For IBM Watson Machine Learning, image classification scoring payloads must not exceed 1 MB. To avoid timeouts:

• Limit image size to 100 × 100 × 3 pixels.
• Send images sequentially, requesting explanations one at a time.

Amazon SageMaker BlazingText not supported

Applies to: 5.4.0 and later

The input payload format for the Amazon SageMaker BlazingText algorithm is not supported.

Scoring payload width and feature count limits

Applies to: 5.4.0 and later

Scoring payloads must fit within the maximum width allowed by the payload logging table in the datamart database. There is also a hardcoded limit of 1,012 features. Use the following guidelines to estimate feature size:

• int64, float64, or strings ≤ 64 characters: count as 64.
• Strings 65–2048 characters: count as 2048.
• Strings 2049–32,768 characters: count as 32K.
• Total feature length should not exceed ~900K.

Metadata synchronization issues with detached prompt templates and deployments

Applies to: 5.4.0 and later

When you make the following updates in a remote cluster, the updates are not synced to the primary governance cluster:

• Edits to the name, description, or tags of a detached prompt template
• Edits to the name or description of a deployment

Case sensitivity required for certain column names

Applies to: 5.4.0 and later

When you persist payload or feedback data using a scoring model deployment or the watsonx.governance REST API or SDK, the feature column names and other relevant fields must match the exact case used during model training.

For example, if a feature column is named LoanDuration, you must use the same casing when persisting data. Using variations such as loanduration or LOANDURATION may result in errors or unexpected behavior.

Deployment creation failure for tracked prompt template that references a tuned or custom foundation model

Applies to: 5.4.0 and later

When you create a prompt template for a tuned or custom model, track the prompt template in an AI use case, then promote the prompt template to a deployment space and create a deployment for the tracked prompt template, you might see the following error:

An error occurred patching the prompt template reference.

This error can occur when the associated model changed or was deleted.

To fix the issue, in the factsheet, click Automatically reconfigure. The reconfiguration refreshes the association so that you can proceed with creating a deployment and capturing the governance facts.

Metrics computed for a prompt template on payload and feedback data are not synced completely to Governance console

Applies to: 5.4.0 and later

This issue can occur when:

• You are tracking a prompt template in an AI use case that is synced with the Governance console.
• The metrics for an evaluation in a production space use both feedback and payload data.

The metric information that is computed on payload data is not synced to Governance console.

The following steps illustrate the problem.

1. Track a prompt template asset to a use case synced with Governance console.
2. Promote the prompt template to a production deployment space.
3. Create a new deployment for the prompt template.
4. Evaluate the prompt template, using both feedback and payload data. For example, evaluate the output for the Flesch readability score.
5. Review the factsheet for the results of the evaluation. You will see metric values for both payload and feedback data.
6. On the Governance console, the metrics show a value for the feedback data only. No result for the payload data displays.

Deleted space is not synced to the primary governance cluster

Applies to: 5.4.0 and later

If you are managing governance activities across multiple clusters with a custom connection, deleting a space does not sync deletion metadata to the primary governance cluster. To make sure model data is deleted, you must delete all the models in the space before you delete the space.

WML patch upgrade fails when WML is installed through watsonx.governance

Applies to: 5.4.0 and later

When Watson Machine Learning (WML) is installed as part of a watsonx.governance installation (driven by the watsonx_governance operator), running the apply-patch command upgrades only the WML operator. The WML custom resource (CR) version is not updated. Because watsonx.ai depends on WML, the upgraded watsonx.ai operator waits for WML to finish upgrading and the patch upgrade stalls.

Workaround: Manually patch the WML CR with the correct version and then let the operator reconcile:

1. List available patches:

   cpd-cli manage list-patch --instance_ns=${PROJECT_CPD_INST_OPERANDS}
   

2. Run case-download.
3. Patch the WML CR with the version for your target patch release. Get the correct version string from the official documentation for that release. For example, for patch 6:

   oc patch wmlbase wml-cr \
     -n ${PROJECT_CPD_INST_OPERANDS} \
     --type=merge \
     -p '{"spec":{"version":"5.3.4"}}'
   

4. After the CR is patched, continue with the apply-patch command:

   cpd-cli manage apply-patch --operator_ns=${PROJECT_CPD_INST_OPERATORS} --instance_ns=${PROJECT_CPD_INST_OPERANDS}
   

Installation fails because the OpenPages wait-for-encryption job remains stuck

Applies to: 5.4.0

During installation of watsonx.governance, the OpenPages wait-for-encryption job can remain stuck and prevent the installation from completing.

Workaround: Delete all OpenPages-related jobs and then delete the OpenPages operator pod. After the operator pod restarts, the OpenPages custom resource starts reconciling and the installation can complete.

Subscription removed when model is deleted after deployment replacement

When you replace the model that is associated with a deployment in watsonx.ai and then delete the model, the subscription is removed in watsonx.governance.

For example, if you replace Model A with Model B and then delete Model A, the subscription is removed in watsonx.governance.

Watsonx.data Presto connector is not supported in watsonx.governance

Watsonx.governance doesn't support evaluations that use data from a watsonx.data Presto connection in the Asset browser. This limitation occurs when you are evaluating assets in projects or deployment spaces.

Load the data into watsonx.governance by importing data files to your project or deployment space, or use a native connector, such as the Db2 connector.

External model governance in mult-cluster environments

If any cluster in a multi‑cluster environment is running an IBM Software Hub version earlier than 5.2.0, external model governance must be configured on the primary governance cluster.

Special characters are not supported

Special characters, such as é, are not supported. Errors can occur, for example, when you use special characters in the names of evaluations, models, or metrics. Replace special character with standard ASCII characters.

Cannot add both V1 and V2 use cases to a project or space

A project or space can have either V1 use cases or V2 use cases, not both.

A use case is "V1" if it was created prior to 5.2.0.

Embedding Drift unsupported with large content in Db2 Datamart

Embedding drift evaluation is not currently supported in IBM Db2 Datamart when working with large character data types at runtime.

Unsupported property for string meta-fields

In classification task types, meta-fields with the string data type do not support the expect_large_content property.

If you select a string-type meta-field column and enable support for large content, the system displays the following error message:

The property 'expect_large_content' is not supported for meta-fields in classification task type.

Watson OpenScale does not support Secrets Manager for database credentials

Watson OpenScale does not currently support retrieving credentials from Secrets Manager in IBM Cloud Databases for PostgreSQL. To configure the database connection, you must provide the credentials directly.

Attachments restricted by size

The file size of an attachment cannot exceed 210 MB.

Short text responses generate lower answer relevance scores

When your LLM model generates responses for retrieval augmented generation (RAG) tasks with short or single word answers to prompts, your prompt template evaluation might calculate answer relevance metric scores with lower values.

Scan files for malicious content

Files you upload are not automatically checked for malicious content. Before you upload a file, run a static scan against the file to ensure it does not contain malicious content.

Resource limitation on number of service providers

You cannot create more than 150 service providers in watsonx.governance due to a resource limitation.

If you attempt to create more than 150 service providers, the following error message appears:

 Failed to create service provider. Status code: 403, Error: {"errors":[{"code":"AIQCS0026E","message":"Quota exceeded on resource: service_provider","parameters":["service_provider"]}],"trace":"config-MmUwMjQ3M2MtM2QxNS00M2U5LTg5NzAtNzE3ZjEyNTcyZDgx"}

This error can also occur if you attempt to run evaluations in more than 150 projects and spaces in watsonx.governance. You can fix this error by removing the unused projects, spaces, or service providers.

Evaluation tab is restricted to specific platforms for different asset types

When you add assets in watsonx.governance deployment spaces or projects, the Evaluation tab might not be available if you don't specify the correct platform for your asset type.

You can view the Evaluations tab for generative AI assets only on the watsonx platform. You can view the tab for machine learning models only on the Cloud Pak for Data or watsonx platforms.