Known issues and limitations for Watson OpenScale

The following list contains the limitations and known issues for IBM Watson OpenScale.

Limitations

• Watson OpenScale does not support models where the data type of the model prediction is binary. You must change such models so that the data type of their prediction is a string or integer data type.
• Support for the XGBoost framework has the following limitations for classification problems: For binary classification, Watson OpenScale supports the binary:logistic logistic regression function with an output as a probability of True. For multiclass classification, Watson OpenScale supports the multi:softprob function where the result contains the predicted probability of each data point belonging to each class.
• Fairness and drift metrics are not supported for unstructured (image or text) data types.
• Having an equals sign (=) in the column name of a dataset causes an issue with explainability and generates the following error message: Error: An error occurred while computing feature importance. Do not use an equals sign (=) in a column name. It is not supported.

Known issues

Watson OpenScale has the following known issue:

There’s a difference between Spark and Python payload tables

The payload table for a Spark classification model is different from a Python one. An Apache Spark payload table has three columns for the predicted results (prediction, probability, and prediction_probability) while the Python payload table only has two columns (prediction and probability). For the Spark engine, the probability field receives an array, such as [0.1,0.9] as a string column. The prediction_probability field requires a numeric value, such as 0.9 and is most similar to the Python probability field.

Unexpected data type causes automatic payload logging to fail

If your model output includes a field with a probability value, it must be a vector. Otherwise, automatic payload scoring is disabled.

Limit on the number of features for a model

Scoring payloads for a model must fit within the maximum width allowed for the table created by payload logging in the datamart database (with some buffer for the internal-use columns that Watson OpenScale itself adds). In addition, apart from the width there is also a hard-coded limit of 1012 features.

The following table summarizes what this means for models with different sizes of features:

Table 1: Feature column limits

Because many models have features of mixed types, the following sample configurations can be used for planning purposes:

• For int64 or float64 or strings of length 64 or less, count as 64.
• For strings from 65 to 2048, count as 2048.
• For strings from 2048 to 32K, count as 32K.
• The total length of all features should be no more than ~900K.

Not all Db2 instances function identically

Watson OpenScale supports Db2 Warehouse add-on, Db2 Advanced Enterprise Server Edition add-on, as well as Db2 Enterprise Server Edition (v. 11.5.1 or later) installation that is accessible to the cluster. Be aware of the following limitation:

• Watson OpenScale requires a tablespace with a page size of 32k or larger.

Drift configuration errors prevent configuration of drift monitor

The flexibility of the model configuration screen can also lead to problems later on when you want to configure monitors, such as the drift detection monitor. Because you can choose the data types, you must ensure that your choices match the input schema of the model. The following error may occur if the prediction column type is not properly selected:

error: AIQDD2003E:
Message: "The {0} model predictions are different from class names in the {1} training data for the {2} subscription of the {3} datamart and the {4} service binding."

The following cases are the most-likely cause:

• The class label is of string type and modeling_role prediction is assigned to the prediction column as a double type because that is how the output data schema is defined.
• You select the prediction column of double type in the UI, which is not restricted.

Payload formats

For proper processing of payload analytics, Watson OpenScale does not support column names with double quotation marks (“) in the payload. This affects both scoring payload and feedback data in CSV and JSON formats.

Microsoft Azure ML Studio

• Of the two types of Azure Machine Learning web services, only the New type is supported by Watson OpenScale. The Classic type is not supported.

• Default input name must be used: In the Azure web service, the default input name is "input1". Currently, this field is mandated for Watson OpenScale and, if it is missing, Watson OpenScale will not work.

  If your Azure web service does not use the default name, change the input field name to "input1", then redeploy your web service and reconfigure your OpenScale machine learning provider settings.

• If calls to Microsoft Azure ML Studio to list the machine learning models causes the response to time out, for example when you have many web services, you must increase timeout values. You may need to work around this issue by changing the /etc/haproxy/haproxy.cfg configuration setting:

  • Log into the load balancer node and update /etc/haproxy/haproxy.cfg to set the client and server timeout from 1m to 5m:

      timeout client           5m
      timeout server           5m
    

  • Running systemctl restart haproxy to restart the HAProxy load balancer.

If you are using a different load balancer, other than HAProxy, you may need to adjust timeout values in a similar fashion. {: note}

• Of the two types of Azure Machine Learning web services, only the New type is supported by Watson OpenScale. The Classic type is not supported.

Amazon SageMaker

• BlazingText algorithm is not supported: The Amazon SageMaker BlazingText algorithm input payload format is not supported in the current release of Watson OpenScale.

Custom machine learning service instance

• The Watson OpenScale Python Client SDK does not currently have Explainability working for the Custom serve engine. This is because the Custom serve engine requires a numerical prediction in the response data, which is not included with the module script.

Browser support

The Watson OpenScale service tooling requires the same level of browser software as is required by IBM Cloud. See the IBM Cloud Prerequisites topic for details.

Missing Watson Machine Learning service or auto-setup fails

You find that there are missing Watson Machine Learning instances or, after you run the auto-setup, you see the message, An error occurred while setting up the demo environment. Please try again. Reason: 'resources'

No deployment details were returned for the deployment space because the user does not have access to that deployment space. Ensure that you have the correct permissions.

If there is no existing deployment space, the auto-setup creates a pre-production space called openscale-express-path-preprod-<instance-id> and a production space called openscale-express-path-<instance-id>. It then deploys GermanCreditRisk models into those spaces and proceeds to set up the monitors.

In the auto setup on IBM Cloud Pak for Data version 3.5.0, two preproduction models, GermanCreditRiskModelPreProdICP and GermanCreditRiskModelChallengerICP, are created in the space openscale-express-path-preprod-<instance-id>, and a production model GermanCreditRiskModelICP is created in the space openscale-express-path-<instance-id>. If the spaces already exist, they are used. If the user does not have permission to access those spaces, then this error occurs.

IBM Watson Machine Learning spaces must be shared with the IBM Cloud Pak for Data admin

To perform any of the following tasks, a user must add the IBM Cloud Pak for Data admin to the space:

• add an IBM Watson Machine Learning space to Watson OpenScale
• add a model and set up monitors for models from that space
• remove monitors for models from that space?

Uploading feedback data fails in production subscription after importing settings

After importing the settings from your pre-production space to your production space you might have problems uploading feedback data. This happens when the datatypes do not match precisely. (When you import settings, the feedback table references the payload table for its column types.) You can avoid this issue by making sure that the payload data has the most precise value type first. For example, you must prioritize a double datatype over an integer datatype.

Restarting pods causes loss of log files

Restarting pods resets logging in that pod. In many situations, where a restart is required, you must obtain logs before restarting. For more information about accessing logs, see Retrieving log files.

Image classification models size limit

For IBM Watson Machine Learning, scoring input for image classification models that are sent for payload logging cannot exceed 1 MB. To avoid time out issues, images must not exceed 100 x 100 x 3 pixels and must be sent sequentially so that the explanation for the second image is requested when the first one is completed.

Microsoft Azure Machine Learning Service

When running IBM Watson OpenScale for IBM Cloud Pak for Data, you may encounter issues where Watson OpenScale is not able to communicate with Azure Machine Learning Service, when it needs to invoke deployment scoring endpoints. Security tools that enforce your enterprise security policies, such as Symantec Blue Coat may prevent such access.

If you encounter errors that indicate scoring against Azure Machine Learning Service cannot be reached, such as receiving an HTTP Status Code of 403, check your enterprise security policies and ensure that the scoring URL is properly re-categorized with the tools used, as needed, to allow Watson OpenScale to properly access the scoring endpoints.

IBM SPSS Collaboration and Deployment Services (C&DS)

• Explainability support limited

  • Explainability is supported for binary models and for SPSS multiclass models that return probabilities for all classes.
  • Explainability is not supported for SPSS multiclass models that return only the winning class probability.

• IBM SPSS C&DS (Binary-type only) corrected records count may be wrong

  • For IBM SPSS Collaboration & Deployment Services binary subscriptions, the Corrected Records count might not be accurate when viewed using the View Transactions button.

Explainability is not enabled in model risk management production models when importing settings from a preproduction regression model

Edit one of the tiles in the Model details pane to enable the explainability for the production model deployment.

Missing feature values for deloyments that are not shown on Insights dashboard

After you attempt to view a transaction from a deployed model, an error appears. A problem with pagination hides some transactions from appearing. The following steps fix this problem. You must ensure that all the deployments are loaded onto the Insights dashboard before attempting to view transactions.

1. Go to the Watson OpenScale Insights dashboard.
2. Click Load more until all deployed model tiles are loaded. You should be able to scroll down to see that all the tiles are loaded and that the Load more option disappears.
3. Click the Explain a transaction icon.
4. From the drop-down, select a deployed model.
5. Click Explain to view a transaction ID.

Oracle compatibility mode not supported for Db2 Warehouse

The use of Oracle compatibility mode causes problems for Watson OpenScale. You might receive an error, such as “Drift archive could not be uploaded for service instance” if you attempt to use Db2 Warehouse with Oractly compatibility mode activated. To use Watson OpenScale with Db2 Warehouse, you must disable compatibility mode.

After upgrading, version 3.5.3 is displayed

After you upgrade from Watson OpenScale 3.5.3 to a later version, the instance version displays “Version 3.5.3” in the dashboard. This is a known issue and does not accurately reflect the version. Newly installed instances continue to reflect the correct version number.

Upgrading Watson OpenScale version 3.5.1 to version 3.5.10 fails

Your attempt to upgrade Watson OpenScale version 3.5.1 instances to version 3.5.10 might fail. This failure does not affect how your service functions and you can continue to use your existing instances with Watson OpenScale version 3.5.10.

Next steps

• Get started with the service.
• View the API Reference material.
• Contact IBM.