Known issues and limitations for Watson OpenScale

The following list contains the limitations and known issues for IBM Watson OpenScale.

Limitations

• When configuring batch subscriptions, if the partition column name is changed for an existing table, Watson OpenScale doesn't validate the column if the name is not specified in the table. You must verify that the partition column name that you specify is also in the table. If the partition column isn't in the table, your monitor evaluations might fail or run incorrectly.
• After you create a new table in Watson OpenScale version 4.5 when you configure batch subscriptions, if you change the partition column name, Watson OpenScale does not add the new partition column.

• Watson OpenScale does not support custom metric endpoints that are deployed on remote Cloud Pak for Data clusters. You can use a custom notebook to specify the token_info endpoint to generate a token that you can use to add a custom metric endpoint.

• Support for the zLinux platform has the following limitations:

  • Only scikit-learn, XGBoost frameworks, and Python functions are supported for IBM Watson Machine Learning.

  • Datamart databases must be created with a larger page size than the default value to work well with wide datasets, as shown in the following example:

    CREATE DB {db name} PAGESIZE {PAGESIZE integer} (8192 for 8K or more)
    

  • Hive tables that are created with the ORC format are not supported while monitoring batch subscriptions with IBM Analytics Engine and Hive.

• Watson OpenScale does not support models where the data type of the model prediction is binary. You must change such models so that the data type of their prediction is a string or integer data type.

• Drift is supported for structured data only.
• Although classification models support both data and accuracy drift, regression models support only data drift.

• Drift is not supported for Python functions.

• If the training of the drift detection model doesn't meet the quality standards, then model drift detection is disabled. If model drift detection is disabled, a drop in model accuracy can't be detected. In this scenario, no action is required.

• Support for the XGBoost framework has the following limitations for classification problems: For binary classification, Watson OpenScale supports the binary:logistic logistic regression function with an output as a probability of True. For multiclass classification, Watson OpenScale supports the multi:softprob function where the result contains the predicted probability of each data point belonging to each class.

• Fairness and drift metrics are not supported for unstructured (image or text) data types.

• Having an equals sign (=) in the column name of a dataset causes an issue with explainability and generates the following error message: Error: An error occurred while computing feature importance. Do not use an equals sign (=) in a column name. It is not supported.

• The maximum character limit for a service instance name is 41 characters.

• Support for scikit-learn 0.20 is deprecated with IBM Watson OpenScale version 4.0.2. When you upgrade to IBM Watson OpenScale version 4.0.2., if your existing drift detection model uses scikit-learn 0.20, the drift detection model stops working. If you have configured your IBM Watson OpenScale instance to detect the drift in accuracy and the drift in data, the drift in accuracy detection does not work. If you configured your IBM Watson OpenScale instance to only detect the drift in accuracy, the drift detection monitor does not work as drift is not measured for payload data. Additionally, the model monitor evaluation page does not show a configured drift monitor and you cannot view any past drift metrics. To avoid these limitations, you must retrain your drift detection model and reconfigure the drift detection monitor.

• If you are connecting to a Db2 database to import test data for model evaluations, you must specify uppercase column names in the input schema to correspond with the case-sensitive names in the database.

• If you upload test data for preproduction model evaluations that exceeds the default maximum 10485760 bytes data size for the payload-logging-service-api pod, your upload might cause an error. To avoid this error, you must set the value for the -Dservice.defaults.import.max_csv_line_length option in the ADDITIONAL_JVM_OPTIONS environment variable to a larger size that fits your data set.

• Explainability is not supported for SPSS multiclass models that return only the winning class probability.
• The Amazon SageMaker BlazingText algorithm input payload format is not supported in Watson OpenScale.
• For IBM Watson Machine Learning, scoring input for image classification models that are sent for payload logging cannot exceed 1 MB. To avoid time out issues, images must not exceed 100 x 100 x 3 pixels and must be sent sequentially so that the explanation for the second image is requested when the first one is completed.
• For proper processing of payload analytics, Watson OpenScale does not support column names with double quotation marks (") in the payload. This affects both scoring payload and feedback data in CSV and JSON formats.

Known issues

Watson OpenScale has the following known issues:

• Watson OpenScale upgrade from version 4.0.x to version 4.5.2 fails with a non-default custom resource name
• Watson OpenScale instance does not display correct status after shutdown
• Information missing because a feature name contains a period
• A section is missing from the user interface when partial drift detection is configured
• Limit on the number of features for a model
• Explainability is not enabled in model risk management production models when importing settings from a preproduction regression model
• Explainability is not enabled in model risk management production models when importing settings from a preproduction regression model
• Oracle compatibility mode not supported for Db2 Warehouse

Watson OpenScale upgrade from version 4.0.x to version 4.5.2 fails with a non-default custom resource name

If you don't use the default aiopenscale custom resource name when you install Watson OpenScale version 4.0.x, your attempt to upgrade to version 4.5.2 might fail. During the failed upgrade, the micro-serice pod from the Watson OpenScale dashboard doesn't start. You can use the following steps to fix this issue and complete the upgrade:

1. Log in to Red Hat OpenShift Container Platform with the following command:

   oc login <OpenShift_URL>:<port>
   

2. Pause the Watson OpenScale operator reconciliation of the Watson OpenScale custom resource.

   instanceProjectName='cpd-instance'
   instanceCRName='wos-cr-name' # default custom resource name is aiopenscale
   
   oc patch WOService ${instanceCRName} -n ${instanceProjectName}  --type merge --patch '{"spec": {"ignoreForMaintenance": true}}'
   

   If you did not install Cloud Pak for Data in the cpd-instance project, specify accurate values in the instanceProjectName field. Specify the name of the Watson OpenScale in the instanceCRName field.

3. Add a proxy Watson OpenScale Redis service to your existing Redis service:

   cat <<EOF |oc apply -f -
   apiVersion: v1
   kind: Service
   metadata:
   name: aiopenscale-ibm-aios-redis-master-svc
   namespace: $instanceProjectName
   spec:
   ports:
   - port: 6379
    protocol: TCP
    targetPort: 6379
    name: server-port
   - port: 26379
    protocol: TCP
    targetPort: 26379
    name: sentinel-port
   type: ClusterIP
   selector:
    release: $instanceCRName
    name:  $instanceCRName-ibm-aios-redis
    component: redis
    statefulset.kubernetes.io/pod-name: $instanceCRName-ibm-aios-redis-0
   EOF
   

4. Resume the Watson OpenScale operator reconciliation of the Watson OpenScale custom resource.

   oc patch WOService ${instanceCRName} -n ${instanceProjectName}  --type merge --patch '{"spec": {"ignoreForMaintenance": false}}'
   

5. Check the status of the upgrade with the following command:

   oc get WOService ${instanceCRName} -n ${instanceProjectName} -o jsonpath='{.status.wosStatus} {"\n"}'
   

   The status of the custom resource changes to Completed when the upgrade finishes successfully.

Watson OpenScale instance does not display correct status after shutdown

After you shut down a Watson OpenScale instance, the status of the instance displays as pending or failed on the Cloud Pak for Data service instance details page. The shutdown is still successful and you can use the following command to verify that the instance is not running:

instanceProjectName='cpd-instance'
instanceCRName='aiopenscale'
oc get WOService ${instanceCRName} -n ${instanceProjectName} -o jsonpath='{.status.wosStatus} {"\n"}'

If you did not install Cloud Pak for Data in the cpd-instance project or use aiopenscale as the name of the Watson OpenScale custom resource, specify accurate values in the instanceProjectName and instanceCRName fields.

The status of the instance displays as shutdown to confirm that the shutdown finished successfully.

Information missing because a feature name contains a period

If you have a feature name that contains a period (.), the Chart Builder interface isn't displayed. Instead, you see a Failed to get data error message.

A section is missing from the user interface when partial drift detection is configured

If you enable model drift and don't enable data drift, then the Drop in accuracy section doesn't appear in the right panel of the user interface. This issue occurs only if you intentionally configure partial drift detection in the notebook and drift archive.

Limit on the number of features for a model

Scoring payloads for a model must fit within the maximum width allowed for the table created by payload logging in the datamart database (with some buffer for the internal-use columns that Watson OpenScale itself adds). In addition, apart from the width there is also a hard-coded limit of 1012 features.

The following table summarizes what this means for models with different sizes of features:

Table 1: Feature column limits

Because many models have features of mixed types, the following sample configurations can be used for planning purposes:

• For int64 or float64 or strings of length 64 or less, count as 64.
• For strings from 65 to 2048, count as 2048.
• For strings from 2048 to 32K, count as 32K.
• The total length of all features should be no more than ~900K.

Explainability is not enabled in model risk management production models when importing settings from a preproduction regression model

Edit one of the tiles in the Model details pane to enable the explainability for the production model deployment.

Oracle compatibility mode not supported for Db2 Warehouse

The use of Oracle compatibility mode causes problems for Watson OpenScale. You might receive an error, such as "Drift archive could not be uploaded for service instance" if you attempt to use Db2 Warehouse with Oractly compatibility mode activated. To use Watson OpenScale with Db2 Warehouse, you must disable compatibility mode.

Next steps

• Get started with the service.
• View the API Reference material.
• Contact IBM.