Known issues and limitations for Watson OpenScale

Limitations

• When you enable batch processing, Watson OpenScale has the following limitations:

  • Only support for Structured data
  • Only support for Production environments
  • Combinations of environments supported
  • Remote Spark Cluster + Non-kerberized Hive
  • Remote Spark Cluster + Kerberized Hive
  • Remote Spark Cluster + Db2
  • IAE + Non-kerberized Hive
  • IAE + Db2
  • IAE + Kerberized Hive
  • During an evaluation request, from the model summary screen, you might see an error on the Model Summary window that displays, “Evaluation for Quality/Drift monitor didn’t finish within 900 seconds.” Although you see the error, the actual monitor evaluation finishes to completion. If you encounter such an error, navigate back to the Insights dashboard, check if a quality or drift score is visible in the deployment tile, and then come back to the Model Summary window.
  • You must create a new volume and not use the default volume when you use Analytics Engine Powered by Apache Spark to prepare your deployment environment
  • You must install dependencies by using Python 3.9.x or higher and upload them to the mount path when you use Analytics Engine Powered by Apache Spark to prepare your deployment environment
  • In your Hive table, if there is a column, whether feature or not, that is named rawPrediction, configuring and evaluating the drift monitor fails.
  • If a column named probability is in your Hive table, and it is not configured with modeling-role probability, configuring and evaluating the drift monitor fails.
  • Pyspark ML, the framework that is used to build the drift detection model, does not entertain Boolean fields when the drift model is trained. The training table must have any boolean columns represented as string.
  • If the drift detection model was generated by running the configuration notebook against a Hadoop Cluster (Cluster A) that is different from the Hadoop cluster (Cluster B) that is used for monitoring, evaluating the drift monitor fails. To correct this problem, you must perform the following steps:
  1. Download the drift archive by using the notebook.
  2. Extract the contents of the drift archive to a folder.
  3. In a text editor, open the ddm_properties.json file.
  4. Look for the property drift_model_path. This property has the path where the drift model is stored in HDFS in this cluster.
  5. Download the folder in the drift_model_path to your local workstation.
  6. Copy this folder in an HDFS location /new/path in your production cluster.
  7. Update the property drift_model_path in the ddm_properties.json. The new property looks like the following sample: hdfs://production_cluster_host:port/new/path
  8. Compress the contents of the drift archive folder as a tar.gz file. Do not compress the folder itself, only the contents. All the files must be present at the top location and not inside a folder in the archive.

• When you configure settings for SHAP global explanations, Watson OpenScale has the following limitations:

  • The sample size that you use to configure explanations can affect the number of explanations that Watson OpenScale can generate during specific time periods. If you attempt to generate multiple explanations for large sample sizes, Watson OpenScale might fail to process your transactions.
  • If you configure explanations for multiple Watson OpenScale subscriptions, you must specify the default values for the sample size and number of perturbations settings when your deployment contains 20 features or less.

• When you want to specify a model endpoint to configure a Watson Machine Learning batch deployment, Watson OpenScale has the following limitations:

  • You must create a remote Watson Machine Learning provider.
  • You can't use the Watson Machine Learning deployment space that contains your online deployment to add a batch deployment.

• When configuring batch subscriptions, if the partition column name is changed for an existing table, Watson OpenScale doesn't validate the column if the name is not specified in the table. You must verify that the partition column name that you specify is also in the table. If the partition column isn't in the table, your monitor evaluations might fail or run incorrectly.

• After you create a new table in Watson OpenScale version 4.5 when you configure batch subscriptions, if you change the partition column name, Watson OpenScale does not add the new partition column.

• If you have a feature name that contains a period (.), the Chart Builder interface isn't displayed. Instead, you see a Failed to get data error message.

• Watson OpenScale does not support custom metric endpoints that are deployed on remote Cloud Pak for Data clusters. You can use a custom notebook to specify the token_info endpoint to generate a token that you can use to add a custom metric endpoint.

• Support for the zLinux platform has the following limitations:

  • Only scikit-learn, XGBoost frameworks, and Python functions are supported for IBM Watson Machine Learning.
  • Datamart databases must be created with a larger page size than the default value to work well with wide datasets, as shown in the following example:

  CREATE DB {db name} PAGESIZE {PAGESIZE integer} (8192 for 8K or more)
  

  • Hive tables that are created with the ORC format are not supported while monitoring batch subscriptions with IBM Analytics Engine and Hive.

• Watson OpenScale does not support models where the data type of the model prediction is binary. You must change such models so that the data type of their prediction is a string or integer data type.

• Drift is supported for structured data only.

• Although classification models support both data and accuracy drift, regression models support only data drift.

• Drift is not supported for Python functions.

• If the training of the drift detection model doesn't meet the quality standards, then model drift detection is disabled. If model drift detection is disabled, a drop in model accuracy can't be detected. In this scenario, no action is required.

• Support for the XGBoost framework has the following limitations for classification problems: For binary classification, Watson OpenScale supports the binary:logistic logistic regression function with an output as a probability of True. For multiclass classification, Watson OpenScale supports the multi:softprob function where the result contains the predicted probability of each data point belonging to each class.

• Fairness and drift metrics are not supported for unstructured (image or text) data types.

• Having an equals sign (=) in the column name of a dataset causes an issue with explainability and generates the following error message: Error: An error occurred while computing feature importance. Do not use an equals sign (=) in a column name. It is not supported.

• The maximum character limit for a service instance name is 41 characters.

• Support for scikit-learn 0.20 is deprecated with IBM Watson OpenScale version 4.0.2. When you upgrade to IBM Watson OpenScale version 4.0.2., if your existing drift detection model uses scikit-learn 0.20, the drift detection model stops working. If you have configured your IBM Watson OpenScale instance to detect the drift in accuracy and the drift in data, the drift in accuracy detection does not work. If you configured your IBM Watson OpenScale instance to detect only the drift in accuracy, the drift detection monitor does not work as drift is not measured for payload data. Additionally, the model monitor evaluation page does not show a configured drift monitor and you cannot view any past drift metrics. To avoid these limitations, you must retrain your drift detection model and reconfigure the drift detection monitor.

• If you are connecting to a Db2 database to import test data for model evaluations, you must specify uppercase column names in the input schema to correspond with the case-sensitive names in the database.

• If you upload test data for your preproduction model evaluations that exceeds the default maximum 10485760 bytes data size for the payload-logging-service-api pod, your upload might cause an error. To avoid this error, you must set the value for the -Dservice.defaults.import.max_csv_line_length option in the ADDITIONAL_JVM_OPTIONS environment variable to a larger size that fits your data set.

• For proper processing of payload analytics, Watson OpenScale does not support column names with double quotation marks (") in the payload. This affects both scoring payload and feedback data in CSV and JSON formats.

• Explainability is not supported for SPSS multiclass models that return only the winning class probability.

• For IBM Watson Machine Learning, scoring input for image classification models that are sent for payload logging cannot exceed 1 MB. To avoid time out issues, images must not exceed 100 x 100 x 3 pixels and must be sent sequentially so that the explanation for the second image is requested when the first one is completed.

• The Amazon SageMaker BlazingText algorithm input payload format is not supported in the current release of Watson OpenScale.

• The use of Oracle compatibility mode causes problems for Watson OpenScale. You might receive an error, such as "Drift archive could not be uploaded for service instance" if you attempt to use any supported databases with Oracle compatibility mode activated. To use Watson OpenScale with your database, you must disable compatibility mode.

• Scoring payloads for a model must fit within the maximum width allowed for the table created by payload logging in the datamart database (with some buffer for the internal-use columns that IBM Watson OpenScale itself adds). In addition, apart from the width there is also a hardcoded limit of 1012 features. Because many models have features of mixed types, the following sample configurations can be used for planning purposes:

  • For int64 or float64 or strings of length 64 or less, count as 64.
  • For strings from 65 to 2048, count as 2048.
  • For strings from 2048 to 32 K, count as 32 K.
  • The total length of all features should be no more than ~900 K.

Known issues

Watson OpenScale has the following known issues:

• Watson OpenScale etcd pod fails to start
• Watson OpenScale instance upgrade from version 4.0.x with OCS storage fails
• Attempt to create new service instance fails after upgrade
• Fairness evaluation of Remote Spark model fails
• Watson OpenScale instance does not display correct status after shutdown
• Watson OpenScale etcd cluster pods fail to come up in healthy state on AWS cluster with Portworx storage
• Watson OpenScale instance fails to open

Error loading public access token key

Detected implicit cartesian product for INNER join between logical plans....

instanceProjectName='cpd-instance'
instanceCRName='aiopenscale'
oc get WOService ${instanceCRName} -n ${instanceProjectName} -o jsonpath='{.status.wosStatus} {"\n"}'

cat <<EOF |oc apply -f -
apiVersion: wos.cpd.ibm.com/v1
kind: WOService
metadata:
  name: aiopenscale     # This is the recommended name, but you can change it
  namespace: project-name     # Replace with the project where you will install Watson OpenScale
spec:
  scaleConfig: small | medium   # The default value is `small`, but you can scale up to `medium`
  license:
    accept: true
    license: Standard | Enterprise     # Specify the license you purchased
  version: 4.0.6
  type: service
  storageClass: portworx-shared-gp3
  rwoStorageClass: portworx-metastoredb-sc
EOF