Known issues for Watson OpenScale

The following known issues apply to IBM Watson OpenScale.

Known issues

Watson OpenScale instance fails to open

Your attempt to open a Watson OpenScale instance might fail due to the following error:

Error loading public access token key

To fix this issue, you must complete the following steps:

  1. Log in to Red Hat OpenShift Container Platform with the following command:

    oc login <OpenShift_URL>:<port>
    
  2. Run the following command to force the Watson OpenScale operator to reconcile the Watson OpenScale custom resource:

    instanceProjectName='cpd-instance'
    instanceCRName='aiopenscale'
    
    oc patch WOService ${instanceCRName} -n ${instanceProjectName}  --type merge --patch '{"spec": {"ignoreForMaintenance": true}}'
    oc patch WOService ${instanceCRName} -n ${instanceProjectName}  --type merge --patch '{"spec": {"ignoreForMaintenance": false}}'
    

    If you did not install Cloud Pak for Data in the cpd-instance project or use aiopenscale as the name of the Watson OpenScale custom resource, specify accurate values in the instanceProjectName and instanceCRName fields.

  3. Check the status of the reconcilliation with the following command:

    oc get WOService ${instanceCRName} -n ${instanceProjectName} -o jsonpath='{.status.wosStatus} {"\n"}'
    

    The status of the custom resources changes to Completed when the reconcilliation finishes successfully.

Watson OpenScale instance fails to open when Iam integration is enabled

When you enable Iam integration, your attempt to open a Watson OpenScale instance might fail due to the following error:

An instance of WatsonOpenScale was not found

To fix this issue, you must run the following commands in the Red Hat OpenShift Container platform after you install Watson OpenScale:

oc patch WOService openscale-defaultinstance --type merge --patch '{"spec": {"ignoreForMaintenance": true}}'
oc patch WOService openscale-defaultinstance --type merge --patch '{"spec": {"ignoreForMaintenance": false}}'

Fairness evaluation of Remote Spark model fails

When you run a fairness evaluation in Watson OpenScale on a model that uses Remote Spark version 3.1 or earlier, the evaluation might fail due to the following error:

Detected implicit cartesian product for INNER join between logical plans....

This error occurs when the Remote Spark environment misidentifies the join operation as a cartesian product operation. You can fix this issue by upgrading to Spark version 3.2 or later.

Model scoring payload records fail to reach Watson OpenScale

Applies to: 4.7.0 and later

When you upgrade Watson OpenScale to version 4.7 and later, an intermittent issue might prevent model scoring records from reaching Watson OpenScale due to the following error:

There are no new runtime records in the payload logging table since previous evaluation.

To fix this issue, you must complete the following steps:

  1. Log in to Red Hat OpenShift Container Platform with the following command:
oc login <OpenShift_URL>:<port>
  1. Restart the Watson Machine Learning scoring pods with the following command:
instanceProjectName='cpd-instance'
oc delete pods -n ${instanceProjectName} -l servicename=wml-scoring

If you did not install Cloud Pak for Data in the cpd-instance project, specify accurate values in the instanceProjectName field.

  1. If your Watson Machine Learning instance is tethered across multiple projects, restart the Watson Machine Learning scoring pods in all of the tethered projects as shown in the following example:
oc delete pods -n <DATAPLANE_PROJECT> -l servicename=wml-scoring
oc delete pods -n <TENANT_CONTROLER_PROJECT> -l servicename=wml-scoring

The commands delete pods in the DATAPLANE_PROJECT and TENANT_CONTROLER_PROJECT tethered projects.

  1. Verify that the Watson Machine Learning scoring pods restarted successfully.

Parent topic: Evaluating AI models with Watson OpenScale