Known issues for Watson OpenScale
The following known issues apply to IBM Watson OpenScale.
Known issues
- Watson OpenScale instance fails to open
- Watson OpenScale instance fails to open when Iam integration is enabled
- Fairness evaluation of Remote Spark model fails
- Model scoring payload records fail to reach Watson OpenScale
Watson OpenScale instance fails to open
Your attempt to open a Watson OpenScale instance might fail due to the following error:
Error loading public access token key
To fix this issue, you must complete the following steps:
-
Log in to Red Hat OpenShift Container Platform with the following command:
oc login <OpenShift_URL>:<port>
-
Run the following command to force the Watson OpenScale operator to reconcile the Watson OpenScale custom resource:
instanceProjectName='cpd-instance' instanceCRName='aiopenscale' oc patch WOService ${instanceCRName} -n ${instanceProjectName} --type merge --patch '{"spec": {"ignoreForMaintenance": true}}' oc patch WOService ${instanceCRName} -n ${instanceProjectName} --type merge --patch '{"spec": {"ignoreForMaintenance": false}}'
If you did not install Cloud Pak for Data in the
cpd-instance
project or useaiopenscale
as the name of the Watson OpenScale custom resource, specify accurate values in theinstanceProjectName
andinstanceCRName
fields. -
Check the status of the reconcilliation with the following command:
oc get WOService ${instanceCRName} -n ${instanceProjectName} -o jsonpath='{.status.wosStatus} {"\n"}'
The status of the custom resources changes to
Completed
when the reconcilliation finishes successfully.
Watson OpenScale instance fails to open when Iam integration is enabled
When you enable Iam integration, your attempt to open a Watson OpenScale instance might fail due to the following error:
An instance of WatsonOpenScale was not found
To fix this issue, you must run the following commands in the Red Hat OpenShift Container platform after you install Watson OpenScale:
oc patch WOService openscale-defaultinstance --type merge --patch '{"spec": {"ignoreForMaintenance": true}}'
oc patch WOService openscale-defaultinstance --type merge --patch '{"spec": {"ignoreForMaintenance": false}}'
Fairness evaluation of Remote Spark model fails
When you run a fairness evaluation in Watson OpenScale on a model that uses Remote Spark version 3.1 or earlier, the evaluation might fail due to the following error:
Detected implicit cartesian product for INNER join between logical plans....
This error occurs when the Remote Spark environment misidentifies the join operation as a cartesian product operation. You can fix this issue by upgrading to Spark version 3.2 or later.
Model scoring payload records fail to reach Watson OpenScale
Applies to: 4.7.0 and later
When you upgrade Watson OpenScale to version 4.7 and later, an intermittent issue might prevent model scoring records from reaching Watson OpenScale due to the following error:
There are no new runtime records in the payload logging table since previous evaluation.
To fix this issue, you must complete the following steps:
- Log in to Red Hat OpenShift Container Platform with the following command:
oc login <OpenShift_URL>:<port>
- Restart the Watson Machine Learning scoring pods with the following command:
instanceProjectName='cpd-instance'
oc delete pods -n ${instanceProjectName} -l servicename=wml-scoring
If you did not install Cloud Pak for Data in the cpd-instance
project, specify accurate values in the instanceProjectName
field.
- If your Watson Machine Learning instance is tethered across multiple projects, restart the Watson Machine Learning scoring pods in all of the tethered projects as shown in the following example:
oc delete pods -n <DATAPLANE_PROJECT> -l servicename=wml-scoring
oc delete pods -n <TENANT_CONTROLER_PROJECT> -l servicename=wml-scoring
The commands delete pods in the DATAPLANE_PROJECT
and TENANT_CONTROLER_PROJECT
tethered projects.
- Verify that the Watson Machine Learning scoring pods restarted successfully.
Parent topic: Evaluating AI models with Watson OpenScale