IBM Watson Machine Learning frameworks
You can use IBM Watson Machine Learning to perform payload logging, feedback logging, and to measure performance accuracy, runtime bias detection, drift detection, explainability, and auto-debias function in the IBM Watson OpenScale service.
IBM Watson OpenScale fully supports the following IBM Watson Machine Learning frameworks:
Table 1. Framework support details
Framework | Problem type | Data type |
---|---|---|
AutoAI1 | Classification (binary and multi classes) | Structured (data, text) |
AutoAI | Regression | Structured or Unstructured2 (text only) |
Apache Spark MLlib | Classification | Structured or Unstructured2 (text only) |
Apache Spark MLLib | Regression | Structured or Unstructured2 (text only) |
Keras with TensorFlow3&4 | Classification | Unstructured2 (image, text) |
Keras with TensorFlow3&4 | Regression | Unstructured2 (image, text) |
Python function | Classification | Structured (data, text) |
Python function | Regression | Structured (data, text) |
scikit-learn5 | Classification | Structured (data, text) |
scikit-learn | Regression | Structured (data, text) |
XGBoost6 | Classification | Structured (data, text) |
XGBoost | Regression | Structured (data, text) |
1To learn more about AutoAI, see AutoAI implementation details. For models where the training data is in Cloud Object Storage, there is no support for fairness attributes of type Boolean. However, if the training data is in Db2, Watson OpenScale supports fairness attributes that are Boolean type. When using the AutoAI option, Watson OpenScale does not support models when the model prediction is a binary data type. You must change such models so that the data type of their prediction is a string data type.
2Fairness and drift metrics are not supported for unstructured (image or text) data types.
3Keras support does not include support for fairness.
4Explainability is supported if your model / framework outputs prediction probabilities.
5To generate the drift detection model, you must use scikit-learn version 0.24.1 in notebooks.
6For XGBoost binary and multiple class models, you must update the model to return prediction probability in the form of numerical values for binary models and a list of probabilities per class for multi-class models. Support for the
XGBoost framework has the following limitations for classification problems: For binary classification, Watson OpenScale supports the binary:logistic
logistic regression function with an output as a probability of True
.
For multiclass classification, Watson OpenScale supports the multi:softprob
function where the result contains the predicted probability of each data point belonging to each class.
XGBoost subscriptions limitation for native XGBoost models
With the latest version WML, the xgboost_0.82
framework is deprecated. The supported framework is xgboost_0.90
, which must be used in combination with Python 3.7. To enable and persist XGBoost version 0.90 and Python
3.7, you must patch subscriptions by running the following command:
PATCH /v2/subscriptions/<subscription_id>
For example, the following command provides the full URL and subscription ID:
PATCH https://namespace.apps.server.company.com/openscale/00000000-0000-0000-0000-000000000000/v2/subscriptions/18bbe3f9-c550-44e7-b948-675bafcb1c63
The expected payload is returned:
[{
"op": "replace",
"path": "\/asset\/runtime_environment",
"value": "xgboost_0.90"
}]
After the subscription is successfully patched and scoring is done, the output data schema will have right modeling role set:
{
"metadata": {
"modeling_role": "class_probability"
},
"name": "prediction",
"nullable": true,
"type": "double"
}, {
"metadata": {},
"name": "probability",
"nullable": true,
"type": "double"
}, {
"metadata": {
"modeling_role": "prediction-probability"
},
"name": "prediction_probability",
"nullable": true,
"type": "double"
}, {
"metadata": {
"modeling_role": "prediction"
},
"name": "scoring_prediction",
"nullable": true,
"type": "integer"
}
AutoAI models and training data
AutoAI automatically prepares data, applies algorithms, or estimators, and builds model pipelines that are best suited for your data and use case. Watson OpenScale requires access to the training data to analyze the model.
Because Watson OpenScale cannot detect the training data location for an AutoAI model like it can for a regular model, you must explicitly provide the needed details to access the training data location:
- For the online path, where you manually configuring monitors, you must provide the database details from which training data can be accessed.
- For the custom notebook path, where you upload training data distribution, you can use the JSON file that is produced by running the notebook.
For more information, see Provide model details.
Specifying an IBM Watson Machine Learning service instance
Your first step in the Watson OpenScale tool is to specify an IBM Watson Machine Learning instance. Your Machine Learning instance is where you store your AI models and deployments.
Prerequisites
You should have provisioned an IBM Watson Machine Learning instance in the same account or cluster where the Watson OpenScale service instance is present. If you have provisioned a IBM Watson Machine Learning instance in some other account or cluster, then you cannot configure that instance with automatic payload logging with Watson OpenScale.
Connect your Machine Learning service instance
Watson OpenScale connects to AI models and deployments in an IBM Watson Machine Learning instance. To connect your service to Watson OpenScale, go to the Configure tab, add a machine learning provider, and click the Edit
icon. In addition to a name and description and whether this is a Pre-production or Production environment type, you must provide the following information that is specific to this type of service instance:
-
If you have an instance of IBM Watson Machine Learning, Watson OpenScale detects it, along with the configuration information.
-
To connect to IBM Watson Machine Learning on a separate IBM Cloud Pak for Data instance, provide a URL, username, and API key.
-
To connect to IBM Watson Machine Learning on IBM Cloud, provide a cloud region and API key.
IBM Cloud Pak for Data allows only one API key. When you generate a new API key, the previous API key is automatically revoked. If you update the API key in IBM Cloud Pak for Data, you must also update the API key in Watson OpenScale.
Limitations
The following limitations apply when you connect to IBM Watson Machine Learning on a separate IBM Cloud Pak for Data instance:
- The API key must have administrator access to the deployment space in IBM Watson Machine Learning.
- Automatic payload logging isn't available. Send payload logging by using the Watson OpenScale data set API.
The following limitations apply when you connect to IBM Watson Machine Learning on IBM Cloud:
- The API key must have administrator access to the deployment space in IBM Watson Machine Learning.
- Automatic payload logging isn't available. Send payload logging by using the Watson OpenScale data set API.
- During evaluation, load on your IBM Watson Machine Learning models in IBM Cloud increases.
Parent topic: Supported machine learning engines, frameworks, and models