Drift detection
Watson OpenScale detects both drift in accuracy and drift in data.
-
Estimates the drop in accuracy of the model at runtime. Model accuracy drops if there is an increase in transactions that are similar to those that the model did not evaluate correctly in the training data. This type of drift is calculated for structured binary and multi-class classification models only.
-
Estimates the drop in consistency of the data at runtime as compared to the characteristics of the data at training time.
A drop in either model accuracy or data consistency lead to a negative impact on the business outcomes that are associated with the model and must be addressed by retraining the model.
Drift analysis
Depending on whether you perform drift detection as part of batch processing or not, the drift displays are different.
Drift visualization for non-batch processing data
The drift visualization includes both graphical and numeric statistical data. By clicking the chart, you can display specific transactions that contribute to drift. The top reasons for detected drift display and includes a natural-language description of the observation as well as a list of unexpected values.
Specifically, from the Select a transaction set from the chart or list below section, you can choose the following views:
- Transactions responsible for drop in accuracy
- Transactions responsible for drop in accuracy and data consistency
- Transactions responsible for drop in data consistency
- Drift transactions are available in the transaction details screen, where you can click Explain to understand how a specific transaction has made it into the drift category.
Drift analysis for batch processing data
For the large quantity of data that can be produced by the batch processor, you receive a count of records that contribute to a drop in accuracy, a drop in data consistency, and both. In addition to this summary display, you can run a specialized analysis notebook: Notebook for analyzing payload transactions causing drift.
Enabling notifications
To send email notifications, click the Share the recommendations button. To enable this feature, you must first connect to an SMTP server that is configured in IBM Cloud Pak for Data. For more information, see Enabling email notifications. (IBM Watson OpenScale for IBM Cloud Pak for Data only.)
Limitations
The following limitations apply to the drift monitor:
- Drift is supported for structured data only.
- Although classification models support both data and accuracy drift, regression models support only data drift.
- Drift is not supported for Python functions.
Some questions and answers about drift
Why an error “Training complete with errors” is shown on the UI when configuring drift?
It is because your drift model is partially configured. For more information, read the message that is shown on the UI by clicking the information icon in Drift Model tile.
What are the different kinds of drift that IBM Watson OpenScale detects?
Watson OpenScale detects both drift in model accuracy and drift in data.
What is model accuracy drift?
Watson OpenScale estimates the drop in accuracy of the model at run time. Model accuracy drops if there is an increase in transactions that are similar to those that the model did not evaluate correctly in the training data.
This type of drift is calculated for structured binary and multi-class classification models only.
What is data drift?
Watson OpenScale estimates the drop in consistency of the data at runtime as compared to the characteristics of the data at training time. This drop in consistency of data is also termed as data drift. This type of drift is calculated for all structured models.
Why should one be concerned about model accuracy drift or data drift?
A drop in either model accuracy or data consistency leads to a negative impact on the business outcomes that are associated with the model and must be addressed by retraining the model.
Does Watson OpenScale detect drift in accuracy and drift in data?
Watson OpenScale detects both drift in accuracy and drift in data:
- Drift in accuracy estimates the drop in accuracy of the model at run time. Model accuracy drops when there is an increase in transactions that are similar to those that the model did not evaluate correctly in the training data.
- This type of drift is calculated for structured binary and multi-class classification models only. Whereas, drift in data estimates the drop in consistency of the data at runtime as compared to the characteristics of the data at training time.
Are there any limitations for the drift monitor in IBM Watson OpenScale?
The following limitations apply to the drift monitor:
• Drift is supported for structured data only. • Although classification models support both data and accuracy drift, regression models support only data drift. • Drift is not supported for Python functions.
How is drop in accuracy that is, model accuracy drift calculated in Watson OpenScale?
Watson OpenScale learns the behavior of the model by creating a proxy model, also known as a drift detection model. It looks at the training data and how the model is making predictions on the training data.
For more information, see Drift detection.
How is the drop in data consistency calculated in IBM Watson OpenScale?
IBM Watson OpenScale learns single and two-column constraints or boundaries on the training data at the time of configuration. It then analyzes all payload transactions to determine which transactions are causing drop in data consistency. For more information, see .
https://cloud.ibm.com/docs/ai-openscale?topic=ai-openscale-behavior-anomalies#behavior-anomalies-works
Can Watson OpenScale detect drift in my classification model?
Yes, Watson OpenScale can detect both drop in model accuracy and drop in data consistency for structured classification models.
Can Watson OpenScale detect drift in my regression model?
Watson OpenScale can detect a drop in data consistency only for structured regression models.
Can Watson OpenScale detect drift in my model that is trained on text corpus?
Watson OpenScale cannot detect drift in text-based models as of now.
Can Watson OpenScale detect drift in my model that is trained on image data?
Watson OpenScale cannot detect drift in image-based models as of now.
Can Watson OpenScale detect drift in my Python function that is deployed on IBM Watson Machine Learning?
Watson OpenScale cannot detect drift in Python functions as of now.
Go further
Read about a scenario that uses drift: