Configuring model evaluations with automatic setup

The automatic setup option for machine learning model evaluations sets up a machine learning environment, a database, and a sample model for you. Follow the steps in the guided tour to learn how to evaluate the sample model. After the setup is complete, you can add your own model to the dashboard.

Sample model

The automatic setup uses the sample data set German Credit Risk to demonstrate key features of model evaluations.

Overview of the sample data

The German Credit Risk sample data provides a collection of records for bank customers who were used to train the sample model. It contains 20 attributes for each loan applicant. The sample models provisioned as part of the automatic setup are trained to predict level of credit risk for new customers. Two of the attributes considered for the prediction - sex and age - can be tested for bias to make sure that outcomes are consistent regarding gender or age of customers.

To evaluate the outcomes, results are divided into groups. The Reference groups are the groups that are considered most likely to have positive outcomes. In this case, the Reference groups are male customers and customers over the age of 25. The Monitored groups are the groups that you want to review to ensure that the results do not differ greatly from the results for the monitored groups. In this case, the Monitored groups are females and customers aged 19 - 25.

Running the automatic setup

To quickly see how models are evaluated, run the demo scenario option that is provided when you first launch Watson OpenScale.

  1. From the Navigation Menu Navigation menu, choose Services > Instances.
  2. On the Instances page, for your Watson OpenScale instance, click the Overflow menu Overflow menu, and choose Open.
  3. When the Model evaluation page displays, click Auto setup.
  4. If prompted, select your Watson Machine Learning instance, and click Next. You must use the locally installed instance of Watson Machine Learning. There is no option for a remote instance.
  5. Provide either the Host name or IP address without the preceding https:// or final forward slash (/), Port, Database, Username, and Password for your Db2 database. For Db2 options that are part of your cluster, see Data Database where you find options, such as Db2 Warehouse and Db2 Advanced Enterprise Server Edition. For an external database, you can use IBM Db2 Database. Click Prepare.

As the model evaluation services are being configured, you can review the demo scenario that displays. When configuration is complete, choose whether to take a tour or exit to the dashboard.

  • To take the tour, click Start tour.
  • To exit the automatic setup and go to the dashboard, click Explore on my own.

Guided tour highlights

The guided tour demonstrates these features:

  1. Introduction to the user interface (UI): The four main areas of the UI include Insights, Explanations, Configuration, and Support.
  2. Monitoring and viewing results for the German credit risk model: Use predefined monitors to evaluate your model for fairness, quality, and drift. You can also use custom monitors for model evaluation.
  3. Exploring Fairness monitor: Use the Fairness monitor to looks for biased outcomes from your model. If a fairness issue is found, an alert is triggered based on configurable thresholds.
  4. Exploring data sets: Toggle between balanced, payload, training, and debiased data sets to see how they affect the fairness score of your model.
  5. Introduction to transactions: Review transactions from the payload data set for group bias and individual bias.
  6. Explaining model outcomes: Understand the features that led to the model prediction to build trust in the model. Additionally, learn how to change feature values to receive more favorable model outcomes.
  7. Exploring Drift monitor: Use the Drift monitor to determine if the processing of data in the model is causing a drop in accuracy.
  8. Reviewing transactions: Review the transactions list to investigate the drop in accuracy.

Touring a specific page

To use the automatic setup guided tour for a specific page, follow these steps:

  1. Open the page for which you would like to follow the guided tour.
  2. Open the Support tab and select Tour this page.

Resetting the tour

To reset the automatic setup tour, open the Support tab and select Reset auto setup.

Learn more

Parent topic: Setup options for model evaluations