Run a Python notebook to generate results in Watson OpenScale

In this tutorial, you learn to run a Python notebook to create, train, and deploy a machine learning model. Then, you create a data mart, configure performance, accuracy, and fairness monitors, and create data to monitor. Finally, you will be able to view results in the Watson OpenScale Insights tab.

Python client

The Watson OpenScale Python client is a Python library that you use to work directly with the Watson OpenScale service on IBM Cloud. You can use the Python client, instead of the Watson OpenScale client UI, to directly configure a logging database, bind your machine learning engine, and select and monitor deployments. For examples that use the Python client in this way, see the Watson OpenScale sample notebooks.

Scenario

Traditional lenders are under pressure to expand their digital portfolio of financial services to a larger and more diverse audience, which requires a new approach to credit risk modeling. Their data science teams currently rely on standard modeling techniques - like decision trees and logistic regression - which work well for moderate data sets, and make recommendations that can be easily explained. This approach satisfies regulatory requirements that credit lending decisions must be transparent and explainable.

To provide credit access to a wider and riskier population, applicant credit histories must expand. They go beyond traditional credit, such as mortgages and car loans, to alternate credit sources, such as utility and mobile phone plan payment histories, education, and job titles. These new data sources offer promise, but also introduce risk by increasing the likelihood of unexpected correlations, which introduce bias based on an applicant’s age, gender, or other personal traits.

The data science techniques that are most suited to these diverse data sets, such as gradient boosted trees and neural networks, can generate highly accurate risk models, but at a cost. Such models, without understanding the inner workings, generate opaque predictions that must somehow become transparent. You must ensure regulatory approval, such as Article 22 of the General Data Protection Regulation (GDPR) or the federal Fair Credit Reporting Act (FCRA) that is managed by the Consumer Financial Protection Bureau.

The credit risk model that is provided in this tutorial uses a training data set that contains 20 attributes about each loan applicant. Two of those attributes - age and sex - can be tested for bias. For this tutorial, the focus is on bias against sex and age. For more information about training data, see Why does Watson OpenScale need access to my training data?

Watson OpenScale monitors the deployed model’s propensity for a favorable outcome (“No Risk”) for one group (the Reference Group) over another (the Monitored Group). In this tutorial, the Monitored Group for sex is female, while the Monitored Group for age is 18 to 25.

Before you begin

This tutorial uses a Jupyter Notebook that must be run in a Watson Studio project, that uses a “Python 3.5 with Spark” runtime environment. It requires service credentials for the following IBM Cloud services:

When you choose a plan for each of the preceding Cloud Pak for Data services, there are several options, including a Lite plan. Although the Lite plan is free, it is restricted and provides only enough processing to run through the tutorial a couple of times before the monthly limit is reached.

Introduction

Use the Jupyter notebook to train, create and deploy a German credit risk model, configure Watson OpenScale to monitor that deployment, and provide seven days’ worth of historical records and measurements for viewing in the Watson OpenScale Insights dashboard.

Create and deploy a machine learning model

Add the Working with Watson Machine Learning notebook to your Watson Studio project

Edit and run the Working with Watson Machine Learning notebook

The Working with Watson Machine Learning notebook contains detailed instructions for each step in the Python code you run. As you work through the notebook, take some time to understand what each command is doing.

The net result is that you have created, trained, and deployed the Spark German Risk Deployment model to your Watson OpenScale service instance. Watson OpenScale is configured to check the model for bias against sex (in this case, Female) or age (in this case, 18-25 years old).

View results

View insights for your deployment

Using the Watson OpenScale dashboard, click the Insights Insights icon is displayed tab:

The Insights page provides an overview of metrics for your deployed models. You can easily see alerts for Fairness or Accuracy metrics that exceed the threshold set through the notebook. The data and settings that are used in this tutorial create Accuracy and Fairness metrics similar to the ones shown here.

Insight overview dashboard displays with a tile for the German Credit Risk model

View monitoring data for your deployment

  1. To view monitoring details, from the Insights page, click the tile that corresponds to the deployment. The monitoring data for that deployment appears.
  2. Slide the marker across the chart to select data for a specific one-hour window.
  3. Click the View details link.

Monitor data

Now, you can review the charts for the data you monitored. For this example, you can see that for the “Sex” feature, the group female received the favorable outcome “No Risk” less (68%) than the group male (78%).

Insight overview

View explainability for a model transaction

For each deployment, you can see explainability data for specific transactions.

If you already know which transaction you want to view, there’s a quick way to look it up with the transaction ID. After you click the deployment tile, from the navigator, click the Explain a transaction Explain a transaction tab icon, type the transaction ID, and press Enter.

If you use the internal lite version of PostgreSQL, you might not be able to retrieve your database credentials. This might prevent you from seeing transactions.

  1. From the charts for the latest biased data, click the View transactions button.

View transactions

A list of transactions where the deployment has acted in a biased manner appears.

  1. Select one of the transactions and, from the ACTION column, click the Explain link.

Transaction list

You will now see an explanation of how the model arrived at its conclusion, including how confident the model was, the factors that contributed to the confidence level, and the attributes fed to the model.

View Transaction

Next steps