Watson OpenScale tutorial: Training, deploying, and monitoring a model

Data set overview

This tutorial uses the German credit risk data set, which consists of bank information from loan applicants. This data set describes each applicant by using 20 attributes, such as gender, sex loan duration, and loan purpose.
The data set is structured in rows and columns, and saved in a .csv file format.

You can view the data set in a text editor or spreadsheet program:

Adding the notebook

1. Open the Credit risk project and select the Asset tab.

2. Click New asset > Jupyter notebook editor. Enter Credit risk as the notebook name, then click From URL and paste the following URL: https://github.com/IBM/watson-openscale-samples/blob/main/Cloud Pak for Data/WML/notebooks/binary/spark/Watson OpenScale and Watson ML Engine.ipynb

3. Click Create.

Configuring and running the notebook

1. Open the Credit risk notebook.

2. Click the Edit icon to place the notebook in edit mode.

3. Under Configure credentials, complete the following fields:

   1. For WOS_CREDENTIALS, enter your Cloud Pak for Data instance url, username, and password.

   2. For WML_CREDENTIALS, enter your Cloud Pak for Data instance url, username, password, and version number. Leave instance_id as the pre-populated value.

   3. Follow these steps to locate DATABASE_CREDENTIALS and SCHEMA_NAME:

      1. From the navigation menu , click Services > Instances.
      2. Click the Overflow menu and open your Watson OpenScale instance.
      3. Select the Configure icon and ensure you are viewing the Database tab.
      4. For DATABASE_CREDENTIALS, enter the hostname, username, password, database, and port. If SSL is enabled, enter the ssl, sslmode, and certificate_base64 as well.
      5. For SCHEMA_NAME, enter the schema.

4. Under Save training data to Cloud Object Storage, you need to paste credentials from your Cloud Object Storage (COS). Visit IBM Cloud Object Storage connection to learn how to locate existing credentials or create new credentials.

1. Click Cell > Run All to run all of the cells in the notebook. You can also run the notebook cell by cell to explore each cell and its output.

2. Monitor the progress cell by cell. Notice that the asterisk "In [*]" changes to a number, for example, "In [1]" when a cell finishs. The notebook takes 1 - 3 minutes to complete.

3. Try these tips if you encounter any errors while running the notebook:

   • Click Kernel > Restart & Clear Output to restart the kernel, and then run the notebook again.
   • Verify that you copied and pasted credentials with no leading or trailing spaces.

You are now ready to evaluate the deployed model in Watson OpenScale.

Adding the deployment to the Watson OpenScale dashboard

1. Open your Watson OpenScale instance.
2. Click the Configure icon and then select Machine learning providers.
3. Click Add machine learning provider and edit the provider's name to Credit risk model.
4. Edit the Connection section and complete the following fields:
   • Service provider: Watson Machine Learning (V2)
   • Location: Local
   • Environment: Production
   • Deployment space: tutorial-space
5. Save your choices and a notification confirms the new connection.

You are now ready to configure evaluations for a model in the selected space.

Connecting to the deployment space

1. From the Insights tab, select Add to dashboard.
2. Select Credit risk - preproduction as the model location and click Next. Then, select German credit risk online as the deployed model.
3. Complete the following fields for storage type:
   • Storage types: System-managed
   • Data type: Numeric/categorical
   • Algorithm: Binary classification
4. Click View summary to review your selection. Ensure that your Feature columns include Age and Sex.
5. Click Save and continue to confirm your selection.

After you confirm your selection, you should be automatically directed to configure your deployment.

Configuring the deployment

In this section, upload the sample training file to train the Credit risk model.

1. Select Use manual setup as the configuration method and click Next.
2. From the Training data option drop-down, choose Upload file and upload the sample training data file. Then, select Comma (,) as the delimiter and click Next.
3. When prompted, confirm Age and Sex as the feature columns. Then, select any category as the label column and click Next. The label column represents the correct prediction (ground-truth) for each record.
4. Select the model output. The model output data contains the prediction generated by the deployed model.
5. Click View summary to review your selection. Then, click Finish to confirm your selection.

Because the notebook contained the metadata such as authentication credentials for IBM Cloud and the location of your resources in Cloud Object Storage, the rest of the deployment configuration is done automatically.

Running the evaluation and viewing the results

To run the Fairness evaluation:

1. Return to the dashboard and click Evaluate now for the deployment.
2. When you are prompted to add Test data for the evaluation, upload the sample test data file in CSV format that you downloaded as part of setup.
3. Click Evaluate to start the test.

The results of the evaluation show that the fairness test for age passed, but that the fairness test for sex failed, as the outcome for the monitored group (female) was below the fairness threshold set in relation to the outcome for the reference group (male).

The evaluation demonstrated that there could be bias in your model. The source of the bias could be from an insufficient number of records in your sample training data for the monitored group, or it could indicate a problem with your model.