AI governance tutorial: Build and deploy a model

This tutorial is the first in a series of two tutorials. Take this tutorial to build, deploy, and track a model with the AI governance use case. Your goal is to train a model to predict which applicants qualify for mortgages and then deploy the model for evaluation. You must also set up tracking for the model to document the model history and generate an explanation for its performance.

The story for the tutorial is that Golden Bank wants to expand its business by offering low-rate mortgage renewals for online applications. Online applications expand the bank’s customer reach and reduce the bank’s application processing costs. As a data scientist at Golden Bank, you must create a mortgage approval model that avoids unanticipated risk and treats all applicants fairly. You will run a Jupyter Notebook to build a model and automatically capture metadata that tracks the model in an AI Factsheet.

The following animated image provides a quick preview of what you’ll accomplish by the end of the second tutorial where you will use Watson OpenScale to configure and evaluate monitors for the deployed model to ensure that the model is accurate and treating all applicants fairly. Right-click the image and open it in a new tab to view a larger image.

Screenshots of the tutorial

Preview the tutorial

In this tutorial, you will complete these tasks:

Set up the prerequisites
Task 1: Configure IBM OpenPages Model Risk Governance
Task 2: Create a model use case in the model inventory
Task 3: Run the notebook to create the model
Task 4: Track the model lifecycle in the use case
Task 5: Deploy the model
Task 6: Perform model risk assessment

Watch Video Watch this video to preview the steps in this tutorial. There might be slight differences in the user interface shown in the video. The video is intended to be a companion to the written tutorial.

This video provides a visual method to learn the concepts and tasks in this documentation.

Try the tutorial

Expand each section to complete the task.

Tips for completing this tutorial

Here are some tips for successfully completing this tutorial.

Get help in the community

If you need help with this tutorial, you can ask a question or find an answer in the Cloud Pak for Data Community discussion forum.

Set up your browser windows

For the optimal experience completing this tutorial, open Cloud Pak for Data in one browser window, and keep this tutorial page open in another browser window to switch easily between the two applications. Consider arranging the two browser windows side-by-side to make it easier to follow along.

Side-by-side tutorial and UI

Tip: If you encounter a guided tour while completing this tutorial in the user interface, click Maybe later.

Set up the prerequisites

Required services, roles, and permissions

Base Premium Standard Unless otherwise noted, this information applies to all editions of IBM Knowledge Catalog.

The following prerequisites are required to complete this tutorial.

Access type	Description	Documentation
Services	- Watson Studio - Watson Machine Learning - Db2 Either the single service, watsonx.governance, or Individual services: - AI Factsheets - OpenPages - Watson OpenScale	- Watson Studio - Watson Machine Learning - Db2 - watsonx.governance service - AI Factsheets - IBM OpenPages - Watson OpenScale
Role	Data Scientist	- Predefined roles and permissions - Manage roles
Permissions	- Manage deployment spaces - Monitor deployment activity - Manage catalogs - Access governance artifacts - Administer platform	- Predefined roles and permissions - Manage roles
Additional access	- Editor access to Default Catalog - Admin access to the OpenPages instance - MRG - All Permissions access role assigned in OpenPages - Admin access to the Watson OpenScale instance - Completed setup for default instance of Watson OpenScale - Completed integration between AI Factsheets and OpenPages - Completed setup of IBM OpenPages Model Risk Governance	- Add collaborators - Add users for the OpenPages service - Assign and remove a role from a user or group - Manage users for the Watson OpenScale service - Automated setup - IBM OpenPages Model Risk Governance - Manage risk and compliance with Governance console
Additional configuration	Disable Enforce the exclusive use of secrets	Require users to use secrets for credentials

Verify your roles and permissions

Follow these steps to verify your roles and permissions. If your Cloud Pak for Data account does not meet all of the prerequisites, contact your administrator.

Click your profile image in the toolbar.
Click Profile and settings.
Select the Roles tab.

The permissions that are associated with your role (or roles) are listed in the Enabled permissions column. If you are a member of any user groups, you inherit the roles that are assigned to that group. These roles are also displayed on the Roles tab, and the group from which you inherit the role is specified in the User groups column. If the User groups column shows a dash, that means the role is assigned directly to you.

Roles and permissions

Create the sample project

If you did not already create the sample project for this tutorial, follow these steps:

Download the AI-governance.zip file.
From the Navigation Menu , choose Projects > All projects.
On the Projects page, click New project.
Select Local file.
Upload the previously downloaded ZIP file.
On the Create a project page, copy and paste the project name and add an optional description for the project.
```
AI governance
```
Click Create.
Click View new project to verify that the project and assets were created successfully.
Click the Assets tab to view the project's assets.

Check your progress

The following image shows the sample project. You are now ready to start the tutorial.

Task 1: Configure IBM OpenPages Model Risk Governance

This tutorial uses a model risk governance sample. Follow these steps to download the sample files and configure IBM OpenPages Model Risk Governance:

Download the following files:
From the home page, navigate to Services > Instances.
Click your OpenPages instance.
In the Access information section, click the Launch icon next to the URL. The OpenPages dashboard displays.
Import the first configuration:
1. In OpenPages, click the Settings icon .
2. From the menu, select System Migration > Import Configuration.
3. Select Local drive.
4. Click Add file, browse to select the MRG-Users-op-config.xml file, and then click Open.
5. Click Import, then click Submit.
Import the second configuration:
1. In OpenPages, click the Settings icon .
2. From the menu, select System Migration > Import Configuration.
3. Select Local drive.
4. Click Add file, browse to select the MRG-CP4D-trial-contents-op-config.xml file, and then click Open.
5. Click Import, then click Submit.
Import the Golden Bank trial content:
1. Click the Settings icon .
2. From the menu, select FastMap Import.
3. Click Import.
4. Click Choose file, select Golden_Bank_trial_content.xlsx file, and click Open.
5. Click Validate.
6. Click Import to begin the import.
7. When the import completes, close the window.

Task 2: Create the model use case in the model inventory

For this type of project, it is best to create the model use case when a project commences. A model use case can reference multiple machine learning models that you can use to solve business problems. Then, data engineers and model evaluators can add models to the model use case and track the model as it progresses through its lifecycle.

Task 2a: Create the inventory

Use cases are stored in an inventory. Follow these steps to create the inventory and enable OpenPages integration:

From the Navigation Menu , choose AI governance > AI use cases.
Click the Manage settings icon .
Click the Inventories page.
Click New inventory.
1. For the name, type:
```
Golden Bank Inventory
```
2. Clear the Add collaborators after creation option.
3. Click Create.
Click the General page.
Next to Governance Console (IBM OpenPages) integration, click the toggle icon .
1. In the Inventory field, select Golden Bank Inventory.
2. Accept the default values for the rest of the fields.
3. Click Apply.
Close the Manage page.

Check your progress

The following image shows the Governance Console enabled. You are now ready to create the model use case.

Task 2b: Create the model use case

Now you are ready to create the model use case so data scientists and model evaluators can add models to the model use case and track the model progress throughout it's lifecycle. Follow these steps to create the model use case:

Click New AI use case. The Governance Console displays.
For the Model use case name, copy and paste the name exactly as shown with no leading or trailing spaces:
```
Mortgage Approval Model Use Case
```

For the Description, copy and paste the following text:

This model use case is for the Mortgage approval model at Golden Bank.

For the Purpose, copy and paste the following text:

Assists with automating the process of issuing a mortgage to an applicant. Decide if the person should be given a mortgage or not.

For the Owner field, select your user name.
Add a primary business entity:
1. On the Primary Business Entity tab, click Add.
2. Search for catalogs, and select Catalogs (Library > MRG > WKC > Catalogs).
3. Click Done.
Add a second business entity:
1. Click the Other Business Entities tab.
2. Click Add.
3. Select Golden Bank.
4. Click Done.
Click Save.

Check your progress

The following image shows the Governance Console enabled. You are now ready to create the model use case.

Task 2c: Associate the workspaces with the use case

Next, associate the workspaces with the development and validation phases in the use case. You will use the sample project for the Develop phase. Before you can deploy the model, you need to promote the model to a deployment space in the Validate phase. Deployment spaces help you to organize supporting resources such as input data and environments; deploy models or functions to generate predictions or solutions; and view or edit deployment details.

Follow these steps to associate the workspaces with this use case:

Click the link in the Third Party Link field. If you don't see the link, refresh the page. This returns you to the AI use case in Cloud Pak for Data.
Scroll to the Associated workspaces section.
Under the Develop phase, click Associate workspace.
1. Select the AI governance project.
2. Click Save.
Under the Validate phase, click Associate workspace.
1. Click New space.
  1. For the deployment space name, copy and paste the name exactly as shown with no leading or trailing spaces:
```
Golden Bank Preproduction Space
```
  2. For the Deployment stage, select Testing.
  3. Click Create.
  4. Click Close.
2. Select the Golden Bank Preproduction Space from the list.
3. Click Save.

Check your progress

The following image shows your model use case. The model use case is now ready for data engineers and model evaluators to add models and track models as they progress through their lifecycle. The next task is to run the notebook to create the model.

Task 3: Run the notebook to create the model

Now you are ready to run the first notebook included in the sample project. The notebook includes the code to:

Set up AI Factsheets used to track the lifecycle of the model.
Load the training data, which is stored in the Db2 Warehouse connection in the sample project.
Specify the target, categorical, and numerical columns along with the thresholds used to build the model.
Build data pipelines.
Build machine learning models.
View the model results.
Save the model.

Follow these steps to run the notebook included in the sample project. Take some time to read through the comments in the notebook, which explain the code in each cell.

From the Navigation Menu , choose Projects > All projects.
Click the AI governance project name.
Click the Assets tab, and then navigate to Notebooks.

Note:
If you see if Caution icon next to the notebook, then Click the Overflow menu next to the 1-model-training-with-factsheets notebook, and choose Change environment. Select a supported Runtime on Python template, and click Change.
Click the Overflow menu for the 1-model-training-with-factsheets notebook, and choose Edit.
Click Run > Run All Cells to run all of the cells in the notebook. Alternatively, click the Run icon to run the notebook cell by cell if you want to explore each cell and its output.
The first cell requires your input.
1. At the Enter host name prompt, type your Cloud Pak for Data hostname beginning with https://, and press Enter. For example, https://mycpdcluster.mycompany.com.
2. At the Username prompt, type your Cloud Pak for Data username, and press Enter.
3. At the Password prompt, type your Cloud Pak for Data password, and press Enter.
The notebook takes 1 - 3 minutes to complete. You can monitor the progress cell by cell, noticing the asterisk "In [*]" changing to a number, for example, "In [1]".
If you encounter any errors during the notebook run, try these tips:
- Click Kernel > Restart & Clear Output to restart the kernel, and then run the notebook again.
- Verify that you created the model use case by copying and pasting the specified artifact name exactly with no leading or trailing spaces.

Check your progress

The following image shows the notebook when the run is complete. The notebook saved the model in the project, so you are now ready to view and add it to the model inventory.

Task 4: Trck the model lifecycle in the use case

Now you are ready to start tracking the model in the use case.

Task 4a: Track the model in the use case

After running all the cells in notebook, follow these steps to view the model's factsheet in the project and then associate that model with a model use case in the model inventory:

Click the AI governance project name in the navigation trail.
Click the Assets tab, and then navigate to Models.
Click the Mortgage Approval Prediction Model asset that was created by the notebook.
Review the AI Factsheet for your model. AI Factsheets capture model metadata across the model development lifecycle, facilitating subsequent enterprise validation or external regulation. AI Factsheets enables model validators and approvers to get an accurate, always up-to-date view of the model lifecycle details.
In the last task, you ran a notebook containing the AI Factsheets Python client code in the notebook that captured training metadata. Scroll to the Training metrics and Training tags sections to review the captured training metadata.
The following image shows the AI Factsheet for the model:
Scroll up on the model page, and click Track in AI use case.
1. Since this project is associated with a use case, notice that Mortgage Approval Model Use Case is selected.
2. Select Default approach, and click Next.
3. Select New asset record, and click Next.
4. Select Experimental, accept the default version number, and click Next.
5. Review the information, and click Track asset.
Back on the model page, note that the model is in the Develop phase.
Click View details icon .
On the use case page, click the Lifecycle tab to see the model tracking. AI Factsheets track models through their lifecycle. This model is still in the Development phase as it has not been deployed yet.

Check your progress

The following image shows the model use case with the model in the Development phase.

Task 4b: Set the model class and model status

Follow these steps to view the new model record in the Governance console, and set the model class and the model status:

Return to the Overview tab.
Click the Open in Governance Console link to open the same use case in the Governance console.
Under Associations > Models, click the Mortgage Approval Prediction Model name to view the model.
Set the model class:
1. Click the Admin tab.
2. Scroll to the Other fields section
3. Click the Edit icon next to the Model Class field.
4. Select Predictive.
5. Click Save.
Set the model status. Note that you can skip the first two steps of the model lifecycle since you already have a developed model.
1. On the Admin tab, click the Edit icon next to the Model Status field.
2. Select Development Complete.
3. Click Save.

Check your progress

The following image shows the model record with the model status set to Development Complete. Now you are ready to deploy the model.

Task 5: Deploy the model

Before you can deploy the model, you need to promote the model to a new deployment space. Deployment spaces help you to organize supporting resources such as input data and environments; deploy models or functions to generate predictions or solutions; and view or edit deployment details.

Task 5a: Promote the model to a deployment space

Follow these steps to promote the model to a new deployment space:

From the model record, click the link in the Third Party Link field. This returns you to the model in the AI governance project.
On the model page, click the Promote to deployment space icon .
For the Target space, select Golden Bank Preproduction Space.
Check the Go to model in the space after promoting it option.
Click Promote.

Check your progress

The following image shows the model in the deployment space. You are now ready to create a model deployment.

Task 5b: Create an online deployment for the model

Follow these steps to create an online deployment for your model:

When the deployment space opens, click New deployment.
1. For the Deployment type, select Online.
2. For the Name, copy and paste the deployment name exactly as shown with no leading or trailing spaces:
```
Mortgage Approval Model Deployment
```
3. For the Serving name, you can specify a descriptive name to use in place of the deployment ID that will help you to identify this deployment quickly. Copy and paste the serving name with no leading or trailing spaces. The name is validated to be unique per region. If this serving name already exists, then add a number (or any unique character) to the end of the serving name.
```
mortgage_approval_service
```
4. Click Create.
The model deployment may take several minutes to complete. When the model is deployed successfully, return to the model inventory; From the Navigation Menu , choose AI governance > AI use cases.
Select Mortgage Approval Model Use Case.
Click the Lifecycle tab. You can see that the model is now in the Validation phase.

Check your progress

The following image shows the model use case with the model in the Validation phase. Your model is now ready for you to evaluate in Watson OpenScale.

Task 6: Perform model risk assessment

The Model Owner can leverage captured metadata to consider carefully when approving this model as the implications can be large for the end customers and Golden Bank’s reputation. When the data scientist created a test deployment, all of the training time facts were captured. Now you perform a model risk assessment to determine if this model should move to a pre-production environment, and then eventually a production environment.

Return to the Overview tab.
Click the Open in Governance Console link to open the same use case in the Governance console.
Under Associations > Models, click the Mortgage Approval Prediction Model name to view the model.
Retrieve the training accuracy score:
1. Scroll to the Associations section.
2. On the Model Metrics tab, click the training_accuracy_score metric.
3. Copy the value for the training accuracy score for the model. You need this value for the risk assessment.
Return to the model tab to create the model risk assessment:
1. In the Risk and Validation Activity section, click New.
2. For the Description, type:
```
Initial risk assessment for Mortgage Approval model
```
3. Answer the following questions to assess the risk of the model.
  - Model uses other models outputs or feeds downstream models: Choose No since this model makes a decision on the loan application and does not depend on any other models.
  - Model Training Accuracy score: Paste the value for the training accuracy score metric that you copied earlier.
  - Is this model used in granting loans or mortgages?: Choose Yes since this is a mortgage approval model.
  - Is there information on protected groups in the training data?: Choose No since this model is trained with mortgage data that has been anonymized.
4. After you provide the information for all of the required fields, click Save.
When the risk assessment is complete, review the Computed tier. The risk assessment generates a computed risk tier for the model (Tier 3 for low risk, Tier 2 for medium risk, and Tier 1 for high risk). Note that you can override the computed tier if that’s appropriate.
Confirm the assessment.
1. Click Action > Confirm Assessment.
2. Click Contine.
Return to the model page on the Admin tab to assign an owner.
1. Click the Edit icon next to the Model Owner field.
2. Select your user name from the list.
3. Click Save.
Submit the model for validation and confirmation.
1. Click the Task tab.
2. Click Action > Submit Candidate for Confirmation, and then click Continue to send the model for confirmation.
Click the Activity to review the details.

Check your progress

The following image shows the Activity tab for the model in OpenPages.

As a data scientist at Golden Bank, you created a mortgage approval model by running a Jupyter Notebook that built the model and automatically captured metadata to track the model in an AI Factsheet. You then promoted the model to a deployment space, and deployed the model. You tracked all of the activity for the model by using IBM OpenPages.

Next steps

You are now ready to validate and monitor your deployed machine learning model to ensure it is working accurately and fairly. For this task, you will use Watson OpenScale. See the Test and validate the model tutorial.

Learn more

Parent topic: Use case tutorials