Integrating AI use cases with Governance console

If AI use cases are integrated with the Governance console, you can extend governance to include workflows and compliance monitoring.

You can extend the governance capabilities of AI Factsheets to include integration with IBM OpenPages. After you enable the integration, model use cases you create in IBM OpenPages are synced with corresponding AI use cases. You can customize the field groups that govern what facts display for each model use case or model.

Enabling integration and choosing facts to share

A service administrator can enable integration between AI Factsheets and OpenPages.

After you enable the integration, choose the facts to share between OpenPages and AI Factsheets.

Setting up access in Governance console

To set up access for users, do the following steps:

  1. Log in to Governance console.
  2. Open the Administration menu, and then click Users and Security > Users.
  3. Click the user.
  4. Click Allowed user profiles and select the watsonx-governance MRG Master profile.
  5. Click Role Assignments, and then click Assign Roles.
  6. Click Role Template and select one of the MRG role templates. For more information, see Role templates
Tip: You can also assign profiles and role templates to groups.

Creating a model use case

With the OpenPages integration setup complete, a user can create a model use case.

  1. From the Inventory section of the OpenPages navigation menu, click Model use cases.

  2. Click New to add an use case. The use case displays the fields that are selected during setup.

  3. Select primary business entity to specify the IBM Knowledge Catalog for storing the model use case.

    Note:

    If the selected business entity does not specify a IBM Knowledge Catalog, the model use case is stored in the Platform assets catalog.

  4. Save the changes.

New model use cases that are created in OpenPages are synced to the model inventory. These default fields are synced:

  • Status
  • Business Entity Name
  • Risk level
  • Purpose
  • Documentation

Additionally, optional fields that are added from the MRG-UserFact-ModelUseCase field group are synced to model_entry_user_op. Note that updates to the model use case, such as a change in status or a change in MRG-UserFact-ModelUseCase field values must be done from OpenPages, not from IBM Knowledge Catalog. After integration is enabled, you cannot create model use cases from the model inventory.

Important: Updates made to model use cases in AI Factsheets will not be synced to OpenPages. The model use case asset details will not be in sync between AI Factsheets and IBM OpenPages.

Model workflow

You can now create model objects and associate them with model use case objects in OpenPages. For example, you might create a model use case to request a model. A data scientist can then start the development work on the model in Watson Studio or external providers. You can track the progress in the model use case.

Tracking models and deployments

To track a model, the model must be associated, or registered with a model use case. Note that you can track external models in addition to Watson Machine Learning models. See Adding an external model to the model inventory.

  1. Open the model details page from a project or a space.

  2. Click Track this model.

  3. Open a Watson Machine Learning model or an external model.

  4. From the details page, click Track this model.

  5. Select an existing use case or create a new use case.

  6. Select an existing approach where you want to track the model, or create a new approach.

  7. Select a type of version that best matches the state for the model. The type affects the version number, which follows a major.minor.patch semantic versioning format. If this is the first model in the approach, choose from:

    • Experimental: the model is just starting in development and the input and output structure are likely to change. This option increments the patch number of the version.
    • Stable: the model is in production and you don't expect major changes in input or output. This option sets the major number of the version.
    • Custom: define your own starting point. Use custom to create a new version number, following the semantic versioning format of major.minor.patch.
    Note:

    If you are saving a version of an existing model, you can update the version using options for Patch, Minor, Major, or Custom version updates.

  8. Optionally provide a comment for the version.

  9. Select an existing OpenPages model record or create a new one.

  10. Once the model is associated with a model record, click Open in model inventory to view the model that is being tracked as part of the use case. Information about the approach and versions display in the model details panel.

Note: When a model deployment is created and tracked in the model use case, a corresponding model deployment object is created in OpenPages. After the model is registered to a model use case, you cannot turn off tracking. For example, you cannot remove the model from the use case if the workflow status of the associated model on the OpenPages side has advanced to the next state in the lifecycle.

Tracking updates to models

When a model associated with a model use case that is originated in OpenPages is updated, the following is tracked and synced between the model inventory and OpenPages:

  • Updates to pre-defined fields
  • Updates to custom fields
  • Metrics and values that are captured for the model in Watson Machine Learning or with a Jupyter notebook
  • Model object status
Note: Metrics captured at development time of model through notebook and the Watson Machine Learning metrics are synced to OpenPages. For each metric, there is a metric object and metric value object created in the OpenPages side. The metric object is associated as a child to the model object and the metric value object is associated to its corresponding metric as a child in OpenPages.

The model object status on the OpenPages side is reflected in the associated model in the model details page

These fields are synced from AI Factsheets:

  • Model Name
  • Description
  • Model Type
  • Input Type
  • Algorithm
  • Prediction Type
  • Software Specification
  • Training Data Information
  • Hyper Parameters
  • Hybrid pipeline
  • Number of features
  • Input schema
  • Output Schema
  • Training parameters
  • Training tags
  • Training metrics
  • Watson Machine Learning Metrics

The updates to the custom facts defined in modelfacts_user_op are synced to the OpenPages model object.

Note: If these fields are updated from the OpenPages side, they will not be synced to AI Factsheets and thus the data will not be in sync. Updates to the custom facts defined in `modelfacts_user` will not be synced to OpenPages model object.

Tracking updates to deployments

When a deployment that is associated with an OpenPages model use case is updated, the following is tracked and synced between the model inventory and OpenPages:

  • Status of the deployment
  • Updates to the deployment
  • Metrics and values that are collected as part of a Watson OpenScale evaluation
Note:

For each metric, a metric object and metric value object is created in the OpenPages side. The metric object is associated as a child to the model deployment object and the metric value object is associated to its corresponding metric as a child in OpenPages.

Customizing which fields to include

From OpenPages, you can choose which facts to share with AI Factsheets by customizing which fields are included for models or model use cases. From the OpenPages user interface, you can add new fields to the specific field groups. For details on adding fields in OpenPages, see Defining fields and adding them to field groups. Follow the OpenPages steps for including extra fields:

  1. On Settings > Solution Configuration > Object Types, choose Model use case or Model as the Object type to add the new fields for sharing with AI Factsheets Model use case and Watson Machine Learning models.
  2. Add the custom fields that you want to share with the Model use case in AI Factsheets to the field group MRG-UserFact-ModelUseCase. The model_entry_user_op asset type definition gets updated with the custom fields when Sync updates are performed.
  3. Add the custom fields that you want to share from Machine learning models to OpenPages to the field group MRG-UserFact-Model. The modelfacts_user_op asset type definition gets updated with the custom fields when Sync updates are performed.
Note:

Fields that you add on the OpenPages side must be of type String, Date, or Integer to sync with AI Factsheets.

Synching after customization

After adding fields in OpenPages, follow these steps to sync with AI Factsheets.

  1. Click the Manage tab in the model inventory.
  2. Click Check for updates to sync the OpenPages fields.

Customizing the view

You can configure the OpenPages view layout with Viewing information.

Associating a business entity with IBM Knowledge Catalog

If you want model use case objects that are created under the business entity of OpenPages to be synced to IBM Knowledge Catalog in a particular catalog, then you must specify the IBM Knowledge Catalog catalog_id in the field MRG-BusEnt:Catalog ID of a business entity from OpenPages. If the catalog ID is not specified, or the specified catalog ID does not exist, then model use cases are synced to Platform assets catalog by default.

To get the catalog ID:

  1. From Catalogs > View all catalogs, click to open the IBM Knowledge Catalog entity or create a new one.
  2. You can get the ID from the catalog URL.
  3. Update the Catalog ID field of the business entity in OpenPages.

Learn more

Parent topic: AI Factsheets