Integrating watsonx.governance as a Service on AWS with Amazon SageMaker

You can integrate watsonx.governance as a Service on AWS with Amazon SageMaker to bring the governance, risk, and compliance capabilities of watsonx.governance to your Amazon SageMaker models.

Before you begin

You must have the following services:

  • An instance of watsonx.governance
  • An instance of Amazon SageMaker

Set up the following services on AWS:

  • CloudTrail: Amazon SageMaker events are published to CloudTrail.
  • Simple Queue service (SQS): Use this service to set up a queue for your Amazon SageMaker models. For more information, see Setting up Amazon SQS.
  • EventBridge: Use this service to define rules that extract Amazon SageMaker model card changes to a Simple Queue Service (SQS) FIFO queue.

Ensure that the CloudTrail and EventBridge services are running.

In AWS, set up a long-term access key for the integration with watsonx.governance. Your credentials must have the following access:

  • Programmatic access to Simple Queue service (SQS)
  • Programmatic access to the Amazon SageMaker APIs
  • Permissions to update model cards in Amazon SageMaker

For more information, see AWS security credentials.

In Governance console, make sure that your user account has the All/SOX/Administration/SageMaker application permission.

Setting up AWS services for the integration

Do the following steps:

  1. Set up an SQS queue for your Amazon SageMaker models.
  2. Set up an Amazon EventBridge rule and add your SQS queue as the target for the rule.

Setting up an SQS queue

The integration uses an SQS queue to receive notifications for any changes in model cards.

To set up an SQS queue, do the following steps:

  1. From the AWS console, select the Simple Queue Service.
  2. Click Create Queue.
  3. Type a name for the queue, and then enter the following information:
    • Type: FIFO
    • Visibility timeout: 30 seconds
    • Message retention period: 4 days
    • Maximum message size: 256 KB
    • Delivery delay: 0
    • Receive message wait time: 0
  4. Under FIFO queue settings, enable the following options:
    • Content-based deduplication
    • High throughput FIFO queue

The SQS queue is defined. Next, set up an EventBridge rule.

Setting up an EventBridge rule

To set up an EventBridge rule and add your SQS queue as a target, do the following steps:

  1. From the AWS console, select the Amazon EventBridge service.
  2. Click Create Rule.
  3. Type a name for the rule, and then click Next.
  4. For the event source, click Other.
  5. For the creation method, click Custom pattern (JSON editor), and then enter the following JSON for Event pattern:
    {
   "detail": {
      "eventName": ["CreateModelCard", "UpdateModelCard", "DeleteModelCard"],
      "eventSource": ["sagemaker.amazonaws.com"]
  },
   "detail-type": ["AWS API Call via CloudTrail"],
   "source": ["aws.sagemaker"]
}
  6. Click Next.
  7. Set the target to the SQS queue that you created.
  8. Click Next on the remaining pages, and then click Create Rule.

You now have an EventBridge rule with your SQS queue as the target of the rule.

Setting up a parent business entity

Decide which business entity you want to use as the parent business entity for your Amazon SageMaker models. By default, the parent business entity for Amazon SageMaker models is Library/MRG.

If you want to use a different business entity, create it before you configure the integration with Amazon SageMaker. To see a list of business entities, click Primary menu > Organization > Business Entities. To create a business entity, you must have administrative permissions.

For example, if you want to store your Amazon SageMaker models under Library/MRG/SageMaker, go the Library/MRG business entity, click New Business Entity. In the Name field, type SageMaker, in the Type field, select Library. Complete the required fields, and then save.

Configuring the integration

To configure the integration, you need the following information:

  • Your AWS account ID.
  • The access key ID and secret ID that you set up for the integration.
  • The Amazon SQS service URL that you set up for the integration.

Required permission: You need the All/SOX/Administration/SageMaker application permission in Governance console.

To configure the integration, do the following steps:

  1. Log in to Governance console as an administrator.
  2. Click Admin menu > Integrations > Amazon SageMaker.
  3. Under General configuration, do the following steps:
    1. Select the parent business entity for your SageMaker models. Typically, models are stored under the Library/MRG entity.
    2. Select a default user. Models that don't have an assigned owner or who have an owner that doesn't exist in Governance console are assigned to this default user.
  4. In AWS account credentials, enter your AWS account credentials and region.

For the Region use us-east-1. 5. In Model Card SQS URL, enter the URL of the SQS queue that you configured. 6. Click Test connection, and then click Save. 7. Click the toggle to enable the integration. The automatic synchronization process starts. When you create models in Amazon SageMaker, the information is synced with Governance console. To see the progress, click Other > Background Processes.

Your Amazon SageMaker models are now available in Governance console.

The typical workflow for new use cases is:

  • A business user creates a use case and completes the risk assessment and applicability assessment.
  • The model developer is notified that the use case is ready for development.
  • The model developer creates a model card in Amazon SageMaker.
  • The model card information is synced from Amazon SageMaker to Governance console.
  • The business user links the model to the use case.

Data synchronization

The synchronization process listens for the following actions:

  • UpdateModelCard: A model card is edited or updated in Amazon SageMaker.
  • DeleteModelCard: A model card is deleted in Amazon SageMaker.
  • CreateModelCard: A model card is created in Amazon SageMaker.

You might not see changes immediately in Governance console.

  • The sync process runs once per minute.
  • The sync process pauses when System Admin Mode is enabled. Synchronization resumes when System Admin Mode is disabled.
  • The sync process pauses when a manual synchronization is running. Synchronization resumes when the manual process is complete.

Information synced from Governance console to Amazon SageMaker

The following table shows the fields that are synced from Governance console to Amazon SageMaker.

For example, when you update the Model Owner on a model object in Governance console, the Model Owner field on the model card in Amazon SageMaker is updated.

Table 2. Fields that are synched from Governance console to Amazon SageMaker
Governance console SageMaker field
MRG-Model:Model Owner: Model owner (model_owner)
MRG-Model:Status* Model card status (model_approval_status)
MRG-AIFacts-ModelUseCase:Purpose Purpose of model (purpose_of_model)
MRG-AIFacts-ModelUseCase:Risk Level Risk rating (risk_rating)
(Use Case) System Fields:Description Business problem > Description (business_problem)
MRG-ModelUseCase:Owner Business stakeholder (business_stakeholders)

*For the initial synchronization only, MRG-Model:Status is synced from Amazon SageMaker to Governance console. For subsequent synchronizations, the status is synced from Governance console to Amazon SageMaker.

Information synced from Amazon SageMaker to Governance console

The following table shows the fields that are synced from Amazon SageMaker to Governance console.

For example, when you update the Model Creator field on a model card in Amazon SageMaker, the Model Developer field on the model object is updated in Governance console.

Table 3. Fields that are synched from Amazon SageMaker to Governance console
SageMaker Governance console
model_creator MRG-Model:Model Developer If the user doesn't exist in Governance console, the Model Developer is set to the default user that you specified when you configured the integration.
algorithm_type MRG-AIFacts-Model:Algorithm
model_id MRG-AIFacts-Model:External Asset ID
model_artifact MRG-AIFacts-Model:External Asset URL -- External assets are synced only if the folder name of the asset includes ModelArtifacts.
model_name MRG-AIFacts-Model:External Model Name
model_version MRG-Model:Model Version
Training metrics (training_metrics and user_provided_training_metrics) Metric, Metric Value
Hyper parameters (hyper_parameters and user_provided_hyper_parameters) MRG-AIFacts-Model:Hyperparameters (Flattened from an array to a string.)

Training metrics and user-provided training metrics

Training metrics and user-provided training metrics are mapped to Metric and Metric Value objects in Governance console.

Metric and Metric Value objects are created in Governance console if they don't exist. Metric objects are created as children of the Model object. Metric Value objects are created as children of the Metric object.

For training metrics and user-provided training metrics, the following fields in Governance console are updated with the information from Amazon SageMaker:

  • Metric.Description: The name of the metric in Amazon SageMaker.
  • Metric.Title: The name of the metric.
  • Metric.Status: Set to Active for existing metrics. Set to Inactive for metrics that are deleted in Amazon SageMaker.
  • Metric Value.Description: The name of the metric.
  • Metric Value.MRG-MetricVal Value: The value of the metric.
  • Metric Value.MRG-Metric-Shared:Collection Status: Set to Collected.

Two-way synchronization

The Model Card Description (model_description) field in Amazon SageMaker and the Description field on the Model object (Model.System Fields:Description) in Governance console are synced in both directions.

Parent topic: Setting up your watsonx.governance environment on AWS