Manage your models on multiple clusters with a custom governance connector

Before you begin

Before you follow these steps for model lifecycle management and governance, ensure that you have:

• A custom governance connector for your primary governance cluster and remote clusters
• The service URL for the the remote fact export software development kit (SDK)

Contact your instance administrator to learn which type of connection is set up for your remote cluster, and additional information such as the service URL.

About this task

If you do not have a direct connection, the only way to sync data is by doing it manually.

Govern your models on clusters with a custom connection

Follow these steps to manage governance activities for your models:

1. Log in to your primary governance cluster.

2. Create or upload models on the primary governance cluster. Models that are not created on the primary governance cluster are not registered properly and can lead to data loss.

3. Develop and evaluate the models the same as you would for models on stand-alone clusters.

4. When the models are ready to deploy, move the models to the remote cluster:

   1. Export the models from the primary governance cluster.
   2. Import the models onto the remote cluster where you want to evaluate and deploy them.

5. On the remote cluster, evaluate and deploy your model as you would for regular models. Governance activities will be logged and synced to the primary governance cluster.

   Tip: Only actions of evaluation, deployment, and deletion of a model will be synced to the primary governance cluster. If you create or import a model on the remote cluster, the model will not be logged and synced until you evaluate or deploy the model. After you evaluate or deploy a model, the model’s activity data will be logged.

6. Use POST requests to retrieve and back up the latest events for all models. POST requests require that the watsonx.governance instance administrator has set up the remote fact export software development kit (SDK) and its API endpoint on a VM correctly.

   1. Retrieve the latest events. Send a POST request to your service URL:

      http://<service_url>/event
      

      A payload with the events in JSON format will be saved to data/events.jsonl by default. If the event is successful, you see the following message:
      { "message": "Event saved to data/events.jsonl" }

   2. Back up the latest events. Send a POST request to your service URL:

      http://<service_url>/rollout
      

      A payload with the events in JSON format will be saved to a data/events.jsonl_backup_<timestamp>.jsonl by default, for example events_backup_18-36-50_2024-11-18.jsonl. If the event is successful, events.jsonl is empty and has a file size of 0. You see the following message: { "message": "Backup and rollout completed successfully." }

7. Retrieve the payload of the event data and save it as a local file by running:

   scp root@9.30.123.6:/root/<service folder>/<events_backup_filename>.jsonl ~/DEV/events.jsonl
   

8. Set up the primary fact import SDK that imports data to the primary governance cluster. Download the Primary fact import SDK from GitHub and update the code with your credentials.

9. Import the data to the primary governance cluster. Run the Python script that is in the code samples that you downloaded in the previous step:

   python3 gov_sdk_script.py
   

10. View the data such as the model’s development, evaluation, and deployment histories, in addition to details such as data size or attachments.

    1. On the primary governance cluster, open the model inventory for example by clicking AI uses cases > Lifecycle.
    2. Click on the model for which you want to view recorded activities.

The data that you see will be stale. To continue to get data, repeat steps 6 - 10.

Parent topic: Managing governance activities across multiple clusters