Configuring model evaluations with manual setup

You can use the manual setup to configure machine learning model evaluations. With manual setup, you can use existing assets, such as databases and deployment spaces. You can also choose the environment type (pre-production or production) for your deployment. Unlike the auto setup, the manual setup does not install sample assets to demonstrate model evaluations.

Running the manual setup

Follow these steps to start the manual setup:

  1. Launch Watson OpenScale.
  2. From the main menu, select Services > Instances.
  3. From the Instances page, select the Menu icon for your Watson OpenScale instance and click Open.
  4. When the Model evaluation window appears, select Manual setup.

The System setup page opens. To finish the manual setup, you must complete the steps that are described in the following sections.

Adding a database connection

You can use one of the following databases to manage data for model evaluations in the data mart:

  • Db2 from the Cloud Pak for Data service catalog
  • Db2 Warehouse multinode (MPP) from the Cloud Pak for Data service catalog
  • Db2 Warehouse single node (SMP) from the Cloud Pak for Data service catalog
  • External Db2 with the following settings:
    • Product name: “DB2 Advanced Edition”
    • Version information: "11.5 or later"
  • EDB Postgres
Attention: Do not use the Db2 database instance that is installed with OpenPages when you configure watsonx.governance. The OpenPages database is reserved for OpenPages only. If you select it, the configuration can fail and model evaluations might not run.

When you set up your database, you must also select a schema. A schema is a named collection of tables in the database. For Db2, model evaluations require a table space with a page size of at least 32 kB (32768).

For Db2 options that are part of your cluster, see Services, Data Sources where you find options, such as Db2 Warehouse and Db2 Advanced Enterprise Server Edition. For an external database, you can use the IBM Db2 Database. For EDB Postgres databases, you specify standard or enterprise instances.

Follow these steps to add a database connection for model evaluations:

  1. Click the Configure icon Configure icon to open system setup and then click Database.

  2. Select your database type from the Database drop-down menu, and then specify connection details.

    <service-name>.<namespace>.svc.cluster.local>

    If you want to use an internal TLS to connect to EDB Postgres with SSL, you must also run the following command in the  Red Hat OpenShift Container platform:

    oc patch CPDEdbInstance postgres1 --type merge --patch '{"spec": {"tls": {"customServerCert": "internal-tls", "customClientCert": "internal-tls"}}}'

  3. After you successfully connect, you can select a schema and save your work. The schema name needs to be provided explicitly if you provide a Db2 instance with limited access, which does not allow the schema name to be automatically generated.

Limitations

  • Lite Db2 plans are not currently supported.

  • If you are connecting to a Db2 database to import test data for model evaluations, you must specify uppercase column names in the input schema to correspond with the case-sensitive names in the database.

  • If your models receive 10 records per minute per model for 1 year in your Watson OpenScale service instance, you must use the following sizing guidelines for your Db2 database:

    • Small (5-10 models): 20-50GB
    • Medium (10-20 models): 50-100GB
    • Large (20-50 models): 100-300GB

    If your deployment requirements exceed these guidelines, you must increase the disk size. For nonstructured models, you must increase the disk size based on the size of the data.

  • The Db2 database that is installed with OpenPages cannot be used with watsonx.governance. This database is reserved for OpenPages only and is not supported for other services. Attempting to use it can cause configuration errors and prevent model evaluations from running.

Setting up a machine learning provider

You can connect to deployed models stored in a machine learning environment, including pre-production and production environments. The following machine learning service providers are available for model evaluations:

  • Watson Machine Learning
  • Amazon SageMaker
  • Microsoft Azure ML Studio
  • Microsoft Azure ML Service
  • IBM SPSS Collaboration and Deployment Services (C&DS)

You can also use a custom service environment.

Follow these steps to connect to a machine learning provider for model evaluations:

  1. In the Machine learning providers section, click Add machine learning provider.
  2. Optional: To change the default name, click the Edit Edit icon icon beside Machine learning providers.
  3. Optional: To enter a description, click the Edit Edit icon icon beside Description.
  4. To enter connection information, click the Edit Edit icon icon beside Connection.
  5. Choose a Service provider and specify the connection details.
  6. Click Save.

Managing users and roles

You must add the users that you want to have access to your model evaluations and assign roles to determine which tasks they can complete.

When you assign roles, you can also store your secrets in a vault to manage your data securely. You can use vaults to store usernames and passwords, SSL certificates, and API Keys for model evaluations. Authentication tokens or custom secrets are not supported.

Learn more