Known issues and limitations for AI Factsheets

The GET work space command may not work properly

Applies to: 5.3.1

When using the GET work space command, the phase_name parameter may return as an empty string and the associated usecases are empty.

This means when we associate usecases to a project or spaces, there may be a chance that this association does not go through properly.

Workaround: Attempt to re-associate the the usecases to the project or spaces.

Promoting and importing assets may not get tracked

Applies to: 5.3.1

When an asset is promoted to a space or imported to a space after association, the asset may not be automtically tracked to the usecase.

Workaround: Track the asset manually in the UI.

The project evaluation deployment may be marked as is_delete

Applies to: 5.3.1

When we evaluate DPTA or PTA in the project, and if alerts are enabled, the evalation may not appear in the UI. When checking the output of the API, you may see that the is_delete property is marked as true for the deployment.

Workaround: There is no current workaround, however the UI is not impacted.

Importing more than one model results in models with duplicate names

Applies to: 5.3.0

When importing multiple models in zip files or when importing projects with models to the central model management feature, the models that are imported may have the same names as each other, which will make it confusing to identify which model is correctly which.

Workaround: If you want to import more than one model, you need to import each model separately in a zip file. If you want to import more than one model that is inside an existing project, you will need to export the project with only one model selected during the export process, so when you import the project only one model will be imported.

Factsheet does not display facts for tracked machine learning model

Applies to: 5.3.0

Deployment count in AI factsheet inside a space takes a long time to update

Applies to: 5.3.0

In some scenarios, the number of deployments in a space can differ from the number shown in an associated factsheet. This is caused by a lag in registering the deployment with the factsheet. To resolve the discrepancy, you can refresh the number of deployments shown in the factsheet by following these steps:

1. Evaluate the deployment.
2. View the Lifecycle page of the associated AI use case.

The deployment will then display in the factsheet.

Last modified information for updated assets tracks both user-made and internal changes

Applies to: 5.3.0

The last modified information displayed for an asset tracks both changes done directly by a user and changes done internally by services. For example, the factsheet for a tracked model might indicate that the most recent modification was done by System.

These limitations apply to AI Factsheets.

Upgrade IBM Knowledge Catalog and Watson Studio before you upgrade AI Factsheets

You must upgrade IBM Knowledge Catalog and Watson Studio before you upgrade AI Factsheets. If you try to upgrade the AI Factsheets service to latest version before you upgrade IBM Knowledge Catalog and Watson Studio, users might see some inconsistencies and unexpected results with usage of the AI Factsheets service until all of the upgrades are complete.

Integration limitation with OpenPages

When the AI Factsheets is integrated with OpenPages, the fields created in the field groups MRG-UserFacts-Model or MRG-UserFact-Model and MRG-UserFacts-ModelEntry or MRG-UserFact-ModelUseCase are synced to modelfacts_user_op and model_entry_user_op asset type definitions. However, when the fields are created from the OpenPages application, avoid specifying the fields as required, and do not specify a range of values. If you mark them as required or assign a range of values, the sync will fail.

Common namespace required for related services

For AI Factsheets to log the appropriate information, all supplemental and compatible components must be installed in the same namespace. This includes the following services:

• Watson OpenScale
• IBM OpenPages
• IBM Knowledge Catalog
• Watson Studio
• Watson Machine Learning

Tracking pauses during upgrade process for IBM Knowledge Catalog

If you used AI Factsheets as part of IBM Knowledge Catalog before 4.6 and must install it again as a standalone service, tracking updates will be suspended from the time of the IBM Knowledge Catalog upgrade until AI Factsheets is installed. The facts for the models that are created or updated in that gap will not be tracked by AI Factsheets. You need to update the metadata of the model such as name, description, tags.

See Upgrading AI Factsheets for more details.