What's new and changed in IBM Knowledge Catalog

Installing or upgrading IBM Knowledge Catalog

Ready to install or upgrade IBM Knowledge Catalog?

• To install IBM Knowledge Catalog along with the other Cloud Pak for Data services, see Installing Cloud Pak for Data.
• To upgrade IBM Knowledge Catalog along with the other Cloud Pak for Data services, see Upgrading Cloud Pak for Data.
• To install or upgrade IBM Knowledge Catalog independently, see IBM Knowledge Catalog.
  Remember: All of the Cloud Pak for Data components associated with an instance of Cloud Pak for Data must be installed at the same version.

Cloud Pak for Data Version 5.0.3

A new version of IBM Knowledge Catalog was released in September 2024 with Cloud Pak for Data 5.0.3.

Operand version: 5.0.3

This release includes the following changes:

New features

This release of IBM Knowledge Catalog includes the following features:

Import, enrich, and assess the quality of data from additional data sources

You can now import metadata from the following data sources, enrich that data, and assess its quality:

• Apache Impala
• SAP OData
• SingleStoreDB

For more information, see Supported data sources for curation and data quality.

Enhanced export of the lineage graph to PDF

You can now export your lineage graph to an interactive PDF, that includes detailed information, such as:

• Canvas summary
• Time and date stamp
• Details of each asset
• Column lineage

Export your lineage graph by using the Google Chrome browser. For more information, see Managing business lineage.

New storage for profiling results

Profiling results are now stored in an internal PostgreSQL database instead of the asset-files service. To retain profiling results after an upgrade to Cloud Pak for Data 5.0.3, you must migrate the results to the new storage as a post-upgrade step. For details, see Migrating profiling results after upgrading.

Bulk edit draft artifacts

You can now edit multiple draft artifacts at once. Bulk edits are available for secondary category, relationships, tags, stewards, and custom properties. For more information, see Managing governance artifacts.

Updates

The following updates were introduced in this release:

• You can now group multiple data quality rules for a table into a single DataStage flow.
• You can now include additional metrics in the output tables for data quality rules.
• You can now create a metadata enrichment without directly running it after creation.
• You can now edit and remove classifications and custom properties for multiple assets in a catalog at the same time.
• The filter Enrichment canceled in the metadata enrichment results was removed.
• The following new columns are available in the reporting tables:
  • A total number of records present in a dataset in a container_data_assets table.
  • The identifier of the DataStage flow job and job run in the dq_rule_execution table.
  • The job_run_id column was renamed to execution_id.

Customer-reported issues fixed in this release

The following issues, which were reported by customers, were fixed in this release:

• DT394234: Connector seems to be unable to catch exception causing worker pod to exit.
• DT419038:ENUM custom attributes are not searchable via the the global search UI.

Security issues fixed in this release

The following security issues were fixed in this release:

CVE-2017-1002101

CVE-2018-1002105

CVE-2019-1002100, CVE-2019-1002101, CVE-2019-11246, CVE-2019-11247, CVE-2019-11248, CVE-2019-11249, CVE-2019-11250, CVE-2019-11252, CVE-2019-11253, CVE-2019-11254

CVE-2020-8552, CVE-2020-8554, CVE-2020-8555, CVE-2020-8557, CVE-2020-8558, CVE-2020-8559, CVE-2020-8561, CVE-2020-8562, CVE-2020-8564, CVE-2020-8565

CVE-2021-25735, CVE-2021-25736, CVE-2021-25740, CVE-2021-25741, CVE-2021-25743

CVE-2022-40897

CVE-2023-2431, CVE-2023-2727, CVE-2023-2728, CVE-2023-3676, CVE-2023-39331, CVE-2023-3955, CVE-2023-42809, CVE-2023-5528

CVE-2024-31573, CVE-2024-39705, CVE-2024-41110, CVE-2024-6345

Cloud Pak for Data Version 5.0.2

A new version of IBM Knowledge Catalog was released in August 2024 with Cloud Pak for Data 5.0.2.

Operand version: 5.0.2

This release includes the following changes:

New features

This release of IBM Knowledge Catalog includes the following features:

Changes to IBM Knowledge Catalog Standard Cartridge and IBM Knowledge Catalog Premium Cartridge

Creating models is now optional for IBM Knowledge Catalog Premium and
IBM Knowledge Catalog Standard

The semantic capabilities in metadata enrichment are no longer enabled by default when you install IBM Knowledge Catalog Premium or IBM Knowledge Catalog Standard. You can now enable these capabilities by setting an installation option.

To retain the system setup when you upgrade one of these services from an earlier 5.0.x version, you must now set the enableSemanticAutomation option to true during the upgrade.

For more information, see Installing IBM Knowledge Catalog and the upgrade topics under Upgrading IBM Knowledge Catalog.

Additional capabilities in IBM Knowledge Catalog Standard

Data Refinery is now also included in IBM Knowledge Catalog Standard and you can optionally enable the Knowledge Graph component for this cartridge.

Import, enrich, and assess the quality of data from additional data sources

You can now import metadata from the Hive Metastore in Microsoft Azure Databricks, enrich that data, and assess its quality. For more information, see Supported data sources for curation and data quality.

Assign user groups as asset members

You can now assign user groups as asset members. Previously, you could add only individual catalog users as asset members.

Upload and update assets in bulk

To upload and update multiple assets in bulk, you can now import and export CSV files with either asset metadata details or asset relationship details, or both. For more information, see Adding multiple assets from a CSV file to a catalog .

Configure asset removal

Now, when you create a new catalog, you can also decide how you want to configure the removal of assets. You can either select to purge the assets automatically either immediately after the removal or 30 days after the removal. For previously created catalogs, you can change asset removal settings on the catalog Settings page.

For more information, see:

• Creating a catalog
• Changing catalog settings
• Removing an asset from a catalog

Enhanced governance artifact configuration

You can now change different types of custom properties for multiple governance artifacts at the same time. For more information, see Managing governance artifacts.

Process workflow tasks in bulk

When working with workflow tasks, you can now select a batch of compatible tasks that require the same action and then process them in bulk. For more information, see Identifying tasks that need to be completed.

Import lineage from IBM Cognos Analytics connections

You can now import lineage from IBM Cognos Analytics connections to view the imported business intelligence (BI) reports. For more information, see IBM Cognos Analytics connection.

Updates

The following updates were introduced in this release:

• You can now assign classifications to data assets and columns in metadata enrichment, either automatically based on term or data-class assignment or manually in the enrichment results.
• You can now extend the business vocabulary that is used with the metadata enrichment option Expand metadata with custom abbreviation files.
• You can now group multiple data quality rules for a single file from a file-storage connector or uploaded from the local file system into a single DataStage flow.
• You can now apply column classification in the global search filters to search for assets and assets with columns.
• You can now define conditional mandatoriness for input fields in custom workflow tasks. For more information, see Task input field mandatoriness.
• Data source definitions are now available in the reporting tables in BI data mart.

Customer-reported issues fixed in this release

The following issues, which were reported by customers, were fixed in this release:

• DT391119: v3/search can't find the custom property in assets after setting the relationship by API.
• DT386174: Improve the time of job run displayed in IBM Knowledge Catalog metadata enrichment to include job start time.
• DT395199: Failed to load error when navigating to a governance asset in a catalog.
• DT395201: Data quality dimensions cannot be exported and imported.
• DT395204: Error when opening Data quality rule in a project: Failed An unknown error occurred. Exception IOException was caught during processing of the request: RELSV1032E: Request exceeds the limit of '20' asset relationships.
• DT395214: Data quality rules analyze and display decimal values, such as 1.000000, without the decimals part.
• DT395215: Metadata enrichment job pop-up notification states the job was successful even though the job completed with errors.
• DT391771: Unable to import metadata for Azure Power BI connection when Proxy and Port need to be specified.
• DT395369: Catalog UI and Glossary UI crash when blocked from downloading fonts from external URL.

Security issues fixed in this release

The following security issues were fixed in this release:

CVE-2014-3488

CVE-2015-2156

CVE-2019-16869, CVE-2019-20444, CVE-2019-20445

CVE-2021-21290, CVE-2021-21295, CVE-2021-21409, CVE-2021-29425, CVE-2021-35515, CVE-2021-35516, CVE-2021-35517, CVE-2021-36090, CVE-2021-37136, CVE-2021-37137, CVE-2021-43797

CVE-2022-25881, CVE-2022-29599, CVE-2022-41881, CVE-2022-42003, CVE-2022-42004

CVE-2023-1428, CVE-2023-21930, CVE-2023-21937, CVE-2023-21938, CVE-2023-21939, CVE-2023-21954, CVE-2023-21967, CVE-2023-32731, CVE-2023-32732, CVE-2023-3635, CVE-2023-44487, CVE-2023-46120, CVE-2023-6597

CVE-2024-21634, CVE-2024-22262, CVE-2024-25710, CVE-2024-28182, CVE-2024-28863, CVE-2024-29025, CVE-2024-34750, CVE-2024-35195, CVE-2024-35235, CVE-2024-35326, CVE-2024-35328, CVE-2024-39249, CVE-2024-39689, CVE-2024-5206, CVE-2024-5642

Cloud Pak for Data Version 5.0.1

A new version of IBM Knowledge Catalog was released in July 2024 with Cloud Pak for Data 5.0.1.

Operand version: 5.0.1

This release includes the following changes:

New features

This release of IBM Knowledge Catalog includes the following features:

Import metadata from every database

Now, you don't have to specify the database to which you want to connect for the Informix, SAP ASE, and Microsoft SQL Server connections. With no database specified, you can import metadata from every database that is available for that connection.

Enhancements in governance artifacts

• You can now change the primary or secondary category for multiple governance artifacts at once. For more information, see Managing governance artifacts.
• You can now make bulk edits when updating relationships in governance artifacts. For more information, see Managing governance artifacts.
• When viewing all governance artifacts of a specific type, you can now filter the list by a number of properties, including custom properties. For more information, see Finding and viewing governance artifacts.

Import, enrich, and assess data quality of data from additional data sources

You can now import metadata from Microsoft Azure Databricks, enrich that data, and assess its quality. For more information, see Supported data sources for metadata import, metadata enrichment, and data quality rules.

Updates

The following updates were introduced in this release:

• You can now pause and resume metadata enrichment job runs.
• You can now view information about Notes and Data Type on the Asset page when you click on an LDM or PDM asset. Previously, you had to click a link and were redirected to a new screen.
• Reporting is now supported on two new database types: Microsoft SQL Server 2022 or later, and, Microsoft Azure SQL Database.
• The following new columns are now available in the reporting tables:
  • User group custom property for governance artifacts
  • User group as asset collaborator

Customer-reported issues fixed in this release

The following issues, which were reported by customers, were fixed in this release:

• DT389369: Column names that include special characters such as hash (#), slash (/) and trailing spaces are not handled correctly by the metadata enrichment UI.
• DT389268: Cloud Pak for Data catalogs do not sort with case-insensitivity.
• DT392891: Data Quality rules are unable to run with cs files with custom field delimiters.
• DT392897: Metadata enrichment jobs with many tables take longer than expected when performing the relationship analysis step.
• DT392901: Metadata enrichment job doesn't show profiling results for columns that include spaces in their name.
• DT392904: Metadata Enrichment UI does not display associated business terms in the Governance tab view if the MDE asset name contains includes a slash character.
• DT392907: Catalog API calls to the entitlements API take longer than expected and sometimes fail.
• DT389323: User with Hebrew display name is unable to import lineage with Manta Advanced Metadata Import.

Security issues fixed in this release

The following security issues were fixed in this release:

CVE-2022-0391

CVE-2023-39331, CVE-2023-45288, CVE-2023-6004, CVE-2023-6918, CVE-2023-7008

CVE-2024-20696, CVE-2024-20697, CVE-2024-21890, CVE-2024-21896, CVE-2024-22025, CVE-2024-24806, CVE-2024-25062, CVE-2024-25629, CVE-2024-27982, CVE-2024-27983, CVE-2024-2961, CVE-2024-33599, CVE-2024-33600, CVE-2024-33601, CVE-2024-33602, CVE-2024-36124, CVE-2024-4067, CVE-2024-4068

Cloud Pak for Data Version 5.0.0

A new version of IBM Knowledge Catalog was released in June 2024 with Cloud Pak for Data 5.0.0.

Operand version: 5.0.0

This release includes the following changes:

New features

This release of IBM Knowledge Catalog includes the following features:

Additional IBM Knowledge Catalog editions

You can continue to use the classic IBM Knowledge Catalog service, or you can choose one of the two new, separately priced editions of IBM Knowledge Catalog:

IBM Knowledge Catalog Standard Cartridge

This edition offers basic governance tooling for cataloging and AI-augmented data enrichment.

IBM Knowledge Catalog Premium Cartridge

This edition offers the full governance framework with data privacy, data quality, cataloging, and enrichment across the data lifecycle with a generative AI layer for enhanced data enrichment.

In addition to governance capabilities as in the classic IBM Knowledge Catalog service, the cartridges provide semantic and AI-augmented data enrichment:

• Recommend descriptive names for data assets and columns based on the collected metadata and a predefined glossary.
• Suggest and assign semantic descriptions for data assets and columns that are easy to understand. The descriptions are generated based on the surrounding columns and the context of the data assets.
• Generate semantic term assignments for data assets and columns.

For details, see IBM Knowledge Catalog.

Import metadata from additional data sources

You can now import metadata and lineage metadata from the following data sources:

MicroStrategy

Use a new connection to import data. For details, see Supported data sources for metadata import, metadata enrichment, and data quality rules.

OpenLineage

Import the data from a .zip file. For details, see Importing ETL jobs and Getting ETL job lineage.

Data quality enhancements

You can now add data assets or columns with the new relationship type Validates data quality of to any type of data quality rule to have the quality score and any data quality issues reported for this item on the Data quality page. With this enhancement, data quality rules with externally managed bindings and SQL-based data quality rules can now also contribute to the quality scores of assets and columns.

For details, see Creating rules from data quality definitions and Creating SQL-based data quality rules.

Data protection rules are no longer enforced in projects

Data protection rules are now only enforced in governed catalogs or by a deep enforcement solution. Assets that are added into projects from a governed catalog no longer have preview, download, or profiling restricted by data protection rules. For more information, see Data protection rules no longer enforced in projects.

Enhanced project list view in catalogs

Now, when you are adding assets from a catalog to a project, you can view more than 100 projects in your project list page and add up to 50 assets at a time to your project. For more information, see Add assets from within the catalog.

Enhancements in governance artifacts

• You can now make changes to multiple governance artifacts at once. Bulk edits are available when updating tags and stewards. For more information, see Managing governance artifacts.
• Now you can move any category either to the top level or to any other category as a sub-category. The collaborators are also moved provided they have required permissions on the new parent category. For more information, see Managing categories.
• You can now add custom properties and relationships for reference data sets. For more information, see Designing reference data sets.
• Notifications about changes in governance artifacts, for example, when an artifact is added, updated, or deleted, can now be forwarded to external applications or users. For more information, see Forwarding notifications generated by Cloud Pak for Data services.

Knowledge Accelerators

Additional data classes

There are over 20 new data classes that can be used to identify and classify national identifiers, tax identifiers and social security identifiers for the additional jurisdictions of Argentina, Egypt, Finland, Greece, Hong Kong, Ireland, Malaysia, New Zealand, Pakistan, Peru, Romania, Thailand, Turkey, and United Arab Emirates.

These new data classes supplement previously added data classes to provide an enhanced framework for identifying and classifying data of particular relevance to data privacy.

For more information, see Knowledge Accelerators data classes.

Updated business scopes for Relationship Explorer

The Knowledge Accelerators contain a set of predefined business scopes that group the set of business terms that are relevant to a specific business topic. Many of these scopes were reorganized to ensure that they are optimized for viewing in the new Relationship Explorer capability of IBM Knowledge Catalog. Also, new business scopes were added to Financial Services.

In addition, certain term-to-term relationships across the Knowledge Accelerators were simplified to improve clarity when viewing them in Relationship Explorer.

For more information, see Business scopes for Knowledge Accelerators.

Relationship Explorer to visualize your metadata

Relationship Explorer is now available to help better understand your data. This new feature helps you to visualize, explore and govern your metadata. Discover how your governance artifacts and data assets relate with each other in a single view. For more information, see Relationship Explorer.

Expand DataStage jobs in the lineage graph

When you are viewing a DataStage job in the lineage graph, you can expand the job to view all its stages. For more information, see Lineage.

Enhanced security for profiling results in Data Virtualization and watsonx.data views

To prevent unexpected exposure to value distributions through the profiling results of a view, all users are denied access to profiling results in Data Virtualization and watsonx.data views in all catalogs and projects.

Updates

The following updates were introduced in this release:

• New options were added to the business terms and data classes filters for enrichment results: assigned, suggested, and none.
• Updated the asset membership roles for catalogs. Now, users can hold the asset owner, asset editor, or asset viewer role. The asset editor role replaced the asset member role. Now, to complete any asset-related actions, you must be an asset owner or asset editor.

  Also, catalog assets might have more than one owner now.

  You can change asset user roles on the Access control page of an asset by selecting a role from the Role dropdown menu.

• Added a Add catalog assets to projects user permission. Now, to add assets to projects, you must have the Add catalog assets to projects and the Admin, Editor or Viewer role in the catalog. Users that don't have an existing role with the Manage catalogs or Access catalogs permission must be explicitly granted the Add catalog assets to projects permission.

• You can now edit and remove the business terms, owners or tags on up to 20 catalog assets at a time.
• You can now assign one or more classifications to a column within an asset of your catalog.
• The following new columns are now available in the reporting tables:
  • Column level classifications
  • SQL Query asset type
  • Asset Member Roles
  • Custom Properties of user/group type for asset and columns
  • More column attributes - native & inferred data type, mean value
• When you import metadata with the Discover import goal, you can use a new advanced option Import asset timestamp to include the information about the time when the imported asset was last modified.
• You can configure two new properties for the Qlik Sense connection: Extracted applications and Excluded applications.
• Administrator users can now configure the default reporting settings by using the ccs-features-configmap. For more information, see Configuring reporting settings for IBM Knowledge Catalog.

Customer-reported issues fixed in this release

The following issues, which were reported by customers, were fixed in this release:

• DT378506: Adding a member to a catalog results in the wrong member being added.
• DT242636: The project does not display in Add to project pull down list.
• DT386785: Metadata import job remains in running state if the Jobs API service doesn't respond at the time the job ends.
• DT386793: The Activities for this asset section for catalog assets may appear as if it not updating if there is no lineage information available for an asset.
• DT386829: For assets with short or long column names, the metadata enrichment job may complete with a HTTP 502 Bad Gateway error for ML-based term assignment.
• DT386841: When adding new assets to an existing metadata import scope, user has to re-select previously selected assets along with new assets.
• DT386843: Metadata import job may have timeout errors due to slow or delayed response from the source database.
• DT386875: User may not be able to click Add to project when assets in catalog are selected. User is also not able to see the add to project button for individual asset.
• DT386883: User gets unknown error notification after adding assets to project from catalog and these assets fail to be added to the selected project.
• DT386893: In the Cloud Pak for Data home page, the create time displayed for Catalogs is not displayed in the user’s timezone, if the user is not in the same timezone as the Cloud Pak for Data server.
• DT380903: While creating Connection for Qlik Sense results in an error: The test was not successful. An unexpected error occurred of type Null pointer error.

Security issues fixed in this release

The following security issues were fixed in this release:

CVE-2016-1000027

CVE-2022-24823, CVE-2022-41881

CVE-2023-42282, CVE-2023-44487, CVE-2023-51074

CVE-2024-21892, CVE-2024-22019, CVE-2024-22243, CVE-2024-24762, CVE-2024-28180

Deprecated features

The following features were deprecated in this release:

Egeria synchronization to external repositories is now removed

Egeria synchronization to external repositories is no longer available from IBM Knowledge Catalog.