Importing asset metadata (IBM Knowledge Catalog)

You can import technical metadata for assets from a variety of data sources and create assets in a project or catalog.

Overview

You can import technical metadata that describes the characteristics of data objects such as data tables or files, COBOL copybooks, and transformation scripts into a project or a catalog. The imported technical metadata provides information for asset details, relationships, and the preview of assets. For data assets, the technical metadata allows access to the data and the generation of the data profile and data quality analysis.

Typically, metadata import is part of a larger data curation plan. For example, after you import metadata for data assets, you can add business metadata to your imported data assets by running metadata enrichment. You can also run data quality rules. Finally, you can publish the completed data assets to a catalog to share with your organization. Before you design your metadata import, make sure that you understand the implications of your choices to your overall curation plan. See Planning for curation.

Before you import metadata, design your metadata import so that you understand all your options and make the appropriate choices for your goal. See Designing metadata imports.

Required services
Service Importing metadata requires the following services:
IBM Knowledge Catalog

This service is not available by default. An administrator must install the service. To determine whether a service is installed, open the Services catalog. If the service is installed and ready to use, the tile in the catalog shows Ready to use.

Supported connections
See Metadata import (data assets) column in Supported data sources for curation and data quality.

Required permissions
To create, manage, and run a metadata import, you must have these roles and permissions:

  • The Manage asset discovery user permission.
  • The Admin or the Editor role in the project.
  • The Admin or the Editor role in the catalog to which you want to import or publish the assets.
  • Access to the connections to the data sources of the data assets to be imported and the SELECT or a similar permission on the corresponding databases.

Prerequisites

Before you start creating a metadata import for a new data source, complete the following steps:

  1. Create a data source definition.
  2. Create a connection to the data source in a project.

Each data source requires various connection details. You can find this information in each connection topic in the Connectors section. For more information about data source definition and connections in the context of metadata import, see Designing metadata imports: Data source.

Creating metadata import

To create a metadata import asset and a job for importing metadata into a project or a catalog:

  1. Open a project, go to the project's Asset page and click New asset > Import metadata for data assets.

  2. Specify a name for the metadata import. Optionally, you can provide a description.

  3. Select tags to be assigned to the metadata import asset to simplify searching. You can create new tags by entering the tag name.

  4. Select the Import asset metadata goal.

  5. Select the import target. You can import metadata into the project that you're working in or to any catalog for which you have an editor or admin role. See Import target.

  6. Provide details for the data source for your metadata import. Specify data source definition and connection. See Data source.

  7. Define a scope for the metadata import. See Scope of import. Depending on the size and contents of your data source, you might not want to import all assets but a select subset. You can include complete schemas or folders, or drill down to individual tables or files. When you select a schema or a folder, you can immediately see how many items it contains. Thus, you can decide whether you want to include the whole set or whether a subset serves your purpose better.

  8. Define whether you want to run scheduled import jobs. If you don't set a schedule, you run the import when you save the metadata import asset. You can rerun the import manually at any time. See Scheduling options.

  9. Click Create. The metadata import asset is added to the project, and a metadata import job is created. If you didn't configure a schedule, the import is run immediately. If you configured a schedule, the import runs on the defined schedule.

    Important: Assets from the same connection that were already imported through a different metadata import are not imported anew but are updated. Such assets do no longer show up in the initial metadata import. Only the most recently run metadata import contains the assets.

Depending on the outcome of the metadata import job run, a completion message or an error notification is displayed.

A completion message is displayed when the job run completed successfully, completed with warnings, or completed with errors. An error notification is displayed if the entire job run failed. Either type of notification contains a link to the job run log that provides details about the specific job run.

When the import is complete, you can see the list of assets with the following information:

  • The asset name, which provides a link to the asset in the project or catalog.
  • The asset type, such as Data or Report. For data assets, also the format, such as Relational table, is shown. For other asset types, the format column shows a dash (—).
  • The asset context, such as the parent or file path.
  • The date and time that the asset was last imported.
  • The import status, which can be Imported for successfully imported data, In progress, or Removed if the asset couldn't be reimported.

You can work with most imported data assets in the same way as with connected data assets. Imported assets have a tag automatically assigned that reflects the asset's parent if applicable.

To profile, analyze, and provide business context to imported data assets, create a metadata enrichment asset and include the metadata import asset in the data scope.

Watch this short video to see how to import asset metadata from an external source into a project. Some minor steps and graphical elements in this video might differ from your platform.

This video provides a visual method to learn the concepts and tasks in this documentation.

Next steps

You can work with the existing metadata imports. See:

Learn more

Parent topic: Importing metadata