Supported data sources for metadata import, metadata enrichment, and data quality rules (Watson Knowledge Catalog)

The following table lists the data sources from which you can import metadata, against which you can run metadata enrichment or data quality rules, and to which you can write the output of data quality rules.

Required permissions

Users must be authorized to access the connections to the data sources. For metadata import, the user running the import must have the SELECT or a similar permission on the databases in question.

For metadata import with the goal of discovering data, certain connection types are supported only if the advanced metadata import feature is enabled. These connection types are marked with an asterisk (*). A MANTA Automated Data Lineage for IBM Cloud Pak for Data license key is not required for such imports. For lineage capture, the advanced metadata import and the knowledge graph features must be enabled, and a MANTA Automated Data Lineage for IBM Cloud Pak for Data license key must be installed.

For running metadata enrichment or data quality rules and for writing rule output, a corresponding connection asset must exist in the project. If the asset types imported from a specific connection don't allow for enrichment or running data quality rules, not applicable (abbreviated to N/A) is shown in the Metadata enrichment and rules-related columns. A dash (—) in a column indicates that the data source is not supported for this purpose.

By default, data quality rules and the underlying DataStage flows support standard platform connections. Not all connection types that were supported in traditional DataStage and potentially used in custom DataStage flows are supported in Watson Knowledge Catalog.

In general, the following data formats are supported:

• All: Tables from relational and nonrelational data sources
• Metadata import: Any format from file-based connections to the data sources and tool-specific formats from connections to external tools. For Microsoft Excel workbooks, each sheet is imported as a separate data asset. The data asset name equals the name of the Excel sheet.
• Metadata enrichment: Tabular: CSV, TSV, Avro, Parquet, Microsoft Excel (For workbooks uploaded from the local file system, only the first sheet in a workbook is profiled.)
• Data quality rules: Tabular: Avro, CSV, Parquet, ORC

Notes:

¹ With Data Virtualization Manager for z/OS, you add data and COBOL copybooks assets from mainframe systems to catalogs in IBM Cloud Pak for Data. Copybooks are files that describe the data structure of a COBOL program. Data Virtualization Manager for z/OS helps you create virtual tables and views from COBOL copybook maps. You can then use these virtual tables and views to import and catalog mainframe data from mainframes into IBM Cloud Pak for Data in the form of data assets and COBOL copybook assets.

The following types of COBOL copybook maps are not imported: ACI, Catalog, Natural

When the import is finished, you can go to the catalog to review the imported assets, including the COBOL copybook maps, virtual tables, and views. You can use these assets in the same ways as other assets in Cloud Pak for Data.

For more information, see Adding COBOL copybook assets.

² Table and column descriptions are imported only if the connection is configured with one of the following Metadata discovery options:

• No synonyms
• Remarks and synonyms

³ Some objects in the SFORCE schema are not supported. See Salesforce.com.

⁴ CSV files only.

⁵ Connections must be configured with the authentication method Account key (full JSON snippet).

⁶ Hive connections with Kerberos authentication require some prerequisite configurations. See Configuring Hive with Kerberos for lineage imports.