Supported data sources for curation and data quality

The following table lists the data sources from which you can import metadata, against which you can run metadata enrichment or data quality rules, which you can use to create dynamic views, and to which you can write the output of data quality rules.

Base Premium Standard Unless otherwise noted, this information applies to all editions of IBM Knowledge Catalog.

Required permissions
Users must be authorized to access the connections to the data sources. For metadata import, the user running the import must have the SELECT or a similar permission on the databases in question.

For metadata import with one of the Discover goals, certain connectors are supported only if the advanced metadata import feature is enabled. These connectors are marked with an asterisk (*). A MANTA Automated Data Lineage for IBM Cloud Pak for Data license key is not required for such imports. For lineage capture, the advanced metadata import and the knowledge graph features must be enabled, and a MANTA Automated Data Lineage for IBM Cloud Pak for Data license key must be installed.

Connection assets must exist in the project for connections that are used in these cases:

  • For running metadata enrichment
  • For running advanced analysis on assets in a metadata enrichment: in-depth primary key analysis, in-depth relationship analysis, or advanced data profiling
  • For running data quality rules
  • For creating query-based data assets (dynamic views)
  • For writing output of data quality checks

If the asset types imported from a specific connection don't allow for enrichment or running data quality rules, not applicable (abbreviated to N/A) is shown in the Metadata enrichment, Metadata enrichment: Advanced analysis, and rules-related columns. A dash (—) in a column indicates that the data source is not supported for this purpose.

By default, data quality rules and the underlying DataStage flows support standard platform connections. Not all connectors that were supported in traditional DataStage and potentially used in custom DataStage flows are supported in IBM Knowledge Catalog.

In general, the following data formats are supported:

  • All: Tables from relational and nonrelational data sources
  • Metadata import: Any format from file-based connections to the data sources and tool-specific formats from connections to external tools. For Microsoft Excel workbooks, each sheet is imported as a separate data asset. The data asset name equals the name of the Excel sheet.
  • Metadata enrichment: Tabular: CSV, TSV, Avro, Parquet, Microsoft Excel (For workbooks uploaded from the local file system, only the first sheet in a workbook is profiled.)
  • Data quality rules: Tabular: Avro, CSV, Parquet, ORC

Base Premium Data quality features are available only in IBM Knowledge Catalog and IBM Knowledge Catalog Premium.

Supported connections
Connector Metadata import
(discovery)
Metadata import
(lineage)
Metadata enrichment Metadata enrichment:
Advanced analysis
Bindings in rules created from data quality definitions SQL-based rules SQL-based data assets Output tables
Amazon RDS for MySQL
Amazon RDS for Oracle
Amazon RDS for PostgreSQL
Amazon Redshift 6
Amazon S3
Apache Cassandra
Apache HDFS
Apache Hive 6 8 4
Apache Kafka
Box
Cloudera Impala
FTP
Generic S3
Google BigQuery 7
Google Cloud Storage
Greenplum
Connector Metadata import
(discovery)
Metadata import
(lineage)
Metadata enrichment Metadata enrichment:
Advanced analysis
Bindings in rules created from data quality definitions SQL-based rules SQL-based data assets Output tables
IBM Cloud Data Engine
IBM Cloud Databases for MongoDB
IBM Cloud Databases for MySQL
IBM Cloud Databases for PostgreSQL
IBM Cloud Object Storage
IBM Cognos Analytics 9 6
IBM Data Virtualization
IBM Data Virtualization Manager for z/OS 1
IBM Db2
IBM Db2 Big SQL
IBM Db2 for i
IBM Db2 for z/OS
IBM Db2 on Cloud
IBM Db2 Warehouse
IBM Informix
IBM Match 360
IBM Netezza Performance Server
IBM watsonx.data
Connector Metadata import
(discovery)
Metadata import
(lineage)
Metadata enrichment Metadata enrichment:
Advanced analysis
Bindings in rules created from data quality definitions SQL-based rules SQL-based data assets Output tables
MariaDB
Microsoft Azure Data Lake Storage
Microsoft Azure SQL Database
Microsoft Power BI Desktop (*) N/A N/A N/A N/A N/A N/A
Microsoft Power BI (Azure) (*) N/A N/A N/A N/A N/A N/A
Microsoft SQL Server
Microsoft SQL Server Integration Services (SSIS) (*) N/A N/A N/A N/A N/A N/A
Microsoft SQL Server Reporting Services (SSRS) (*) N/A N/A N/A N/A N/A N/A
MicroStrategy N/A N/A N/A N/A N/A N/A
MongoDB
MySQL
Oracle 2
Oracle Business Intelligence Enterprise Edition (*) N/A N/A N/A N/A N/A N/A
Oracle Data Integrator (*) N/A N/A N/A N/A N/A N/A
Connector Metadata import
(discovery)
Metadata import
(lineage)
Metadata enrichment Metadata enrichment:
Advanced analysis
Bindings in rules created from data quality definitions SQL-based rules SQL-based data assets Output tables
PostgreSQL
Presto
Qlik Sense (*) N/A N/A N/A N/A N/A N/A
Salesforce.com 3 3
SAP ASE
SAP BusinessObjects (*) N/A N/A N/A N/A N/A N/A
SAP HANA
SAP IQ
SAP OData 5
SingleStoreDB
Snowflake
Storage volume
Tableau (*) N/A N/A N/A N/A N/A N/A
Teradata 6

Notes:

1 With Data Virtualization Manager for z/OS, you add data and COBOL copybooks assets from mainframe systems to catalogs in IBM Cloud Pak for Data. Copybooks are files that describe the data structure of a COBOL program. Data Virtualization Manager for z/OS helps you create virtual tables and views from COBOL copybook maps. You can then use these virtual tables and views to import and catalog mainframe data from mainframes into IBM Cloud Pak for Data in the form of data assets and COBOL copybook assets.

The following types of COBOL copybook maps are not imported: ACI, Catalog, Natural

When the import is finished, you can go to the catalog to review the imported assets, including the COBOL copybook maps, virtual tables, and views. You can use these assets in the same ways as other assets in Cloud Pak for Data.

For more information, see Adding COBOL copybook assets.

2 Table and column descriptions are imported only if the connection is configured with one of the following Metadata discovery options:

  • No synonyms
  • Remarks and synonyms

3 Some objects in the SFORCE schema are not supported. See Salesforce.com.

4 To create metadata-enrichment output tables in Apache Hive at an earlier version than 3.0.0, you must apply the workaround described in Writing metadata enrichment output to an earlier version of Apache Hive than 3.0.0.

5 Information whether the data asset is a table or a view cannot be retrieved and is thus not shown in the enrichment results.

6 Specific JDBC drivers are required. See Uploading JDBC drivers for lineage import.

7 Connections must be configured with the authentication method Account key (full JSON snippet).

8 Hive connections with Kerberos authentication require some prerequisite configurations. See Configuring Hive with Kerberos for lineage imports.

9 Cognos Analytics connections that use secrets from a vault as credentials cannot be used for metadata import.

Other data sources

An administrator can upload JDBC drivers to enable connections to more data sources. See Generic JDBC.

Metadata import for discovery supports connections that are established by using third-party JDBC drivers.

Metadata enrichment also can be run on data assets from connections that are established by using third-party JDBC drivers for the following data sources:

  • Snowflake
  • Teradata

You can run data quality rules on data assets from connections that are established by using third-party JDBC drivers for the following sources:

  • Apache Cassandra
  • Apache Hive
  • Apache Kudu
  • Databricks

If the advanced metadata import feature is enabled, metadata import can also import these types of data to catalogs:

  • Business intelligence assets from the following tools:

    • Microsoft SQL Server Analysis Services (discovery and lineage)
    • Statistical Analysis System (SAS) (discovery and lineage)

    To import metadata from these sources, you must provide an input file. For more information, see Preparing manual input for importing business intelligence reports.

  • Data integration assets from the following tools:

    • DataStage on Cloud Pak for Data (discovery and lineage)
    • Informatica PowerCenter (discovery and lineage)
    • InfoSphere DataStage (discovery and lineage)
    • Talend (discovery and lineage)
  • Data models that were created in the following tools:

    • ER/Studio (discovery)
    • erwin Data Modeler (discovery)
    • SAP PowerDesigner (discovery)

A MANTA Automated Data Lineage for IBM Cloud Pak for Data license key is required for importing lineage of business intelligence assets, data integration assets, or data models.

Learn more

Parent topic: Curation