Supported data sources for curation and data quality

Required permissions

Users must be authorized to access the connections to the data sources. For metadata import, the user running the import must have the SELECT or a similar permission on the databases in question.

General prerequisites

Connection assets must exist in the project for connections that are used in these cases:

• For running metadata enrichment including advanced analysis (in-depth primary key analysis, in-depth relationship analysis, or advanced data profiling) on assets in a metadata enrichment
• For running data quality rules
• For creating query-based data assets (dynamic views)
• For writing output of data quality checks or frequency distribution tables

Supported source data formats

In general, metadata import, metadata enrichment, and data quality rules support the following data formats:

• All: Tables from relational and nonrelational data sources

  Delta Lake and Iceberg table format for certain file-storage connectors. For analyses to work as expected, import specific files instead of top-level directories:

  • For Delta Lake tables, import _delta_log files.
  • For Iceberg tables, import metadata/version-hint.text files.

• Metadata import: Any format from file-based connections to the data sources and tool-specific formats from connections to external tools. For Microsoft Excel workbooks, each sheet is imported as a separate data asset. The data asset name equals the name of the Excel sheet.

• Metadata enrichment: Tabular: CSV, TSV, Avro, Parquet, Microsoft Excel (For workbooks uploaded from the local file system, only the first sheet in a workbook is profiled.)

• Data quality rules: Tabular: Avro, CSV, Parquet, ORC; for data assets uploaded from the local file system, CSV only

Base Premium Data quality features are available only in IBM Knowledge Catalog and IBM Knowledge Catalog Premium.

Lineage import

To import lineage information for your data, you must have one of these services installed:

• IBM MANTA Automated Data Lineage. The column title for lineage imports with this service has the subtitle external lineage.
• IBM Manta Data Lineage. The column title for lineage imports with this service has the subtitle unified lineage.

For more information about differences between these metadata import versions, see Importing metadata.

Database support for analysis output tables

In general, output tables that are generated during analysis can be written to these databases:

• IBM Db2
• IBM Db2 on Cloud
• Microsoft Azure SQL Database
• Microsoft SQL Server
• Oracle
• PostgreSQL

If a specific database connector also supports output tables, the Target for output tables column shows a checkmark.

IBM MANTA Automated Data Lineage

You can import metadata from data sources where the asset types such as ETL jobs or data models don't allow for enrichment or running data quality rules. A MANTA Automated Data Lineage for IBM Cloud Pak for Data license key is not required for metadata imports of the type Discover. For lineage capture, the knowledge graph feature and use of the FoundationDB graph database must be enabled in IBM Knowledge Catalog, and a MANTA Automated Data Lineage for IBM Cloud Pak for Data license key must be installed.

The following connectors support metadata import of the types Discover and Get lineage:

• Microsoft Power BI Desktop
• Microsoft Power BI (Azure)
• Microsoft SQL Server Integration Services (SSIS)
• Microsoft SQL Server Reporting Services (SSRS)
• MicroStrategy
• Oracle Business Intelligence Enterprise Edition
• Oracle Data Integrator
• Qlik Sense
• SAP BusinessObjects
• Tableau

In addition, you can import these types of data to catalogs:

• Business intelligence assets from the following tools:

  • Microsoft SQL Server Analysis Services (discovery and lineage)
  • Statistical Analysis System (SAS) (discovery and lineage)
    

  To import metadata from these sources, you must provide an input file. For more information, see Preparing manual input for importing business intelligence reports.

• Data integration assets from the following tools:

  • DataStage on Cloud Pak for Data (discovery and lineage)
  • Informatica PowerCenter (discovery and lineage)
  • InfoSphere DataStage (discovery and lineage)
  • Talend (discovery and lineage)
    

  To import metadata from these sources, you must provide an input file. For more information, see Preparing ETL job files for metadata import.

• Data models that were created in the following tools:

  • ER/Studio (discovery)
  • erwin Data Modeler (discovery)
  • SAP PowerDesigner (discovery)

  To import metadata from these sources, you must provide an input file. For more information, see Preparing data model files for metadata import.

A MANTA Automated Data Lineage for IBM Cloud Pak for Data license key is required for importing lineage of business intelligence assets, data integration assets, or data models.

IBM Manta Data Lineage

The knowledge graph feature must be enabled in IBM Knowledge Catalog and the default Neo4j graph database must be used.

The following connectors support metadata import with the goal Import lineage metadata. You cannot use them to import asset metadata.

• IBM DataStage for Cloud Pak for Data
• Microsoft Power BI (Azure)
• 5.1.1 and later Microsoft SQL Server Integration Services (SSIS)
• MicroStrategy
• OpenLineage
• Tableau

You can also import lineage metadata from InfoSphere DataStage. You must provide an input file. See Configuring metadata import for data integration assets.