Contribute in GitHub: Edit online

Snowflake

Snowflake is a cloud-based data management platform that provides enterprises with a flexible, scalable architecture for effective data collaboration. IBM Manta Data Lineage offers a powerful scanner for Snowflake databases. Manta Data Lineage can connect to the Snowflake database via JDBC and automate the extraction and analysis of metadata, including Snowflake SQL and SQL stored procedures.

Manta Data Lineage currently scans:

• Data dictionaries — get lineage for asset catalogs in hierarchical databases

• Scripts — lineage for custom SQL code and ETL activities in Manta Data Lineage native UI

• Views — visualize the SQL code logic for user-defined tables to boost data accuracy and trust and speed up reporting

• Functions — extract metadata from Snowflake functions for custom logic and pipeline insights

• External tables — see connections and dependencies across offsite data sources for queried files

• Procedures — experimentally enjoy analysis of dataflow of procedural code

Check out the guides below for more details on setting up this scanner.

• Snowflake Scanner Guide
• Snowflake Integration Requirements
• Snowflake Resource Configuration
• Snowflake Manual Inputs

Extraction and Analysis Phase Scenarios

Extraction Phase

For the extraction phase for Snowflake accounts, there are two scenarios.

1. Snowflake dictionary mapping scenario — connects to each configured Snowflake account and stores the mapping between these values: dictionary ID, account name, region, connection ID, included databases/schemas, and excluded databases/schemas

2. Snowflake extractor scenario — connects to each configured Snowflake account and extracts the database dictionary and DDL scripts from the configured databases and schemas

Analysis Phase

For the analysis phase for Snowflake accounts, there are three scenarios.

1. Snowflake dictionary dataflow scenario — analyzes metadata from the extracted Snowflake database dictionaries and saves it in your Manta Data Lineage metadata repository

2. Snowflake DDL dataflow scenario — harvests metadata and lineage from the extracted Snowflake DDL scripts and saves it in your Manta Data Lineage metadata repository

3. Snowflake SQL dataflow scenario — harvests metadata and lineage from the provided Snowflake SQL scripts and saves it in your Manta Data Lineage metadata repository

Related Articles

• How to Handle Snowflake Javascript Stored Procedures

• Lineage Missing for Tables Created by CTAS Statement

• Dictionary Dataflow Scenario Fails with java.langOutOfMemory: Java heap space

• Snowflake: Cannot Create PoolableConnectionFactory (Private Key Provided Is Invalid or Not Supported)

• Snowflake Extraction Error: JDBC Driver Internal Error "Timeout waiting for the download of #chunk0"

• Snowflake Extraction Error: Failure using stage area; Cause: Access Denied (Status Code: 403; Error Code: AccessDenied)

• How to Handle Dynamic SQL Scripts