Google BigQuery lineage configuration

To import lineage metadata from Google BigQuery, create a connection, data source definition and metadata import job.

Google BigQuery is a serverless, scalable data warehouse that enables analysis over petabytes of data.

Processed metadata

The following Google BigQuery metadata is processed and displayed on lineage:

  • Data dictionaries
  • Scripts
  • Views
  • Functions
  • Stored procedures
  • External tables
  • Jobs (only query type)

Limitations

The following data is not processed:

  • Assets that use dynamically executed code through the EXECUTE IMMEDIATE statement.

Prerequisite configuration

Before you import lineage metadata, ensure that the following prerequisites are met:

  • A BigQuery service account used by applications and virtual machines must be configured for each project from which metadata is extracted and must be granted the predefined IAM role roles/bigquery.metadataViewer.
    This role must include the following permissions:
    • bigquery.datasets.get
    • bigquery.datasets.getIamPolicy
    • bigquery.models.getMetadata
    • bigquery.models.list
    • bigquery.routines.get
    • bigquery.routines.list
    • bigquery.tables.get
    • bigquery.tables.getIamPolicy
    • bigquery.tables.list
    • resourcemanager.projects.get
    • resourcemanager.projects.list

Creating a metadata import asset

Data source connection

To connect to the data source from which you want to import lineage metadata, you need to select a data source definition and a connection. You can create them before you start creating the metadata import, or you can create them when you create and configure the metadata import asset.

Data source definition

Select Google BigQuery as the data source type.

Connection

To import lineage metadata from Google BigQuery, use a connection with one of the following authentication methods:

  • Account key (full JSON snippet)
  • Workload Identity Federation with access token
  • Workload Identity Federation with token URL

If you use a connection with the Client ID, Client secret, Access token and Refresh token authentication method, the metada import job fails.

For connection details, see Google BigQuery connection.

Connection mode

You can connect to Google BigQuery by using one of the following connection modes:

Include and exclude lists

You can include or exclude assets up to the dataset level. Provide projects and datasets in the format project/dataset. Each part is evaluated as a regular expression. Assets which are added later in the data source will also be included or excluded if they match the conditions specified in the lists. Example values:

  • myProject/: all datasets in myProject,
  • myProject2/.*: all datasets in myProject2,
  • myProject3/myDataset1: myDataset1 from myProject3,
  • myProject4/myDataset[1-5]: any Dataset in my myProject4 with a name that starts with myDataset and ends with a digit between 1 and 5

External inputs

If you use external Google BigQuery SQL or job scripts, you can add them in a .zip file as an external input. You can organize the structure of a .zip file as subfolders that represent projects and datasets. After the scripts are scanned, they are added under respective projects and datasets in the selected catalog or project. The .zip file can have the following structure:

<project_id>
   <dataset_name>
      <script_name.sql>
<project_id>
        <script_name.sql>
jobs
       <job_name.json>
<script_name.sql>
replace.csv
connectionsConfiguration.prm

The replace.csv file contains placeholder replacements for the scripts that are added in the .zip file. For more information about the format, see Placeholder replacements.

The connectionsConfiguration.prm file contains database connection resource definitions used in federated queries. The file can have the followng structure:

[{Shortcut_Name}] Type={connection_type}
Connection_String={connection_string}
Server_Name={server_name}
Database_Name={database_name}
Schema_Name={schema_name}
User_Name={user_name}

Learn more