Snowflake File Uploader
The Snowflake File Uploader destination writes whole files to an internal Snowflake stage. Use this destination with a whole file origin and the Snowflake executor. For details, see Snowflake File Uploader and Executor Pipelines in the Snowflake executor documentation.
You can use the Snowflake File Uploader destination with any accessible Snowflake database, including those hosted on AWS, Microsoft Azure, and private Snowflake installations. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
When you configure the Snowflake File Uploader destination, you specify Snowflake connection information, such as the Snowflake region and user account. You can optionally specify an organization name to use. You can also use a connection to configure the destination.
You define staging information, such as the database, schema, and name of the internal Snowflake stage to write to, as well as the number of threads to use. You can configure the destination to create the internal Snowflake stage, then specify the file format for the stage and related properties.
Before you use the Snowflake File Uploader destination, you must complete several prerequisite tasks.
You can configure the destination to generate events for another event stream. For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.
Prerequisites
- Optionally, create an internal Snowflake stage.
You can skip this step if you want to stage data on a Snowflake internal user stage or if you want the destination to create the stage.
- Assign required privileges.
Create an Internal Snowflake Stage (Optional)
The Snowflake File Uploader destination writes whole files to an internal Snowflake stage. You can either create the internal Snowflake stage before using the destination in a pipeline, or you can have the destination create the stage.
You can use an internal named, table, or user stage. User stages are created by default for each user. For steps on how to create a named or table stage, see CREATE STAGE in the Snowflake SQL command reference documentation.
You can use the default Snowflake configuration for all stages.
For more information about Snowflake stages, see the Snowflake documentation.
Assign Required Privileges
- Read
- Write
- Create Stage
- Create File Format
The user does not need Create File Format when you use a custom Snowflake file format as the data format for the table.
Implementation Notes
The Snowflake File Uploader destination was designed to be used with a whole file origin and the Snowflake executor. For details on using these stages in a pipeline, see Snowflake File Uploader and Executor Pipelines in the Snowflake executor documentation.
Define a Role
The Snowflake File Uploader destination requires a Snowflake role that grants all privileges needed to load data using the configured load method. Each load method requires a different set of privileges.
Before configuring the destination, ensure that you have granted the required privileges to a Snowflake role, as explained in Assign Required Privileges.
- Assign the custom role as the default role
- In Snowflake, assign the custom role as the default role for the Snowflake user account specified in the stage. A Snowflake user account is associated with a single default role.
- Override the default role with the custom role
- In the stage, use the Role property to specify the name of the custom role. The custom role overrides the role assigned to the Snowflake user account specified in the stage.
- Use a Snowflake connection
- When working with Control Hub, you can configure a Snowflake connection to provide connection details for Snowflake stages.
Event Generation
The Snowflake File Uploader destination can generate events that you can use in an event stream. When you enable event generation, the destination generates event records each time the destination completes streaming a whole file.
- With the Snowflake executor to load files to Snowflake tables.
For a description, see Snowflake File Uploader and Executor Pipelines.
- With the Email executor to send a custom email
after receiving an event.
For an example, see Sending Email During Pipeline Processing.
- With a destination to store event information.
For an example, see Preserving an Audit Trail of Events.
For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.
Event Records
Record Header Attribute | Description |
---|---|
sdc.event.type | Event type. Uses the following type:
|
sdc.event.version | Integer that indicates the version of the event record type. |
sdc.event.creation_timestamp | Epoch timestamp when the stage created the event. |
- File closure
- The destination generates a file closure event record when it closes an output file.
Configuring a Snowflake File Uploader Destination
Configure a Snowflake File Uploader destination to load whole files to an internal Snowflake stage. Use this destination with a whole file origin and the Snowflake executor. For details, see Snowflake File Uploader and Executor Pipelines in the Snowflake executor documentation.
Before you use the destination in a pipeline, complete the prerequisite tasks.