Azure Blob Storage
The Azure Blob Storage destination writes data to Microsoft Azure Blob Storage. You can use the Azure Blob Storage destination in standalone and cluster batch pipelines. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
When you configure the Azure Blob Storage destination, you specify connection information for Azure Blob Storage, including the storage container or file system and authentication method. You can also use a connection to configure the destination.
You specify the blob type to write data to along with a prefix and optional suffix for blob names. For page blobs, you specify the blob and page size, and can configure a start offset to determine where in a page the destination starts writing data.
The destination can generate events for an event stream. For more information about the event framework, see Dataflow Triggers Overview.
Before you use the destination, you must perform a prerequisite task.
Prerequisite
Before configuring the Azure Blob Storage destination, you must ensure that the destination can write the needed objects. The destination uses an authentication method to establish its identity with Azure.
Azure authorizes an identity to read and write objects. Azure provides various authorization methods for specific types of objects. The methods offer different levels of security. You must choose compatible authentication and authorization methods for your objects. Consult the Microsoft Azure Blob Storage documentation for information about the authorization methods.
You must complete appropriate steps for the chosen method, such as configuring an Azure Active Directory application, creating a shared key, or creating a shared access signature (SAS) token.
The Azure Blob Storage destination can use one of the following authentication methods:
- OAuth with Service Principal
- Connections made with OAuth with Service Principal authentication require
the following information:
- Application ID - Application ID for the Azure Active Directory Data Collector
application. Also known as the client ID.
For information on accessing the application ID from the Azure portal, see the Azure documentation.
- Tenant ID - Tenant ID for the Azure Active Directory
Data Collector application. Also known as the directory ID.
For information on accessing the tenant ID from the Azure portal, see the Azure documentation.
- Application Key - Authentication key or client secret
for the Azure Active Directory application. Also known as the
client secret.
For information on accessing the application key from the Azure portal, see the Azure documentation.
- Application ID - Application ID for the Azure Active Directory Data Collector
application. Also known as the client ID.
- Azure Managed Identity
- Connections made with Azure Managed Identity authentication
require the following information:
- Application ID - Application ID for the Azure Active Directory Data Collector
application. Also known as the client ID.
For information on accessing the application ID from the Azure portal, see the Azure documentation.
- Application ID - Application ID for the Azure Active Directory Data Collector
application. Also known as the client ID.
- Shared Key
- Connections made with Shared Key authentication require the following
information:
- Account Shared Key - Shared access key that Azure
generated for the storage account.
For more information on accessing the shared access key from the Azure portal, see the Azure documentation.
- Account Shared Key - Shared access key that Azure
generated for the storage account.
- SAS Token
- Connections made with SAS Token authentication require the following
information:
- Azure SAS Token - Shared access signature
(SAS) token that provides secure access to the needed resources
in Azure Blob Storage.
For more information on SAS tokens for storage containers, see the Azure documentation.
- Azure SAS Token - Shared access signature
(SAS) token that provides secure access to the needed resources
in Azure Blob Storage.
Blob Types
The Azure Blob Storage destination can write batches to Azure Blob Storage using one of the following blob types:
- Append
- The destination creates a single blob for all batches from a thread.
- Block
- The destination creates a blob for each thread and batch.
- Page
- The destination appends a page of the configured page size for every batch containing processed records of the specified data format.
For more information about Azure blob types, see the Azure documentation.
Event Generation
The Azure Blob Storage destination can generate events that you can use in an event stream. When you enable event generation, the Azure Blob Storage destination generates event records each time the destination completes writing to a blob.
- With the Email executor to send a custom email
after receiving an event.
For an example, see Sending Email During Pipeline Processing.
- With a destination to store event information.
For an example, see Preserving an Audit Trail of Events.
For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.
Event Records
The Azure Blob Storage destination event records include the following event-related record header attributes. Record header attributes are stored as String values.
Record Header Attribute | Description |
---|---|
sdc.event.type | Event type. Uses the following type:
|
sdc.event.version | Integer that indicates the version of the event record type. |
sdc.event.creation_timestamp | Epoch timestamp when the stage created the event. |
- Azure blob written
- The destination generates a file closure event record when it completes writing to a blob.
Data Formats
The Azure Blob Storage destination writes data to Azure Blob Storage based on the data format that you select. You can use the following data formats:
- Avro
- The destination writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Parquet
- The destination writes a Parquet file for each partition and includes the Parquet schema in every file.
- Protobuf
- Writes a batch of messages in each file.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
- Whole File
- Streams whole files to the destination system. The destination writes the data to the file and location defined in the stage. If a file of the same name already exists, you can configure the destination to overwrite the existing file or send the current file to error.
- XML
- The destination creates a valid XML document for each record. The
destination requires the record to have a single root field that
contains the rest of the record data. For details and
suggestions for how to accomplish this, see Record Structure Requirement.
The destination can include indentation to produce human-readable documents. It can also validate that the generated XML conforms to the specified schema definition. Records with invalid schemas are handled based on the error handling configured for the destination.
Configuring an Azure Blob Storage Destination
Configure an Azure Blob Storage destination to write data to Microsoft Azure Blob Storage. Before you use the destination, you must perform a prerequisite task.