Kinesis Firehose
The Kinesis Firehose destination writes data to an Amazon Kinesis Firehose delivery stream. Firehose automatically delivers the data to the Amazon S3 bucket or Amazon Redshift table that you specify in the delivery stream. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
To write data to Amazon Kinesis Streams, use the Kinesis Producer destination. To write data directly to Amazon S3, use the Amazon S3 destination.
When you use the Kinesis Firehose destination to deliver data to Amazon S3, Firehose can buffer incoming records into larger file sizes before delivering the data to Amazon S3. You configure the buffer size and buffer interval when you create the delivery stream.
When you configure the Kinesis Firehose destination, you specify an existing delivery stream to write to, AWS credentials and region, and the data format to use.
You can also use a connection to configure the destination.
Authentication Method
You can configure the Kinesis Firehose destination to authenticate with Amazon Web Services (AWS) using an instance profile or AWS access keys.
For more information about the authentication methods and details on how to configure each method, see Security in Amazon Stages.
Delivery Stream
The Kinesis Firehose destination writes data to an existing delivery stream in Amazon Kinesis Firehose. Before using the Kinesis Firehose destination, use the AWS Management Console to create a delivery stream to an Amazon S3 bucket or Amazon Redshift table.
For more information about creating a Firehose delivery stream, see the Amazon Kinesis Firehose documentation.
Data Formats
The Kinesis Firehose destination writes data to a Kinesis Firehose delivery stream based on the data format that you select.
The Kinesis Firehose destination processes data formats as follows:
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. Use the multiple objects format,
where each file includes multiple JSON objects. Each object is a JSON
representation of a record.Note: The JSON array of objects format is not supported for the Kinesis Firehose destination.
Configuring a Kinesis Firehose Destination
Configure a Kinesis Firehose destination to write data to an Amazon Kinesis Firehose delivery stream.