Kinesis Producer
The Kinesis Producer destination writes data to Amazon Kinesis Streams. The destination can also send responses to a microservice origin when used in a microservice pipeline. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
To write data to an Amazon Kinesis Firehose delivery system, use the Kinesis Firehose destination. To write data to Amazon S3, use the Amazon S3 destination.
When you configure Kinesis Producer, you specify the Amazon Web Services connection information for your Kinesis cluster and the data format to use. You can also configure any additional Kinesis configuration properties that you require.
You can also use a connection to configure the destination.
When you want the destination to send responses to a microservice origin within a microservice pipeline, you specify the type of response to send.
- Writes a maximum of 500 messages and 4.5 MB in a batch.
- Allows a maximum of 50 kb/message. Larger messages are sent to the stage for error handling.
Authentication Method
You can configure the Kinesis Producer destination to authenticate with Amazon Web Services (AWS) using an instance profile or AWS access keys.
For more information about the authentication methods and details on how to configure each method, see Security in Amazon Stages.
Additional Kinesis Properties
You can add custom Kinesis configuration properties to the Kinesis Producer destination.
When you add a Kinesis configuration property, enter the exact property name and the value. The destination does not validate the property names or values.
Send Microservice Responses
The Kinesis Producer destination can send responses to a microservice origin when you use the destination in a microservice pipeline.
- All successfully written records.
- Responses from the destination system - For information about the possible responses, see the documentation for the destination system.
Data Formats
The Kinesis Producer destination writes data to Kinesis based on the data format that you select.
The Kinesis Producer destination processes data formats as follows:
- Avro
- The stage writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Protobuf
- Writes one record in a message. Uses the user-defined message type and the definition of the message type in the descriptor file to generate the message.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
Configuring a Kinesis Producer Destination
Configure a Kinesis Producer destination to write data to Amazon Kinesis Streams.