Amazon S3
The Amazon S3 executor performs a task in Amazon S3 each time it receives an event. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
- Create a new Amazon S3 object for the specified content
- Copy an object under 5 GB to another location in the same bucket and optionally delete the original object
- Adds tags to an existing object
Each Amazon S3 executor can perform one type of task. To perform additional tasks, use additional executors.
Use the Amazon S3 executor as part of an event stream. You can use the executor in any logical way, such as writing information from an event record to a new S3 object, or copying or tagging objects after they are written by the Amazon S3 destination.
When you configure the Amazon S3 executor, you specify the connection information, such as access keys, region, and bucket. You configure the expression that represents the object name and location. When creating new objects, you specify the content to place in the objects. When copying objects, you specify the location of the object and the location for the copy. You can also configure the executor to delete the original object after it is copied. When adding tags to an existing object, you specify the tags that you want to use.
You can configure the executor to use Amazon Web Services server-side encryption to protect the data written to Amazon S3. You can optionally use an HTTP proxy to connect to Amazon S3.
You can also use a connection to configure the executor.
You can also configure the executor to generate events for another event stream. For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.
Authentication Method
You can configure the Amazon S3 executor to authenticate with Amazon Web Services (AWS) using an instance profile or AWS access keys. When accessing a public bucket, you can connect anonymously using no authentication.
For more information about the authentication methods and details on how to configure each method, see Security in Amazon Stages.
Create New Objects
You can use the Amazon S3 executor to create new Amazon S3 objects and write the specified content to the object when the executor receives an event record.
When you create an object, you specify where to create the object and the content to write to the object. You can use an expression to represent both the location for the object and the content to use.
For example, say you want the executor to create a new Amazon S3 object for each object that the Amazon S3 destination writes, and to use the new object to store the record count information for each written object. Since the object-written event record includes the record count, you can enable the destination to generate records and route the event to the Amazon S3 executor.
${record:value('/bucket')}/${record:value('/objectKey')}.recordcount
${record:value('/recordCount')}
Copy Objects
You can use the Amazon S3 executor to copy an object to another location within the same bucket when the executor receives an event record. You can optionally delete the original object after the copy. The object must be under 5 GB in size.
When you copy an object, you specify the location of the object to be copied, and the location for the copy. The target location must be within the same bucket as the original object. You can use an expression to represent both locations. You can also specify whether to delete the original object.
${record:value('/bucket')}/${record:value('/objectKey')}
${record:value('/bucket')}/completed/${record:value('/objectKey')}
You can then select Delete Original Object to remove the original object.
To do something more complicated, like move only the subset of objects with a _west suffix to a different location, you can add a Stream Selector processor in the event stream to route only events where the /objectKey field includes a _west suffix to the Amazon S3 executor.
Tag Existing Objects
You can use the Amazon S3 executor to add tags to existing Amazon S3 objects. Tags are key-value pairs that you can use to categorize objects, such as product: <product>.
You can configure multiple tags. When you configure a tag, you can define a tag with just the key or specify a key and value.
For more information about tags, including Amazon S3 restrictions, see the Amazon S3 documentation.
Event Generation
The Amazon S3 executor can generate events that you can use in an event stream. When you enable event generation, the executor generates events each time it creates a new object, adds tags to an existing object, or completes copying an object to a new location.
- With the Email executor to send a custom email
after receiving an event.
For an example, see Sending Email During Pipeline Processing.
- With a destination to store event information.
For an example, see Preserving an Audit Trail of Events.
For more information about dataflow triggers and the event framework, see Dataflow Triggers Overview.
Event Records
Record Header Attribute | Description |
---|---|
sdc.event.type | Event type. Uses the following event types:
|
sdc.event.version | Integer that indicates the version of the event record type. |
sdc.event.creation_timestamp | Epoch timestamp when the stage created the event. |
- file-changed
-
The executor generates a file-changed event record when it adds tags to an existing object.
File-changed event records have thesdc.event.type
record header attribute set tofile-changed
and include the following field:Event Field Name Description object_key Key of the tagged object. - file-created
-
The executor generates a file-created event record when it creates a new object.
File-created event records have thesdc.event.type
record header attribute set tofile-created
and include the following field:Event Field Name Description object_key Key of the created object.
- file-moved
-
The executor generates a file-moved event record when it completes copying an object to a new location.
File-moved event records have thesdc.event.type
record header attribute set tofile-moved
and include the following field:Event Field Name Description object_key Key of the copied object.
Server-Side Encryption
You can configure the stage to use Amazon Web Services server-side encryption (SSE) to protect data written to Amazon S3. When configured for server-side encryption, the stage passes required server-side encryption configuration values to Amazon S3. Amazon S3 uses the values to encrypt the data as it is written to Amazon S3.
- Amazon S3-Managed Encryption Keys (SSE-S3)
- When you use server-side encryption with Amazon S3-managed keys, Amazon S3 manages the encryption keys for you.
- AWS KMS-Managed Encryption Keys (SSE-KMS)
- When you use server-side encryption with AWS Key Management Service (KMS), you specify the Amazon resource name (ARN) of the AWS KMS KMS encryption key that you want to use. You can also specify key-value pairs to use for the encryption context.
- Customer-Provided Encryption Keys (SSE-C)
- When you use server-side encryption with customer-provided keys, you specify
the following information:
- Base64 encoded 256-bit encryption key
- Base64 encoded 128-bit MD5 digest of the encryption key using RFC 1321
For more information about using server-side encryption to protect data in Amazon S3, see the Amazon S3 documentation.
Configuring an Amazon S3 Executor
Configure an Amazon S3 executor to create new Amazon S3 objects or to add tags to existing objects.