Adapter Properties and Commands

Access Key

Specifies the key of AWS user for the connection. This access key is used to sign programmatic requests to AWS API calls through the adapter. The corresponding adapter command is -AK (or -ACCESSKEY).

Secret Key

Specifies the key of AWS user for the connection. Like the username and password, both the access key ID and secret access key are mandatory to authenticate S3 object for a Get or Put request. The corresponding adapter command is -SK (or -SECRETKEY).

Region Name

Specifies the name to connect to an AWS region for the IAM user account. The corresponding adapter command is -R (or -REGION).

Bucket Name

Specifies the name in which an S3 object would reside. During the action configuration, S3 Adapter provides options to select bucket name among available buckets. The corresponding adapter command is -B (or -BUCKET).

Folder Name

Specifies the folder name where S3 object may reside inside a bucket. This property is optional because an object may directly reside inside a bucket or under folder. When the folder property is not specified, adapter takes the direct path of the bucket to access the object. Otherwise, the folder name is prefixed to the key name. The corresponding adapter command is -F (or -FOLDER).

Object Name

Specifies the name of the S3 object to be accessed. A key can reside directly in a bucket or a folder inside the bucket. The corresponding adapter command is -K (or -KEY).

Content Type

Specifies the field that indicates the media type of the entity-body sent to Amazon S3. If the type is not set, adapter takes the default type, which is a standard MIME-type that describes the format of the content. For more information, go to https://www.w3.org/Protocols/rfc1341/4_Content-Type.html.

Type: String

Default: binary/octet-stream

Valid Values: MIME types

Constraints: None

The corresponding adapter command is -CT (or -CONTENTTYPE).

Content Language

Sets the Content-Language HTTP header, which specifies the natural language(s) of the intended audience for the enclosed entity. If no language is set, S3 adapter takes the default value ‘en’. The corresponding adapter command is -CL (or -CONTENTLANG).

Encryption

Specifies the server-side encryption algorithm while encrypting the object using the AWS-managed keys. When the property is not specified, data uploaded to S3 remains unencrypted. The corresponding adapter command is -E (or -ENCRYPTION).

Valid Values: AES256, KMS.

AES256: 256 bit – Advanced Encryption Standard is a data or file encryption technique. The result of the process is downloadable in a text file. AES performs all its computations on bytes rather than bits. Hence, AES treats the 256 bits of a plaintext block as 16 bytes. It uses 14 rounds for 256-bit keys. For more information, see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html.

KMS: It represents the AWS Key Management Service (AWS KMS). The master keys that the user creates in AWS KMS are used to encrypt or decrypt the object uploaded to S3. For more information on KMS encryption technique, see https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html.

Logging

Specifies the level of logging to use the log (trace) file produced by the adapter. The default is Off. The value Information means log informational, the value Errors Only means log error messages only, and the value Verbose means log debug and trace level messages along with the informational and error messages.

Logging can be specified using command:

-T [E|V] [+] [file_path]

-T Log adapter informational messages.

-TE Log only adapter errors during map execution.

-TV Use verbose (debug) logging. The log file records all activity that occurs while the adapter is producing or consuming messages.

+ Appends the trace information to the existing log file. Omit this keyword to create a new log file.

file_path The full path to the adapter trace log. If you omit this keyword, the adapter creates the m4azblob.mtr log file in the map directory.

Append Log

Specifies the flag that indicates the action to take when the specified log file already exists. When set to true, the log messages are appended to the file. When set to false, the file is truncated, and the messages are written to the empty file. The default value is true.

Read Mode

The default value is Keep.

The corresponding adapter command is -RM list|keep|delete (or -READMODE list|keep|delete). The command values are case-insensitive.

Key Prefix

Specifies the prefix to use for filtering objects from the specified bucket when the Read Mode property is set to value List. Only the objects with keys that begin with the specified prefix are included in the returned list. By default, no filtering is applied, and all the objects are returned in the list.

The corresponding adapter command is -KP prefix (or -KEYPREFIX prefix) where prefix is the key prefix value to use for the filtering.

For example, presume the bucket contains objects with the following keys:

key1

keyX

mykey1

mykeyX

test.txt

If -RM list -KP mykey commands are specified, the adapter will return the following list:

mykey1

mykeyX

Key Delimiter

Specifies the delimiter to use for filtering objects from the specified bucket when the Read Mode property is set to value List. Only the objects with keys that do not contain the specified delimiter will be included in the returned list. By default, no filtering is applied, and all the objects are returned.

The corresponding adapter command is -KD delimiter (or -KEYDELIMITER delimiter) where delimiter is the key delimiter value to use for the filtering.

For example, presume the bucket contains objects with the following keys:

dir1/obj1

dir1/obj2

dir1/subdirA/obj1

key1

key2

keyX

mykey1

mykeyX

test.txt

If -RM list -KD / commands are specified, the adapter will return the following list:

key1

key2

keyX

mykey1

mykeyX

test.txt

If -RM list -KP /dir1/ -KD / commands are specified, the adapter will take the specified key prefix dir1/ into account as well, and will return the following list:

dir1/obj1

dir1/obj2

Endpoint

Specifies the endpoint to connect and access the S3 or S3 compatible buckets by the Amazon S3 adapter. If not specified, Amazon S3 adapter connects to the Amazon S3 Service endpoint based on the region specified.

To access an S3 compatible bucket hosted by an object storage service other than the default Amazon S3 service, Endpoint option must be specified. Region option is optional. The corresponding adapter command is -EP (or -ENDPOINT).

Log File Path

Specifies the location of the log file to which the log messages are written. If not specified, the default log file name m4s3.trc is used, and the file is stored to the directory in which the executed compiled map resides.

Convert Data

Specifies to convert data to parsed form. if enabled, support for converting CSV, Avro, Excel, Parquet and JSON documents into sets of data records in the CSV format. This means that the data produced by the converters can be loaded directly into relational databases.

Data Format

Header

If enabled, specifies whether the CSV data contains a header row.

Quote

Specifies the quote character. Default value is ".

Delimiter

Specifies the field delimiter character. Default value is ,.

Escape

Specifies the escape character.

Separator

Specifies the row separator characters. Default value is \r\n.

Character Set

Specifies the character set of the data. Default value is UTF-8.

Limit

Specifies the maximum number of records to return.

Sample Size

Specifies the number of records to analyze when determining the data structure. Default value is 100.

Worksheet Index

Specifies the index of the worksheet. The index of the first worksheet is 1 (default).

Select Source Tables

Select which arrays of objects in the AVRO data you wish to map to tables in the target.

Find Array

If enabled, specifies that the output will iterate on the first array found in the JSON document.

Array Path

Specifies the path of the JSON array within the document. Omit if the document is an array. For example,"/resources".