Error record handling properties

7.4 and later

To connect to an external system, you can select a connection that contains the details, or you can specify the details in local properties. When you select a connection, the flow canvas hides properties that are defined in the connection.

• AWS Keys - Authenticates using an AWS access key pair.
• Environment Variables - Authenticates with environment variables defined on the host.
• Profile - Authenticates with temporary credentials created with the Amazon IAM Roles Anywhere credential helper tool credential process command.
• Instance Profile - Authenticates using an instance profile associated with the Data Collector EC2 instance.
• None - Connects to a public bucket using no authentication.

Amazon resource name (ARN) of the role to assume, entered in the following format:

arn:aws:iam::<account_id>:role/<role_name>

Where <account_id> is the ID of your AWS account and <role_name> is the name of the role to assume. You must create and attach an IAM trust policy to this role that allows the role to be assumed.

Available when assuming another role.

Optional name for the session created by assuming a role. Overrides the default unique identifier for the session.

Available when assuming another role.

Maximum number of seconds for each session created by assuming a role. The session is refreshed if the flow continues to run for longer than this amount of time.

Set to a value between 3,600 seconds and 43,200 seconds.

Available when assuming another role.

Sets a session tag to record the name of the currently logged in user that starts the flow or the job for the flow. AWS IAM verifies that the user account set in the session tag can assume the specified role.

Select only when the IAM trust policy attached to the role to be assumed uses session tags and restricts the session tag values to specific user accounts.

When cleared, the connection does not set a session tag.

Available when assuming another role.

When cleared, the stage uses the Amazon S3 default global endpoint, s3.amazonaws.com.

When cleared, the stage uses the region specified in the endpoint.

Enter a bucket name or define an expression that evaluates to bucket names.

When using datetime variables in the expression, be sure to configure the time basis for the stage.

Use a specific partition prefix or define an expression that evaluates to a partition prefix.

When using datetime variables in the expression, be sure to configure the time basis for the stage.

You can include periods within the suffix, but do not start the suffix with a period. Forward slashes are not allowed.

Not available for the whole file data format.

• SSE-S3 - Use Amazon S3-managed keys.
• SSE-KMS - Use Amazon Web Services KMS-managed keys.
• SSE-C - Use customer-provided keys.

Default is SSE-S3.

<arn>:<aws>:<kms>:<region>:<acct ID>:<key>/<key ID>

Used for SSE-KMS encryption only.

Amazon S3 recommends using the default ownership to keep ACLs disabled except in unusual circumstances where you must control access for each object individually. For more information, see the Amazon S3 documentation.

Default is slash ( / ).

Used for SSE-KMS encryption only.

Used for SSE-C encryption only.

Used for SSE-C encryption only.

Time zone for the target system. Used with the time basis to resolve datetimes in a time-based bucket or partition prefix.

When the Bucket and Partition Prefix properties have no time component, you can ignore this property.

Default is ${time:now()}.

You can configure multiple tags. When you configure a tag, you can define a tag with just the key or specify a key and value.

When working with multiple partitions, setting this property up to the number of partitions being used to can improve performance.

For more information about this and the following properties, see the Amazon S3 TransferManager documentation.

To retrieve a connection string key, after you access the list of available shared access policies, click the policy name, and then copy the Primary Connection String value.

The value typically begins with Endpoint.

http://<host1>,http://<host2>

You can specify a port number in the URLs to override the default port defined in the HTTP Port property, as follows:

http://<host1>:<port>,http://<host2>:<port>

When a port number is defined in both this property and in the HTTP Port property, the port in this property takes precedence. For example, if you define this property as follows:

http://server1,http://server2:1234

And you define the default HTTP Port property as 9200, then server1 uses the default port of 9200 and server2 uses the port 1234.

The default HTTP port is 9200. The default HTTPS port is 443.

For example, if you enter customer as the index, the target writes the document within the customer index.

If you use datetime variables in the expression, make sure to configure the time basis appropriately. For details about datetime variables, see Datetime variables.

By default, the target allows Elasticsearch to create the ID.

Selecting this property is the equivalent to setting the client.transport.sniff Elasticsearch property to true.

Use only when Data Collector shares the same network as the Elasticsearch cluster. Do not use for Elastic Cloud or Docker clusters.

• ${time:now()} - Uses the processing time as the time basis. The processing time is the time associated with the Data Collector running the flow.
• An expression that calls a field and resolves to a datetime value, such as ${record:value(<date field path>)}. Uses the datetime result as the time basis.

When the Index property does not include datetime variables, you can ignore this property.

Default is ${time:now()}.

Use to establish a parent-child relationship between documents in the same index.

Elasticsearch routes a document to a particular shard in an index based on the routing value defined for the document. You can define a custom value for each document. If you don’t define a custom routing value, Elasticsearch uses the parent ID (if defined) or the document ID as the routing value.

Character encoding of the data to be processed.

For example, you can use the _retry_on_conflict field to specify how many times an update is retried when there is a version conflict. To specify three retries, include the following:

"_retry_on_conflict" : 3

You can use record functions and delimited data record functions when configuring this property.

For more information, see the Elasticsearch documentation.

• Basic - Authenticate with Elasticsearch user name and password. Select this option for Elasticsearch clusters outside of Amazon OpenSearch Service.
• AWS Signature V4 - Authenticate with AWS. Select this option for Elasticsearch clusters within Amazon OpenSearch Service.

Available when using Basic authentication.

Available when using Basic authentication.

Available when using AWS Signature V4 authentication.

Available when using AWS Signature V4 authentication.

Available when using AWS Signature V4 authentication.

Configuring this property is the equivalent to configuring the shield.ssl.truststore.path Elasticsearch property.

Not necessary for Elastic Cloud clusters.

Configuring this property is the equivalent to configuring the shield.ssl.truststore.password Elasticsearch property.

Not necessary for Elastic Cloud clusters.

Uses the prefix sdc-${sdc:id()} by default. The prefix evaluates to sdc-<Data Collector ID>. This provides default differentiation in case several Data Collectors write to the same directory.

The Data Collector ID is stored in the following file: $SDC_DATA/sdc.id file.

You can enter a number of seconds or use the default expression to enter the time in minutes.

Use 0 to avoid using this property.

Use a specific partition prefix or define an expression that evaluates to a partition prefix.

When using datetime variables in the expression, be sure to configure the time basis for the stage.

Time zone for the target system. Used to resolve datetimes in a time-based partition prefix.

When the Partition Prefix property has no time component, you can ignore this property.

Default is ${time:now()}.

Not required for the whole file data format.

7.0 and later

To connect to an external system, you can select a connection that contains the details, or you can specify the details in local properties. When you select a connection, the flow canvas hides properties that are defined in the connection.

Google Cloud project ID to use.

• Default credentials provider - Uses Google Cloud default credentials.
• Service account credentials file (JSON) - Uses credentials stored in a JSON service account credentials file.
• Service account credentials (JSON) - Uses JSON-formatted credentials information from a service account credentials file.

Enter a path relative to the Data Collector resources directory, $SDC_RESOURCES, or enter an absolute path.

Enter JSON-formatted credential information in plain text, or use an expression to call the information from runtime resources or a credential store.

Google Cloud project ID to use.

• Default credentials provider - Uses Google Cloud default credentials.
• Service account credentials file (JSON) - Uses credentials stored in a JSON service account credentials file.
• Service account credentials (JSON) - Uses JSON-formatted credentials information from a service account credentials file.

Enter a path relative to the Data Collector resources directory, $SDC_RESOURCES, or enter an absolute path.

Enter JSON-formatted credential information in plain text, or use an expression to call the information from runtime resources or a credential store.

Default is 1000.

Default is 100.

Default is 1.

Set to 0 to never control the flow based on message count. To control the message flow when using batch processing, set to a number larger than the message count threshold.

Set to 0 to never control the flow based on message size. To control the message flow when using batch processing, set to a number larger than the request bytes threshold.

• Throw Exception - Triggers flow error processing.
• Block - Stops processing new messages until stored messages have been successfully written.
• Ignore - Discards new messages until stored messages have been successfully written.

7.0 and later

To connect to an external system, you can select a connection that contains the details, or you can specify the details in local properties. When you select a connection, the flow canvas hides properties that are defined in the connection.

To ensure a flow can connect to Kafka in case a specified broker goes down, list as many brokers as possible.

With Data Collector 7.2 and later, the topic must exist before you run the flow.

Not available when using runtime topic resolution.

Use an asterisk (*) to allow writing to any topic name. By default, all topic names are valid.

Use the property names and values as expected by Kafka. Do not use the broker.list property.

• None (Security Protocol=PLAINTEXT) - Uses no authentication or encryption.
• SSL/TLS Encryption (Security Protocol=SSL)
• SSL/TLS Encryption and Authentication (Security Protocol=SSL)
• SASL Authentication (Security Protocol=SASL_PLAINTEXT)
• SASL Authentication on SSL/TLS (Security Protocol=SASL_SSL)
• Confluent Cloud (Security Protocol=SASL_SSL)
• Custom Authentication (Security Protocol=CUSTOM)

Enabling security requires completing several prerequisite tasks and configuring additional security properties, as described in Security in Kafka stages.

For more information about working with Kafka message key values, see Kafka message keys.

• Round Robin - Takes turns writing to different partitions.
• Random - Writes to partitions randomly.
• Expression - Uses an expression to write data to different partitions. Writes records to the partitions specified by the results of the expression.
  Note: The expression results are written to a specified Kafka message key attribute, overwriting any existing values. Because this partition strategy uses Kafka message keys, you cannot use the Kafka Message Key property in the target to pass other Kafka message keys to Kafka. If records already contain Kafka message keys that you want to pass to Kafka, use a different partition strategy.
• Default - Uses an expression to extract a partition key from the record. Writes records to partitions based on a hash of the partition key.

When using the default partition strategy, specify an expression that returns the partition key from the record. The expression must evaluate to a string value.

When using the expression partition strategy, specify an expression that evaluates to the partition where you want each record written. Partition numbers start with 0. The expression must evaluate to a numeric value.

Optionally, click Ctrl + Space Bar for help with creating the expression.

Enter an expression that specifies the attribute where the message keys are stored.

${record:attribute('<message key attribute name>'}

${avro:decode(record:attribute('avroKeySchema'),base64:decodeBytes(record:
attribute('<messsage key attribute name')))}

For more information, about working with Kafka message keys, see Kafka message keys.

• AWS Keys - Authenticates using an AWS access key pair.
• Instance Profile - Authenticates using an instance profile associated with the Data Collector EC2 instance.

Amazon resource name (ARN) of the role to assume, entered in the following format:

arn:aws:iam::<account_id>:role/<role_name>

Where <account_id> is the ID of your AWS account and <role_name> is the name of the role to assume. You must create and attach an IAM trust policy to this role that allows the role to be assumed.

Available when assuming another role.

Optional name for the session created by assuming a role. Overrides the default unique identifier for the session.

Available when assuming another role.

Maximum number of seconds for each session created by assuming a role. The session is refreshed if the flow continues to run for longer than this amount of time.

Set to a value between 3,600 seconds and 43,200 seconds.

Available when assuming another role.

Sets a session tag to record the name of the currently logged in user that starts the flow or the job for the flow. AWS IAM verifies that the user account set in the session tag can assume the specified role.

Select only when the IAM trust policy attached to the role to be assumed uses session tags and restricts the session tag values to specific user accounts.

When cleared, the connection does not set a session tag.

Available when assuming another role.

When you add a configuration property, enter the exact property name and the value. The stage does not validate the property names or values.

• Random - Generates a random partition key.

• Expression - Uses the result of an expression as the partition key.

Use for the expression partition strategy.

<tcp | ssl>://<hostname>:<port>

Use ssl for secure connections to the broker.

tcp://localhost:1883

tcp://hostA:1883,tcp://hostB:1883,tcp://hostC:1883

${flow:id()}

If a flow includes multiple MQTT stages and you want to use the unique flow ID as the client ID for both stages, prefix the client ID with a string like this:

sub-${flow:id()} and pub-${flow:id()} 

• At Most Once (0)
• At Least Once (1)
• Exactly Once (2)

For more information, see the HiveMQ documentation on quality of service levels.

• Memory - Store messages in memory on the Data Collector machine until the delivery of the message is complete.
• File - Store messages in a local file on the Data Collector machine until the delivery of the message is complete.

Not used when the quality of service level is at most once.

For more information, see the HiveMQ documentation on client persistence.

The user who starts Data Collector must have read and write access to this directory.

Default is 60 seconds.

When selected, the MQTT broker retains the last message published by the target. Any messages published earlier are lost. When cleared, all messages published by the target are lost.

For more information about MQTT retained messages, see http://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages.

Using simple or bulk edit mode, click the Add icon to add additional certificates.

Path to the local truststore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory:

${runtime:resourcesDirPath()}/truststore.jks

By default, no truststore is used.

Default is Java Keystore File (JKS).

Password to the truststore file. A password is optional, but recommended.

Algorithm to manage the truststore.

Default is SunX509.

Enter the Java Secure Socket Extension (JSSE) name for the additional cipher suites that you want to use.