Configuring flow properties

About this task

Configure flow properties to define details about the flow, such as the runtime environment and error record handling.

Procedure

  1. While editing the flow in the canvas, click the Settings icon.
  2. In the Flow Settings dialog box, on the General tab, you can configure the following properties:
    General Property Description
    Name Name of the flow.
    Description Optional description of the flow.
    Retry Flow on Error Retries the flow upon error.
    Retry Attempts Number of retries attempted. Use -1 to retry indefinitely.

    The wait time between retries starts at 15 seconds and doubles until reaching five minutes.

    Available when Retry Flow on Error is enabled.
    Max Runners The maximum number of flow runners to use in a multithreaded flow.

    Use 0 for no limit. When set to 0, Data Collector generates up to the maximum number of threads or concurrency configured in the source.

    You can use this property to help tune flow performance. For more information, see Tuning threads and runners.

    Default is 0.

    Runner Idle Time (sec)

    Minimum number of seconds a flow runner waits when idle before generating an empty batch. The number of empty batches that are generated by flow runners displays as the Idle Batch Count in the monitor mode runtime statistics.

    Use to ensure that batches are generated periodically, even when no data needs to be processed.

    Use -1 to allow flow runners to wait indefinitely when idle without generating empty batches.

    For standalone flows only.
  3. On the Run tab, select the environment to run the flow.
    For more information about environments, see Running a Data Collector engine.
  4. On the Error Records tab, configure the following error record handling options:
    Error Records Property Description
    Error Record Policy Determines the version of the record to use as a basis for an error record. For more information, see Error records and version.
    Error Records Determines how to handle records that cannot be processed as expected.

    Use one of the following options:

    • Discard - Discards error records.
    • Send Response to Origin - Passes error records back to the microservice source stage to be included in a response to the originating REST API client. Use in microservice flows only.
    • Write to Amazon S3 - Writes error records to Amazon S3.
    • Write to Azure Event Hub - Writes error records to the specified Microsoft Azure Event Hub.
    • Write to Elasticsearch - Writes error records to the specified Elasticsearch cluster.
    • Write to File - Writes error records to a file in the specified directory.
    • Write to Google Cloud Storage - Writes error records to Google Cloud Storage.
    • Write to Google Pub/Sub - Writes error records to Google Pub/Sub.
    • Write to Kafka - Writes error records to the specified Kafka cluster.
    • Write to Kinesis - Writes error records to the specified Kinesis stream.
    • Write to MQTT - Writes error records to the specified MQTT broker.
  5. If you set the Error Records property to any option except Discard, configure the related error record handling properties.
  6. Click Save to save your changes.