Pipeline Finisher

When the Pipeline Finisher executor receives an event, the executor stops a flow and transitions it to a Finished state. This allows the flow to complete all expected processing before stopping.

Use the Pipeline Finisher executor as part of an event stream. You can use the Pipeline Finisher executor in any logical way, such as stopping a flow upon receiving a no-more-data event from the JDBC Query Consumer source.

For example, you might use the executor in a flow designed to migrate all existing data from Microsoft SQL Server to HDFS. And then use a separate flow to process incremental updates. Or, you might use the executor to perform traditional batch processing - to process data, then stop when all data is processed rather than waiting indefinitely for more data.

When you configure a Pipeline Finisher executor, you can specify whether the executor should reset the source after each flow run. When needed, you can use a precondition to limit the records that enter the stage to stop the flow. You might also configure the flow to notify you when the Pipeline Finisher executor stops the flow.

Before using the Pipeline Finisher executor, review the recommended implementation information.

For a solution that describes how to use the Pipeline Finisher executor, see Stopping a flow after processing all available data. For more information about dataflow triggers and the event framework, see Dataflow triggers overview.

Recommended implementation

The Pipeline Finisher executor is designed to stop and transition a flow to a Finished state after processing available data in the source system. For example, you might use the executor to stop the flow after the JDBC Query Consumer processes all available data specified in the query.

When a source generates only the no-more-data event, you can simply connect the event output to the Pipeline Finisher executor. When a source generates multiple event types, you need to ensure that the Pipeline Finisher stops the flow only after receiving the no-more-data event.

Here are some ways you can ensure the executor receives only the no-more-data event:
Configure a precondition for the Pipeline Finisher
In the executor, add a precondition to allow only a no-more-data event into the stage to trigger the executor. You can use the following expression:
${record:eventType() == 'no-more-data'}
Tip: Records dropped because of a precondition are handled based on the stage error handling configuration. So to avoid racking up error records, you might also configure the Pipeline Finisher executor to discard error records.
Use this method when flow logic allows you to discard other event types generated by the source.
Add a Stream Selector before the Pipeline Finisher
You can add a Stream Selector between the source and the executor to route only the no-more-data event to the Pipeline Finisher. Use this option when you want to pass other event types to a different branch for processing.
For example, say you're using JDBC Query Consumer source, which generates no-more-data, query success, and query failure events. And say you want to store the query success and query failure events. You can use a Stream Selector with the following condition to route the no-more-data event to the Pipeline Finisher:
${record:eventType() == 'no-more-data'}
Then you can connect the default stream - which receives the query success and query failure events - to a target for storage.

Related event generating stages

Best practice is to use the Pipeline Finisher executor only with sources that generate no-more-data events.

The following sources generate no-more-data events:
  • Amazon S3 source
  • Azure Blob Storage source
  • Azure Data Lake Storage Gen2 source
  • Azure Data Lake Storage Gen2 (Legacy) source
  • Directory source
  • Google Cloud Storage source
  • JDBC Multitable Consumer source
  • JDBC Query Consumer source
  • MongoDB Atlas source
  • Oracle Multitable Consumer source
  • Salesforce source
  • Salesforce Bulk API 2.0 source
  • SAP HANA Query Consumer source
  • SFTP/FTP/FTPS Client source
  • SQL Server CDC Client source
  • SQL Server Change Tracking source
  • Web Client source

Source reset for additional flow runs

When you want a flow to process all available data each time that the flow runs, configure the Pipeline Finisher executor to reset the source after it stops a flow. When the executor resets the source, the restart behavior for a flow is the same for all sources: the source processes all available data.

By default, the restart behavior depends on the source used in the flow. When a source does not save an offset, when you restart the flow, the source processes all available data again. For example, when the JDBC Query Consumer runs in full mode, the source processes the full query each time you restart the flow.

When a source stores an offset, when you restart the flow, the source begins at the last-saved offset by default. For example, when the JDBC Query Consumer runs in incremental mode, by default, the source continues where it left off when you restart the flow.

When you want the source to process all available data with each flow run, configure the Pipeline Finisher executor to reset the source. Though this property has no effect on sources that do not save an offset, those sources already process all available data with each flow run.

For information about resetting the source at a flow level, see Managing job offsets.

Notification options

Data Collector can notify you when the Pipeline Finisher stops a flow.

You can use an Email executor in the flow to send email notification. The Email executor allows you to configure the condition to use to send the email, email recipients, subject, and message. You can also use expressions in any property to include details from the event record in the email. Use this option to send a customized email upon receiving an event.

To send a custom email, route the same event that triggers the Pipeline Finisher to the Email executor. After the Email executor and all other stages in the flow complete their tasks, the Pipeline Finisher transitions the flow to a Finished state.

For more information about using the Email executor, see Email.

Configuring a Pipeline Finisher executor

About this task

Configure a Pipeline Finisher executor to stop and transition the flow to a Finished state when the executor receives an event record.

Procedure

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Stage Library Library version that you want to use.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling configured for the flow.
    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. All other records are handled based on the On Record Error property.

    Click Add to create additional preconditions.

    Tip: To allow only the no-more-data event to pass to the executor, use the following condition:
    ${record:eventType() == 'no-more-data'}
    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the flow for error handling.
    • Stop Flow - Stops the flow.
    Tip: When using preconditions to limit the event type that enters the executor, you might set this property to Discard to avoid processing other event types.
  2. On the Finisher tab, optionally configure the following property:
    Finisher Property Description
    Reset Source Resets the flow source after the Pipeline Finisher executor stops the flow.

    Enable this option to process all available data each time the flow runs. When disabled, the flow restart behavior depends on the source configuration.