Event generation

The event framework generates flow events for Data Collector standalone flows when the flow starts and stops.

In a Data Collector standalone flow, you can pass flow events to an executor or to another flow for additional processing. By default, these events are discarded. For more information about flow events, see Flow event generation.

For general information about the event framework, see Dataflow triggers overview.

Flow event records

Flow event records have the following event-related record header attributes. Record header attributes are stored as String values:

Record Header Attribute Description
sdc.event.type Event type. Uses one of the following types:
  • pipeline-start - Generated as the flow starts.
  • pipeline-stop - Generated as the flow stops.
sdc.event.version Integer that indicates the version of the event record type.
sdc.event.creation_timestamp Epoch timestamp when the stage created the event.
The event framework generates the following types of flow events:
pipeline-start
The event framework generates start events as the flow initializes, immediately after it starts and before individual stages are initialized.
The start event record has sdc.event.type set to pipeline-start, and the following fields:
Flow Start Event Field Description
pipelineId ID of the flow that started.
pipelineTitle User-defined name of the flow that started.
jobId ID of the job associated with the flow.
jobName User-defined name of the job associated with the flow.
user User who started the flow.
parameters Any parameters used when starting the flow.
pipeline-stop
The event framework generates stop events as the flow stops, either manually, programmatically, or due to a failure. The stop event is generated after all stages have completed processing and cleaning up temporary resources, such as removing temporary files.
The stop event record has sdc.event.type set to pipeline-stop, and the following fields:
Flow Stop Event Field Description
pipelineId ID of the flow that stopped.
pipelineTitle User-defined name of the flow that stopped.
jobId ID of the job associated with the flow.
jobName User-defined name of the job associated with the flow.
reason Reason why the flow stopped. Can be set to the following reasons:
  • Error - An error occurred as the flow was running.
  • Finished - The flow finished all expected processing.
  • User - A user stopped the flow.
user User who stopped the flow, when relevant.