StreamSets
provides two engine types. When you create a StreamSets environment, you
select the engine type to configure.
- Data Collector
- Data Collector is a
self-managed runtime engine for processing streaming jobs.
You run Data Collector engines
near where the data resides, which can be on-premises or on a protected cloud computing platform.
- When you create a flow, the version of the associated Data Collector engine determines the
stages and other functionality that you can use when you create Data Collector flows. The engine also
runs the Data Collector flows
that you create.
- StreamSets developed
the Data Collector engine over
many years. The engine includes stages that you can use to process data from a wide range of systems
and to perform in-flight processing. The engine also provides advanced features, such as data drift
handling and event-based dataflow
triggers that perform tasks in Data Collector and external systems.
- Use Data Collector engines to
process streaming flows with a single origin and advanced Data Collector features.
- Jetstream
- Jetstream is an engine that
executes jobs on a Flink installation on
Confluent Platform.
- As with Data Collector, the
version of the associated Jetstream engine determines the
stages and other functionality that you can use when you create Jetstream flows. Unlike Data Collector, the Jetstream engine does not
process Jetstream flows.
Instead, the engine passes Jetstream flows to Flink for
processing.
- Jetstream is a modern
engine that is based on Data Collector and is optimized to
run on Flink. The engine includes certain Data Collector stages and features.
The engine also includes Jetstream-only stages and
features, such as using multiple origins within a flow. Because Jetstream flows run on Flink,
the flows can perform window-based transformations, such as joins and aggregates, on windows of
time.
- You can use a Flink cluster that is embedded with the
Jetstream engine to
quickly get started with configuring and testing Jetstream flows. You can currently use a limited set of stages in Jetstream flows.
- Use Jetstream engines if
your environment includes Flink on Confluent Platform.