Streaming real-time data

Build streaming data flows to act on time-sensitive data, rather than waiting to process data on an intermittent or scheduled basis.

Data engineers use the StreamSets tool to build and run streaming data flows that access and connect data across various types of data sources. A streaming flow runs continuously to read, process, and write data when the data becomes available. Streaming data flows support light in-flight transformations.

With StreamSets, data engineers can:
  • Access data from multiple types of external systems, including cloud data lakes, cloud data warehouses, and storage systems installed on-premises such as relational databases.
  • Build streaming data flows that use an intuitive graphical design interface.
  • Detect and correct unexpected data drift.

Requirements

The following requirements exist for StreamSets:

Cloud platforms
IBM Cloud AWS

Some functionality might not be available in all regions on all cloud platforms. See Regional availability.

Required service
IBM watsonx.data integration
Data formats
StreamSets supports the following data formats:
  • Tables in relational data sources
  • Avro
  • Binary
  • Datagram
  • Delimited
  • Excel
  • JSON
  • Log
  • Parquet
  • Protobuf
  • Text
  • Whole File
  • XML
For more information, see Data formats overview.
Data size
StreamSets works with data of any size.
Required permissions
Your role determines which tasks you can complete:
  • To administer a StreamSets environment for a project, you must have the Editor or Admin role in the project.
  • To create a StreamSets flow and run a job for the flow, you must have the Editor or Admin role in the project.
  • To view the job run details created for a StreamSets flow, you can have the Viewer, Editor, or Admin role in the project.

Streaming environments and engines

Before you build StreamSets flows, create a StreamSets environment to configure engines and compute resources for your project. Run the engines where your data resides, which can be on-premises systems or protected cloud platforms.

You build and manage flows in the IBM watsonx.data integration control plane. You run flows as jobs on engines in the data plane.

An engine uses the flow configuration to process the data. As a job runs, the engine sends status updates and metrics to watsonx.data integration so that you can monitor progress in real time. Because jobs run in the data plane, you maintain all ownership and control of your data.

The following image shows how watsonx.data integration works with StreamSets engines in the data plane:

Streaming real-time data tasks

Complete the following high-level tasks to stream real-time data: