Running StreamSets jobs in sequence with IBM Orchestration Pipelines

Use IBM Orchestration Pipelines to run StreamSets jobs in sequence and control when each job runs based on defined conditions.

You can also use IBM Orchestration Pipelines to schedule and run pipelines that include StreamSets jobs.

You design a pipeline by dragging nodes onto the canvas, specifying the node configuration options, then running and monitoring the pipeline.

To run and schedule StreamSets jobs in a pipeline, complete the following high-level steps:

1. Create a StreamSets flow.
2. Start the flow, then stop the initial job run.
3. Within Orchestration Pipelines, create a pipeline.
4. Add one or more Run StreamSets job nodes to the pipeline.
5. Define conditions on the connections between nodes to control when the pipeline proceeds to the next node.
6. Run the pipeline in real time or on a schedule.

For more information about designing pipelines, see Orchestrating tasks with Orchestration Pipelines.