Managing Pipelines and Fragments
You manage pipelines and pipeline fragments in the same way.
Managing pipelines in the Pipelines view and fragments in the Fragments view includes duplicating an existing pipeline or fragment and filtering the list of displayed pipelines or fragments.
You can add sticky notes to the pipeline canvas to communicate with other users as you build a pipeline or fragment. As you build a pipeline, you can start a draft run of the pipeline to quickly test the pipeline logic.
When you have finished building a pipeline, you publish or check in the pipeline so that it can be added to a job. From the Pipelines view, you can also create a job for a published pipeline and view a list of jobs that include the published pipeline.
Similarly, when you have finished building a fragment, you publish or check in the fragment so that it can be added to a pipeline.
When needed, you can delete pipeline versions that are not included in jobs and delete fragment versions that are not included in pipelines.
Duplicating a Pipeline or Fragment
Duplicate a pipeline or fragment when you want to keep the existing version while continuing to configure a duplicate version. A duplicate is an exact copy of the original pipeline or fragment.
You can duplicate draft and published pipelines and fragments. When you duplicate a pipeline or fragment, you can rename the duplicate and specify the number of copies to make.
Filtering Pipelines and Fragments
In the Pipelines and Fragments views, you can filter the list of displayed pipelines or fragments by execution engine type, status, or label. Or you can search for pipelines or fragments by name, description, or commit message.
Using Sticky Notes
As you build a pipeline or pipeline fragment, you can add sticky notes to the pipeline canvas to include notes that might be useful to you or your team.
For example, you might add a note reminding you to test newly added expressions, as follows:
When you duplicate or export a pipeline or fragment with sticky notes, the notes are retained. However, if you add a published fragment with sticky notes to a pipeline, you cannot view the fragment notes when you expand the fragment in the pipeline. You can view fragment sticky notes only when you open the fragment in the pipeline canvas.
- Add
- To add a sticky note, select the pipeline canvas or select a stage in the
canvas, and then click the Stickies icon (
). When you add a sticky note with a stage selected, the note is added near the stage.
- Edit
- To edit the sticky note text, click inside the note and then add or remove text.
- Move
- To move a sticky note, select the note header and then drag the note to another location on the canvas.
- Delete
- To delete a sticky note, click the Delete icon
(
) in the note header.
Starting a Draft Run of a Pipeline
While editing a pipeline in the pipeline canvas, you can start a draft run to quickly test the pipeline logic.
You can start a draft run of a fully configured draft pipeline. The Draft Run menu becomes active in the pipeline canvas when a draft pipeline is complete.
You cannot start a draft run of a published pipeline. To run a published pipeline, you must first create a job for the published pipeline and then start the job.
For more information, see Draft Runs.
Viewing Published Pipelines and Fragments
You can view the complete configuration details of each published pipeline or pipeline fragment in the pipeline canvas. To view a published pipeline, click the pipeline name in the Pipelines view. To view a published fragment, click the fragment name in the Fragments view.
When the pipeline or fragment is in development, the pipeline canvas displays the pipeline or fragment in edit mode. When the pipeline or fragment is published, it displays in read-only mode. You can navigate through the canvas to review pipeline or fragment details.
All properties display in the Configuration tab in the properties panel.
In a pipeline, you can click the Rules tab in the properties panel to view metric, data, and data drift rules configured for the pipeline. You can select any stage in the pipeline and then click the Configuration tab in the properties panel to view the stage properties.
In a fragment, you can view data rules and data drift rules configured in the fragment by selecting the stream where the rule is defined. You can also select any stage in the pipeline and then click the Configuration tab in the properties panel to view the stage properties.
Viewing Jobs that Include a Pipeline
When viewing a pipeline version in the pipeline canvas, you can view the complete list of job instances and job templates that include that pipeline version.
Publishing a Pipeline
You publish pipelines after you design them in the pipeline canvas. You can publish valid pipelines. Control Hub performs explicit validation before publishing a pipeline.
Publish a pipeline to create a job that runs the pipeline, to use the pipeline as a sample pipeline, or to retain the published pipeline version for historical reference. You can only use published pipelines in jobs or as sample pipelines.
Publishing a Fragment
You publish fragments after you design them in the pipeline canvas. You can publish valid fragments. Control Hub performs explicit validation before publishing a fragment.
Publish a fragment to use the fragment in a pipeline. Pipelines can only use published fragments.
Deleting Pipelines or Fragments
You can delete pipeline versions that are not included in jobs. To delete pipeline versions that are included in jobs, you must first delete the job or edit the job so that it no longer includes the pipeline version that you want to delete.
Similarly, you can only delete fragments that are not used in pipelines. To delete fragment versions that are included in pipelines, you must first remove the fragment versions from all pipelines or delete those pipelines.