Orchestrating flows with Watson Pipelines

With Watson Studio Pipelines, you can create a pipeline to orchestrate the running of DataStage® flows in sequence and according to conditions of your choice. This tool is provided as a beta release and is not supported for use in production environments.

To run DataStage in a pipeline, you can create a new pipeline and add components to run a selected flow. You can also import an existing ISX file to migrate an existing DataStage sequence job.

A pipeline flow is made up of node objects. Use these components to run DataStage jobs in the order of your choice, define actions to take based on job success, run scripts, send notifications, and run sections on loop. The components and features most relevant to DataStage are listed in Pipeline components for DataStage.

You can specify conditions on connections between pipeline components to determine when a flow will proceed to the next node. For more information, see Adding conditions to a pipeline. To define a complex condition, write an expression in the Expression Builder.

You can also write expressions to define user variables shared between nodes, or to define an input to a node if the component includes the Enter expression option. The Expression Builder uses CEL syntax and has a number of built-in functions. To reference a pipeline component as a variable within an expression, copy the Node ID from the node's properties or use tasks. to autofill the Node ID of the previous node in the flow.

Migration requires DataStage expressions to be translated into CEL expressions. For more information about CEL and the Expression Builder, see CEL expressions and limitations in DataStage. For more information about other migration requirements and limitations, see Migrating and constructing pipeline flows for DataStage.