Pipeline components for DataStage

Edit online

The listed pipeline components are useful for DataStage®. To set default policy for error handling and node caching, click Manage default settings in the toolbar.

For more control over how the pipeline responds to errors and restores information, you can use the manual option to override default behavior on specific nodes. For more information about error handling, see Managing pipeline errors.

You can use the following components to orchestrate the running of DataStage flows. For more information about each component, see Configuring pipeline components.
  • Send email
  • Wait for any result
  • Wait for all results
  • Wait for file
  • Loop in sequence
  • Terminate loop
  • Terminate pipeline
  • Set user variables
  • Run Bash script
  • Run DataStage flow
  • Run Pipeline
  • Error handling

Components

Run DataStage flow
Run the specified DataStage flow. Set runtime environment, environment variables, and job parameters.
Run bash script
Enter code to run an inline script. Set environment variables to reference upstream node outputs, parameters, and variables in your script.
Run Pipeline
Run the specified Watson pipeline. Set job parameters.
Set user variables
Define global variables for the pipeline to be referenced by later nodes. Use the Expression Builder to set values through expressions.
Send email
Send an email to a list of recipients. Provide a string of comma-separated addresses. Input the subject, body, and attachments. All fields can be parametrized.
Wait for any result
Run the downstream nodes as soon as any of the upstream conditions are met.
Wait for all results
Run the downstream nodes only once all the upstream conditions are met.
Wait for file
Wait for the specified file to either appear or disappear within a specified timeframe. If 00:00:00 is the specified timeout length, the flow waits indefinitely.
Loop in sequence
Loops can iterate over a numeric range, a list, or text with a delimiter. Define a subpipeline flow inside the loop to run until the loop is complete.
Terminate loop
Use this component inside a loop to exit the loop. When the node runs, the pipeline returns to the main flow.
Terminate pipeline
Terminate the pipeline. You can specify whether to cancel, terminate, or finish running jobs.
Error handling
Create a custom error-handling response to apply to all nodes that are configured to fail on error. Add inner nodes to the Error handling node to construct a subpipeline flow. The Error handling node also runs if a DataStage flow fails to compile or is not found.