DataStage optimized runner

The DataStage optimized pipeline runner is an alternative execution engine that is designed to streamline the orchestration of nested pipelines within a single job execution. Instead of relying on REST API calls, it starts generated Python code directly within the engine runtime. This approach significantly reduces overhead and resource utilization due to its lightweight architecture.

Optimized pipeline runner setting

You can enable the optimized pipeline runner from project-level settings.
  1. Create or open an existing project.
  2. In the Manage tab, choose the Pipeline section from the left-side options.
  3. Go to the DataStage optimized pipeline runner section.
  4. Select the Run using optimized pipeline runner option. Click Save.
  5. Open any pipeline from Assets > Flows > Pipelines.
  6. Open an existing pipeline. Find a Run pipeline button, and click it. Your job is run as an optimized pipeline.
All new pipelines run using the optimized pipeline runner when the Run using optimized pipeline runner option is enabled in the project settings.

Tracking the pipeline status

You can see the compile status of pipelines that use the DataStage optimized runner by viewing the Project assets tab. Pipelines are compiled only after you start a pipeline run from the UI. You can see three statuses: stale, not compiled, or successfully compiled.

Running a job as an optimized pipeline by using cpdctl

You can run a job as an optimized pipeline by using cpdctl. For more information, see: Run pipelines using optimized runner.