Dynamic workload management

Use dynamic workload management to more quickly and efficiently run and manage work flows in DataStage®.

Dynamic workload management provides automatic load balancing to maximize throughput and minimize resource congestion. More compute pods are deployed as needed to meet increasing work loads. When work loads decrease, the pods are scaled back down until they are needed again.

Dynamic workload management also handles, among other tasks, queuing jobs when instance limits are reached, allocating compute pods for APT_CONFIG_FILE environment variables, and auto-scaling compute pods.

Dynamic workload management generates apt config files in real time based on the resources (memory and CPUs, not necessarily number of jobs) that are available on the currently running compute pods. If no compute pods are available because they are all at their resource limit and auto scaling is disabled, then jobs are queued.

You can specify a configuration file by setting the APT_CONFIG_FILE environment variable. Because you can have many different apt config files on disk, the environment variable tells the job which apt config file to use. If you don't specify APT_CONFIG_FILE, a configuration file is automatically generated for you when the job runs. An APT_CONFIG_FILE environment variable defines the logical nodes that a job uses to run along with disk resources such as scratch space. You can create a dynamic configuration file that automatically chooses which compute pod to run jobs on at run time, eliminating the need to make the configuration manually.

Dynamic workload management is automatically used by default in DataStage versions 4.0.2 and later.