Enabling Dynamic Workload Management

You can enable Dynamic Workload Management to more quickly and efficiently run and manage work flows in DataStage®.

About this task

Dynamic Workload Management provides automatic load balancing to maximize throughput and minimize resource congestion. Additional compute nodes are deployed as needed to meet increasing work loads. When work loads decrease, the nodes are scaled back down until they are needed again.

Before you enable Dynamic Workload Management, you first enable PX runtime. PX runtime has the advantage of generating logs in a simple text file, which can be consumed by common log aggregation stacks. These logs are then more easily searchable for management and troubleshooting purposes.

You must use the web client experience to compile jobs if you enable the PX runtime. When the PX runtime is enabled, create a new project through the web client. This project will be enabled for the PX runtime.

You create a dynamic configuration file that automatically chooses which compute instance to run jobs on at run time, eliminating the need to make the configuration manually.

Procedure

Enable PX runtime.

Go to the conductor node:
```
$ oc exec -it is-en-conductor-0  bash
```

Create the configuration file that enables PX runtime:

echo "com.ibm.iis.cognition.engine.type=dspxnew" >> /opt/IBM/InformationServer/Server/Projects/runtimeProperties

Enable Workload Manager (WLM).

Edit the WLM config file on the conductor pod and change the existing lines in the file to look like the following example:

$ cd /opt/IBM/InformationServer/Server/DSWLM
$ vi dist/lib/wlm.config.xml

<!-- Enable kubernets metrics based functionality -->
    <Parameter name="metricsEnabled" value="true" />
    <!-- Refresh rate to gather metrics -->
    <Parameter name="metricsFrequency" value="2" />
    <!-- Boolean to enable/disable historic metrics for queuing -->
    <Parameter name="historicMetricQueuing" value="true" />
    <!-- Minimum number of compute pods to be running -->
    <Parameter name="computePodsMin" value="1" />
    <!-- Maximimum number of compute pods to be running. If max = min autoscaling is disabled -->
    <Parameter name="computePodsMax" value="3" />
    <!-- Autoscale frequency in seconds. Determines how often WLM will scale down unused pods -->
    <Parameter name="computePodsScaleFreq" value="60" />

The two main values that you adjust for your particular environment are "computePodsMin" and ""computePodsMax".

Restart the services in the conductor pod:

$ASBHOME/CognitiveDesignerEngine/stopCognitiveDesignerServer.sh

$ASBHOME/CognitiveDesignerEngine/startCognitiveDesignerServer.sh

$DSHOME/bin/uv -admin -stop

$DSHOME/bin/uv -admin -start

Create a dynamic configuration file.

See the following example of a typical dynamic configuration file:

{
        node "node1"
        {
                fastname "is-en-conductor-0.en-cond"
                pools "conductor"
                resource disk "/opt/IBM/InformationServer/Server/Datasets" {pools ""}
                resource scratchdisk "/opt/IBM/InformationServer/Server/Scratch" {pools ""}
        }
        node "node2"
        {
                fastname "$pod"
                pools ""
                resource disk "/opt/IBM/InformationServer/Server/Datasets" {pools ""}
                resource scratchdisk "/opt/IBM/InformationServer/Server/Scratch" {pools ""}
        }
        node "node3"
        {
                fastname "$pod"
                pools ""
                resource disk "/opt/IBM/InformationServer/Server/Datasets" {pools ""}
                resource scratchdisk "/opt/IBM/InformationServer/Server/Scratch" {pools ""}
        }
   }

What to do next

After you enable PX runtime, you can use the dxjob command to start jobs, stop jobs, and retrieve information about job runs. For more information, see Commands for controlling data transformation jobs.