Enabling auto-scaling in DataStage

You can enable auto-scaling by editing the dynamic workload management configuration file. When auto-scaling is enabled, dynamic workload management adds or subtracts compute pods according to the workload.

About this task

The minimum and maximum number of compute pods is set in the configuration file wlm.config.xml. The default value for both the minimum and maximum number of pods is 2. To enable auto-scaling, you edit the values for the minimum and maximum number of pods so that the values are not equal to each other.

Procedure

  1. Log in to your Red Hat® OpenShift® cluster as a project administrator: 
    oc login -u kubeadmin -p xxxxx-xxxx-xxxx
  2. Find the runtime instance that you want to update. For example, to update the default instance: 
    oc get pods | grep ds-px-default-ibm-datastage-px-runtime
  3. Open a shell on the runtime pod: 
    oc rsh ds-px-default-ibm-datastage-px-runtime-7b5b7975b8-jnzbg
  4. Change directory to the workload management home directory, for example: 
    cd /opt/ibm/PXService/Server/DSWLM
  5. Stop workload management: 
    ./stopwlm.sh
  6. Edit the configuration file: 
    nano /px-storage/config/wlm/wlm.config.xml
  7. Change the minimum and maximum values for the number of pods so they are not equal to each other, similar to the following example: 
        <!-- Minimum number of compute pods to be running -->
    <Parameter name="computePodsMin" value="2" />
    <!-- Maximimum number of compute pods to be running. If max = min autoscaling is disabled -->
    <Parameter name="computePodsMax" value="4" />
  8. Start workload management: 
    nohup ./startwlm.sh &
    Example results:
    [1] 16034 nohup: ignoring input and appending output to 'nohup.out'
  9. Confirm that workload management is running: 
    ps -ef | grep WLM | grep -v grep
    Example results:
    1000630+ 16034 15905 3 15:02 pts/0 00:00:00 ./../../jdk/bin/java -Xmx2048m -classpath ./dist/lib/commons-lang-2.6.jar:./dist/lib/commons-codec-1.15.jar:./../../ASBNode/lib/java/jsr311-api-1.1.1.jar:./../../ASBNode/lib/java/slf4j-api-1.6.1.jar:./../../ASBNode/lib/java/wink-1.2.1-incubating.jar:./../../ASBNode/lib/java/wink-client-1.2.1-incubating.jar:./../../ASBNode/lib/java/wink-common-1.2.1-incubating.jar:./../../ASBNode/lib/java/wink-server-1.2.1-incubating.jar:./dist/lib/wlm.jar:./dist/lib/wlmstart.jar com.ibm.iis.common.wlm.service.server.ds.DSWLMServer

What to do next

For more settings that you can change by editing the configuration file, see Configuring workload management policies for DataStage.