Enabling multiple conductor (PX runtime) pods in DataStage

You can increase compute availability and improve workload balancing by increasing the number of DataStage PX runtime conductor pods.

About this task

Your current resource limit (most commonly CPU) may constrain the number of concurrent jobs that can be run, which may lead to jobs queuing. You can scale up conductor pods to improve computing performance. Scaling can also increase computing availability for concurrently compiling a large number of transformers.

Compute pods can scale horizontally and vertically, with dynamic workload management distributing the workload across the available compute pods. By default, runtime instances are created with one DataStage PX runtime conductor pod. Each job is executed by one conductor pod across multiple compute pods. You can increase compute availability by increasing the number of PX runtime conductor pods. New jobs can be submitted as long as a conductor pod is available. Multiple pods from the same service will avoid deploying on the same worker node.

Procedure

Set the number of replicas for px-runtime (DataStage PX instance conductor) to 2 with the following command. You can scale the replicas to 2 or more.
```
oc scale deploy <px-runtime-deployment> --replicas=2
```
Example:
```
oc scale deploy px-instance-ibm-datastage-px-runtime --replicas=2
```

Modify the DataStage PX runtime instance custom resource (CR) to update the default. Otherwise, the next update to the CR will revert the change.

# retrieve pxruntime cr
oc get pxruntime

# edit pxruntime cr
oc edit pxruntime <cr-name>

spec:
  ignoreForMaintenance: false  # set to true to ignore refresh
  scale:                                        # override the t-shirt size
    pxcompute:           
      replicas: 2
      resources:
        limits:
          cpu: 3
          memory: 8Gi
        requests:
          cpu: 1
          memory: 2Gi
    pxruntime:
      replicas: 2
      resources:
        limits:
          cpu: 2
          memory: 4Gi
        requests:
          cpu: 500m
          memory: 2Gi