Changing the degree of parallelism at run time

You can change the width of a parallel region while the job is running.

Increasing the parallel region width can improve application throughput, but can use more resources. You can make more resources available to the parallel region when you change its width. Resources that are not needed are released automatically. Alternatively, you can decrease the parallel region width to reduce the required resources.

Restrictions:

• You cannot modify the parallel region width of a running job when the original job was submitted with the parallel region fusion (fusionType) parameter set to channelExlocation (prevent fusion across channels). You must resubmit the job with the fusionType parameter set to noChannelInfluence (default) or to channelIsolation.
• You must have add and delete authority for the appropriate jobgroup_name instance object. By default, the DomainAdministrator and InstanceAdministrator roles have this authority. The user who submits the job also has this authority.
• You can add resources only if the instance's resource allocation mode is scoped to the job (instance.applicationResourceAllocationMode=job).
• Checkpointing is not supported in the parallel region, including collateral operators.
• Consistent regions are not supported in the parallel region, including collateral operators.

Caveats:

• When you change the width of a parallel region, the operators within the region are stopped and restarted, as well as the PEs that contain them. If those operators are in PEs that also contain operators from outside of the parallel region, those operators from outside of the parallel region are also stopped and restarted. This operation can result in the loss of tuples and the loss of operator state.
• Which operators are in each PE can change. The PE ID might not be consistent for the lifetime of an operator. You can use the capturestate command to determine the PE metrics or the streamtool lspes --long command to see the PE IDs.
• The operation might take some time, depending on the number of operators that are affected.
• Confirm with the application developer that changing the parallel region width will not interfere with any settings in the running application. For example, if the region includes sink operators that are matched with an externally partitioned system.
• You can change the parallel width at any level of a nested parallel region, but the change will apply to all parallel replicas at that level. For information, see Nested parallel regions.

You can change the parallel region width of a running job in the following ways:

• Run the streamtool updateoperators command. See information about the --parallelRegionWidth and the --addJobResources parameters: streamtool updateoperators.
• Create a job configuration overlay file that includes the targetParallelRegion and the numberOfResourcesToBeAdded parameters. Then you can run the streamtool updateoperators command with those parameters. For information about job configuration overlay, see Job configuration overlays and the Job configuration overlays reference.
• Streams Console: The action can be launched from multiple locations. One way is from the Application Dashboard. Open the Streams Graph and select the job. Open the job properties by hovering your mouse over the information (i) icon, and then click Modify Parallel Region.
• Streams Studio: In the Instance Graph, right-click the operator that has a parallel region, and then select Modify parallel region width.