Job configuration overlays reference

You can specify or change configuration parameters when you use the submitjob or updateoperators operations. The settings control submission constraints, resource (host) constraints and locations, and operator status. Parameters are captured and updated in the configuration JSON file. When you change the values in the job configuration overlay file, the changes override any previously set values.

Important: When you set a value in the job configuration overlay file, you might override application configuration settings that the application developer defined and the changes can potentially conflict with the application developer's intention.

A value that you set in a job configuration overlay file will override any corresponding value provided by a submission-time parameter or a corresponding value provided in the application. If your application requires submission-time values, you must still set those values in the application, even though they will be overridden.

Job parameters format

The following categories classify parameters as input or output related:

AttributeName	Type	Value
comment (optional)	String	Comment text.
jobConfigOverlays (optional)	Array `<JobConfigOverlay>`	Each element of the array contains the job configuration settings. Only the first jobConfigOverlay element is supported. Other elements in the array are ignored.
results (optional)	Results	Produced from the submitjob --preview command or from the updateoperators command. This parameter is output only and is ignored for job preview, update, or submission. Results can be successful, partially successful, or failure.

jobConfigOverlays: Single element array of jobConfigOverlays.
Results: Output from the submitjob --preview command or the updateoperators command. This parameter is output only and it is ignored for job preview, update, or submission. For parameters, see Results.

JobConfigOverlay

The following parameters define the contents of the job configuration overlay:

AttributeName	Type	Value	Operation
jobConfig (optional)	Object `<JobConfig>`	Job configuration values	`submitjob`
deploymentConfig (optional)	Object `<DeploymentConfig>`	Deployment configuration values.	`submitjob`
operatorConfigs (optional)	Object `<OperatorConfigs>`	Operator configuration values.	`submitjob`
configInstructions (optional)	Object `<ConfigInstructions>`	Configuration instructions.	`submitjob` `updateoperators`
operationConfig (optional)	Object `<OperationConfig>`	Operation configuration values.	`submitjob` `updateoperators`

JobConfig

The following parameters define the contents of the job configuration:

AttributeName	Type	Value	Operation
jobName (optional)	String	The name that is assigned to the job.	`submitjob`
jobGroup (optional)	String	The job group to use to control the permissions for the submitted job.	`submitjob`
submissionParameters (optional)	Array `<SubmissionParameter>`	Job-specific key: value submission parameters.	`submitjob`
dataDirectory (optional)	String	Specifies the location of the data directory.	`submitjob`
tracing (optional)	String	Specifies the trace setting for the processing elements. The valid values are `error`, `warn`, `info`, `debug`, and `trace`.	`submitjob`

SubmissionParameter

The following elements comprise the submission parameters:

AttributeName	Type	Value	Operation
name	String	Parameter name.	`submitjob`
value	String	Parameter value.	`submitjob`

Operation Config

The following elements comprise the contents of the operation configuration:

AttributeName	Type	Value	Operation
forcePeStopped (optional)	Boolean	`True`: If a currently running processing element is required to be stopped by the operation, this operation will stop the processing element. `False`: (default). If a processing element is required to stop, but it is not stopped, then the operation will fail.	`updateoperators`

DeploymentConfig

The following elements comprise the contents of the deployment configuration:

AttributeName	Type	Value	Operation
threadingModel (optional)	String	`manual`: Threads are manually placed by the application developer. All threads come from either operators, explicitly requested threaded ports, or processing element input ports. `automatic`: The SPL runtime decides on thread placement based on runtime information like necessary threads, number of operators that are available in the processing element, and number of available logical processors. `dynamic`: Each operator input port can be executed by any thread. The assignment of threads to input ports can change at runtime as can the number of available threads. `dedicated`: Each operator input port has a dedicated thread that processes all of the tuples on that input port, regardless of whether or not those operators have the threadedPort configuration.	`submitjob`
dynamicThreadingThreadCount (optional)	Number	Specifies the number of threads that scheduled ports should use.	`submitjob`
dynamicThreadingElastic (optional)	Boolean	Specifies whether the number of dynamic threads as well as the number of operators under the dynamic threading model can change at runtime to maximize throughput. Specify `true` to make elastic adjustment or `false` to keep the number of dynamic threads and the number of operators under the dynamic threading model static. Note: This setting can only be used if the application configuration threadingModel is set to `dynamic`.	`submitjob`
parallelRegionConfig (optional)	Object `<parallelRegionConfig>`	Configuration information for all the parallel regions in the job.	`submitjob`
numberOfResourcesToRequestForJob (optional)	Integer greater than 0.	Specifies the number of resources to be requested for the job. Resources are proportionally spread across the host pool definitions. Note: This setting can result in the total number or some of the resource host pools to be adjusted automatically.
cpuSpecificationForJob (optional)	Object `<ResourceSpecification>`	Defines the resource specification for the job. The `resourceRequest` value must be greater than zero and the `resourceLimit` value must greater than or equal to the `resourceRequest` value.
jobServiceConfig (optional)	Object `<JobServiceConfig>`	Configuration information for job service resources.	`submitjob`

ResourceSpecification

The following elements comprise the contents of a resource specification:

AttributeName	Type	Value	Operation
resourceRequest	Number	CPU: The number of CPUS for the resource CPU request. Decimal numbers are supported. For example: 1.75 CPUs Memory: The number of gibibytes for the memory request. For example: 1.75 Gi
resourceLimit	Number	CPU: The number of CPUS for the resource CPU limit. Decimal numbers are supported. For example: 1.75 CPUs Memory: The number of gibibytes for the memory limit. For example: 1.75 Gi

JobServiceConfig

The following elements comprise the contents of a job service configuration:

AttributeName	Type	Value	Operation
cpuSpecification (optional)	Object `<ResourceSpecification>`	Defines the CPU resource specification for each job service resource. The `resourceRequest` value must be greater than zero and the `resourceLimit` value must be greater than or equal to the `resourceRequest` value.
memorySpecification	Object `<ResourceSpecification>`	Defines the memory resource specification for each job service resource. The `resourceRequest` value must be greater than zero and the `resourceLimit` value must be greater than or equal to the `resourceRequest` value.

parallelRegionConfig

The following elements comprise the submission parameters:

AttributeName	Type	Value	Operation
fusionType	String	`channelIsolation`: Operators within a channel are fused into a processing element only with operators from the same channel. Operators outside the parallel region or from other channels in the same region are fused into different processing elements. One or more processing elements for a channel can be created depending on the fusion constraints. You can explicitly collocate an operator in a channel with an operator outside the region or from a different channel in the same region. A separate processing element is created if all of the explicitly colocated operators are not members of the same channel. `channelExlocation`: Operators within a channel are fused with operators from the same channel or with operators from outside the region. Operators from other channels in the same region are not fused into the same processing elements. One or more processing elements for a channel can be created depending on the fusion constraints. You can explicitly collocate an operator in a channel with an operator outside the region or from a different channel in the same region. A separate processing element is created if all of the explicitly colocated operators are not members of the same channel. If you use this value, you cannot change the width of a parallel region while the job is running. You would need to resubmit the job with the value set to `channelIsolation` or to `noChannelInfluence`. `noChannelInfluence`: (default). Inclusion in this parallelRegion has no impact on the fusion process.	`submitjob`

OperatorConfigs

The following parameters define the contents of the operator configurations:

AttributeName	Type	Value	Operation
operators (optional)	Array `<Operator>`	Configuration attributes for individual operators. Not all existing operators must be in the array, but only operators defined by the application are allowed. If an operator is referenced multiple times, the configuration updates are made one at a time in the order that they appear in the JSON file. The resulting configuration will be cumulative from the application of all the configurations.	`submitjob`
hostColocationGroups (optional)	Array `<LocationGroup>`	Each element of the array represents a hostColocationGroup, where each member of the group must be placed onto the same resource as other operators in this group. These groups are combined with groups that are defined in the application bundle.	`submitjob`
hostExlocationGroups (optional)	Array `<LocationGroup>`	Each element of the array represents a hostExlocationGroup, where each member of the group cannot be placed onto the same resource as any other operators in this group. These groups are combined with groups that are defined in the application bundle.	`submitjob`
partitionColocationGroups (optional)	Array `<LocationGroup>`	Each element of the array represents a partitionColocationGroup, where each member of the group must be fused together with other operators in the group into the same processing element. These groups are combined with groups that are defined in the application bundle.	`submitjob`
partitionExlocationGroups (optional)	Array `<LocationGroup>`	Each element of the array represents a partitionExlocationGroup, where each member of this group cannot be fused together with any other operators from this group into the same processing element. These groups are combined with groups that are defined in the application bundle.	`submitjob`
poolPlacementGroups (optional)	Array `<PoolPlacementGroup>`	Each element of the array represents a host pool definition.	`submitjob`

LocationGroup

The following parameters define the location groups:

AttributeName	Type	Value	Operation
groupName (optional)	String	Name of a location group.	`submitjob`: If this name matches the name of a group that is defined in the application bundle, this group replaces the group in the bundle. `updateoperators`: If this name matches the name of an existing group, it will replace it. All current members of the group must be included in the target zone, or an error is reported.
members (optional)	Array `<String>`	Each element of the array is a nameSpec of the operator or operators that are included in this group. If this attribute is not specified, or the array is empty, then a group with a matching groupName that is defined in the application bundle is removed.	`submitjob` `updateoperators`

PoolPlacementGroup

The following parameters define the details for pool placement groups:

AttributeName	Type	Value	Operation
name (optional)	String	Name of the host pool. For the `submitjob` operation: If this name matches the name of a group that is defined in the application bundle, this group replaces the group in the bundle.	`submitjob`
operatorsInPool (optional)	Array `<OperatorInPool>`	A specific operator can be specified in only one PoolPlacementGroup	`submitjob`
tags (optional)	Array `<String>`	Defines what the tagging requirements are for resources to be considered for membership of this pool.	`submitjob`
numberOfResourcesToRequest (optional)	Number	The `submitjob` operation will request this number of resources that match the requirements for the poolPlacementGroup parameter. If operators with different poolPlacementGroups are partition colocated together, the largest number for this attribute will be used.	`submitjob`
cpuSpecification (optional)	Object `<ResourceSpecification>`	Defines the resource specification that will be shared across all the processing elements that are referenced in this poolPlacementGroup. The `resourceRequest` value must be greater than zero and the `resourceLimit` value must greater than or equal to the `resourceRequest` value.	`submitjob`

OperatorInPool

The following parameters define the operator pools:

AttributeName	Type	Value	Operation
nameSpec	String	nameSpec of the operators in the pool.	`submitjob`

Operator

The following parameters define the details for operator configuration:

AttributeName	Type	Value	Operation
nameSpec	String	Specification of a name of an existing operator instance or instances in an application after UDP transformations. The name might contain zero or more '' characters in the specification. The '' represents 0 or more occurrences of any legal name character. For example, "Main..MyComp2.Filter[]" '[ ]' are normal characters, and do not have any regular expression meaning.	`submitjob`
partitionIsolation (optional)	Boolean	`true`: A processing element that contains this operator cannot be fused with any other operator. `false`: A processing element that contains this operator can be fused with other operators. If you set the value to `false`, it will not be specified in the preview output. If you specify this parameter, it will override the currently active value.	`submitjob`
partitionGroupIsolation (optional)	Boolean	`true`: The only operators that are allowed to be fused with this operator are those that have been configured with partitionColocation placement with it. `false`: No additional restrictions to what operators this operator can be fused with. If you set the value to `false`, it will not be specified in the preview output. If you specify this parameter, it will override the currently active value.	`submitjob`
hostIsolation (optional)	Boolean	`true`: A processing element that contains this operator cannot be placed on the same resource as any other processing element. `false`: A processing element that contains this operator can be placed on the same resource with other processing elements. If you set the value to `false`, it will not be specified in the preview output. If you specify this parameter, it will override the currently active value.	`submitjob`
cpuSpecification (optional)	Object `<ResourceSpecification>`	Defines the CPU resource specification that will be allocated to the processing element that contains this operator. If multiple operators in a processing element contain this attribute, the resources are accumulated. The `resourceRequest` value must be greater than zero and the `resourceLimit` value must greater than or equal to the `resourceRequest` value.	`submitjob` `updateoperators`
memorySpecification (optional)	Object `<ResourceSpecification>`	Defines the memory resource specification that will be allocated to the processing element that contains this operator. If multiple operators in a processing element contain this attribute, the resources are accumulated. The `resourceRequest` value must be greater than zero and the `resourceLimit` value must greater than or equal to the `resourceRequest` value.	`submitjob` `updateoperators`

ConfigInstructions

The following parameters define the details for the configuration instructions:

AttributeName	Type	Value	Operation
ignoreBundleRelationalPlacements (optional)	String	`PARTITION`: Clear all current partition relational placements from bundle (partitionIsolation, partitionColocationGroups, partitionExlocationGroups). `HOST`: Clear all current resource relational placements from bundle (hostIsolation, hostColocationGroups, hostExlocationGroups). `ALL`: Clear partition and resource relational placements. `NONE`: (default). Bundle relational placements are left as is. Tip: Be careful when you use this keyword because the effects can counteract the intentions of the application developer.	`submitjob`
ignoreBundleHostPlacements (optional)	Boolean	`true`: All operators are placed in the default pool. (default=`false`) Tip: Be careful when you use this keyword because the effects can counteract the intentions of the application developer.	`submitjob`
convertTagSet (optional)	Array `<tagConversion>`	Replace tags in the poolPlacementGroups with tags in the convertTagSet. If poolPlacementGroup.tags is a superset of originalTagSet, then tags in originalTagSet are removed from poolPlacementGroup.tags, and tags in targetTagSet are inserted in its place. If the originalTagSet is not specified, then a poolPlacementGroup with an empty tag set, known as a default host pool, has its tags changed to include targetTagSet. Tag conversions are performed in the order specified upon the results from the previous conversion.	`submitjob`
changeThreadedPortSettings (optional)	Object `<ThreadedPortSetting>`	Change settings for all threaded ports that are defined in the application.	`submitjob`
adjustmentSection (optional)	Array `<AdjustmentSubsection>`	Adjustment section definition. The operator set that makes up the adjustment section of the job is defined by the union of the composite set of operators of all the subsections. The composite set of operators for each subsection is defined by the union of the target operators and the operator sets that make up the target processing elements.	`submitjob`: Multiple sections are supported. `updateoperators`: Multiple sections are supported, but only one parallel region can be specified.

AdjustmentSubsection

The following parameters define the adjustment subsection:

AttributeName	Type	Value	Operation
targetParallelRegion (optional)	object `<TargetParallelRegion>`	Parallel region to be targeted.	`submitjob` `updateoperators`
cpuSpecificationToAdjust (optional)	Object `<ResourceSpecification>`	Specifies the number of CPUs to add to or remove from the target operators. The resourceRequest and resourceLimit values can be positive or negative or zero, and do not relate to each other.	`updateoperators`

TargetParallelRegion

The following parameters define the target parallel region:

AttributeName	Type	Value	Operation
regionName	NameSpec	Name of parallel region. The parallel region name is the logical name of the operator that the `@parallel` annotation was applied to. The asterisk (`*`) wildcard character is supported, except when the wildcard character resolves to multiple parallel regions. Ensure that any regular expression you use applies to only one level of a nested parallel region.	`submitjob` `updateoperators` Important: Changes to the width of multiple parallel regions at once is supported only for a `submitjob` operation. For an `updateoperators` operation, only one width change at a time is allowed. (You cannot change multiple widths of a running job.)
newWidth (optional)	Number	New width for parallel region. If the new width is same as current, then no adjustment is performed. The new width must be greater than 0. The fusionType parameter must be set to `noChannelInfluence` or to `channelIsolation`.	`submitjob` `updateoperators`

ThreadedPortSetting

The following parameter defines the threaded port settings:

AttributeName	Type	Value	Operation
queueSize	Number	Size of threaded port queue. Must be greater than or equal to 1.	`submitjob`

TagConversion

The following parameters define the tag conversion:

AttributeName	Type	Value	Operation
originalTagSet (optional)	Array `<String>`	An array of strings that represent a list of tags to convert from.	`submitjob`
targetTagSet	Array `<String>`	An array of strings that represent a list of tags to convert to.	`submitjob`

For complex transformation for tagging definitions that go beyond a one-to-one transformation, you can define a convertTagSet:

OriginalTagSet	TargetTagSet	poolPlacementGroupTags Before	poolPlacementGroupTags After	Notes
"A"	"X"	"A,B"	"X,B"	One to one
"A"	"X,Y"	"A,B"	"X,Y,B"	One to many
"A,B"	"Y"	"A,B,C"	"Y,C"	Many to one
"A,B"	"W,Y,Z"	"A,B,C"	"W,Y,Z,C"	Many to many
"A,B"	"Z"	"A,C,D" Not a superset; missing B.	"A,C,D"	No change, since not all tags exist in the "Before" set
" "	"X,Y"	" "	"X,Y"	Adding tags to default host pool.

Results

The following parameters define the submission results. These parameters are output only and are ignored on input:

AttributeName	Type	Value
placementResults (optional)	Array `<PlacementResult>`	Each element of the array describes a resulting processing element. This attribute is not specified if fusionErrors occur, but is specified if only placementErrors occur.
fusionErrors (optional)	Array `<FusionError>`	Each element of the array provides information on errors that occurred while the system is trying to fuse operators into a processing element.
placementErrors (optional)	Array `<PlacementError>`	Each element of the array provides information on errors that occurred while the system is trying to place a processing element onto a resource.
jobResources (optional)	Array `<ResourceConfig>`	Each element describes a resource that is being used by an operator in the target parallel region, a processing element in the target parallel region, or by a collateral operator. Collateral operators are operators that have a relationship with the operators in the targeted parallel region. The collateral operators are either fused in same processing element as a target operator or they connect with the targeted parallel region with a splitter or a merger.
collateralOperators (optional)	Array `<String>`	Names of the collateral operators that are affected by the `updateoperators` operation. Collateral operators are operators that have a relationship with the operators in the targeted parallel region. The collateral operators are either fused in same processing element as a target operator or they connect with the targeted parallel region with a splitter or a merger.
parallelRegionsInfo (optional)	Array `<ParallelRegionInfo>`	Each element provides information on a parallel region defined in the job.
deploymentConfig (optional)	Object `<DeploymentConfig>`	The deployment config information that is active for this job.

PlacementResult

If the fusion phase completes successfully, a placementResults element is returned whether the placementPhase was successful or not. If the placement phase fails, the hostPlacement attribute is not set. This partial return is useful if you want to use submitjob --preview to get feedback on the fusing phase before you have your test instance fully configured.

AttributeName	Type	Value
peId (optional)	Number	Processing element identifier.
names	Array `<String>`	Names of the operators to fuse together into this processing element.
tags (optional)	Array `<String>`	The host (resource) placement tags that are associated with this fused processing element.
hostPlacement (optional)	String	The target resource for processing element placement if the job was submitted. If placementPhase is not successful, then this attribute is not set. Since the current runtime situation is temporal and affects the resource selection, this hostPlacement might not be the same for subsequent `submitjob` operations.

ResourceConfig

AttributeName	Type	Value
resourceId	String	ID of the resource.
numberOfCores	Number	Number of CPU cores in the resource.
amountOfMemory	Number	Number of megabytes of memory in the resource.
networkBandwidth	Number	Number of kilobytes per second that are supported in the resource.
tags	Array `<String>`	Each element is a tag identifier that is assigned to the resource.

FusionError

The FusionError information describes the set of operators that were intended to be fused together, as well as what factors might have prevented successful fusion.

AttributeName	Type	Value
operatorNames	Array `<String>`	Names of operators that were not successfully fused.
detailedProblemMessages	Array `<Message>`	Messages that describe in detail the problems that were detected that prevented successful fusion. The messages are scoped to the collection of operators and not to a single operator. For example: CDISR3203E One or more of the operators are specified as partitionIsolate which prohibits from being fused with any other operator. CDISR3210E Different indexes [0, 1] within the 0 host pool were specified by the operators.

AttributeName

Type

Value

operatorNames

Array <String>

Names of operators that were not successfully fused.

detailedProblemMessages

Array <Message>

Messages that describe in detail the problems that were detected that prevented successful fusion. The messages are scoped to the collection of operators and not to a single operator. For example:

CDISR3203E One or more of the operators are specified as partitionIsolate which prohibits from being fused with any other operator.

CDISR3210E Different indexes [0, 1] within the 0 host pool were specified by the operators.

PlacementError

The processing element placement process is hierarchical in nature. A job contains one or more processing element colocation groups. A processing element colocation group contains one or more processing elements. A processing element contains one or more operators. The placement process cycles through various constraints to arrive at a candidate resource set for each placementNode. If the resource set is empty, that node in the placement tree cannot be placed.

The error messages are also hierarchical. In the previously described scenario, the reported error is not the last constraint that makes the set empty, but rather a composite message structure that describes the bigger picture of what causes the failure. At each node in the placement tree, there can be constraints that nominate or eliminate resources to the candidate resource set.

For example, each child node of a node nominates its candidate resource set to its parent's candidate set. The parent node eliminates all resources from its candidate set that are not included in each of its child's candidate set. That is, the parent set starts with the intersection of all child sets.

AttributeName	Type	Value
placementNodeInfo	PlacementNodeInfo	Contains information about the overall status of the placement node, including whether the node is able to be successfully placed.
contributingMessages	Array `<Message>`	A set of messages that describe the contributions that produce the resulting placementNodeInfo status. Some contributing messages describe why a set of resources is nominated to the candidate set, for example: CDISR3051I The following resources were included in the scheduling process because the resources are eligible to be included in the MyHostPool host pool by having the following host pool tags. The hosts are: 10.4.24.226, 10.4.40.236. The host pool tags that are shared by the hosts are: [Red,Blue]. Some contributing messages describe why a set of resources is eliminated from the candidate set, for example: CDISR3066I: Because the following resources are not supported by all the operators in the partition, these resources cannot be included in the scheduling process: 10.4.24.226, 10.4.40.247.
childContributions (optional)	Array `<PlacementError>`	A set of placement errors that detail how the child placement nodes contribute to these placement node results.

AttributeName

Type

Value

placementNodeInfo

PlacementNodeInfo

Contains information about the overall status of the placement node, including whether the node is able to be successfully placed.

contributingMessages

Array <Message>

A set of messages that describe the contributions that produce the resulting placementNodeInfo status. Some contributing messages describe why a set of resources is nominated to the candidate set, for example:

CDISR3051I The following resources were included in the scheduling process because the resources are eligible to be included in the MyHostPool host pool by having the following host pool tags. The hosts are: 10.4.24.226, 10.4.40.236. The host pool tags that are shared by the hosts are: [Red,Blue].

Some contributing messages describe why a set of resources is eliminated from the candidate set, for example:

CDISR3066I: Because the following resources are not supported by all the operators in the partition, these resources cannot be included in the scheduling process: 10.4.24.226, 10.4.40.247.

childContributions (optional)

Array <PlacementError>

A set of placement errors that detail how the child placement nodes contribute to these placement node results.

PlacementNodeInfo

AttributeName	Type	Value
success	Boolean	If the value is `true`, this node has at least one valid candidate resource. If the value is `false`, there are no candidate resources remaining for this placementNode, and placement fails for this node.
name (optional)	String	If present, the placement node is related to the operator that is named by this parameter.
peId (optional)	String	If present, the placement node is related to a processing element. The peOperatorSet attribute contains the related operator name set for this processing element.
peOperatorSet (optional)	Array `<String>`	If present, the placement node is related to a processing element, and this is the set of operator names that make up the processing element. The ID for this processing element is specified in the peID attribute.
colocatedPeSet (optional)	Array `<String>`	If present, this placement node is related to a colocated set of processing elements. This parameter expresses the array of processing element IDs as a string.
placementMessage	Message	A message that describes the placement status for this placement node. For example: CDISR3046E The processing element ID 3: op1 cannot be scheduled. For more information, see the messages that follow this one. CDISR3047I: A constraint on the op1 operator requires that it be scheduled from the following resources: 10.4.24.226, 10.4.40.236.. For more information, see the messages that follow this one.

Message

AttributeName	Type	Value
messageCode	String	The code for the message, for example, CDISR3200E.
substitutionTexts (optional)	Array `<String>`	The substitution text string for the message.
displayMessage (optional)	String	Human readable display message with substitutionTexts inserted and instantiated by the client in the client locale. This attribute is transient and is included to improve the readability of error messages in the JSON file.

ParallelRegionInfo

AttributeName	Type	Value
name	String	Name of the parallel region.
width	Number	Width of parallel region after an `updateoperators` operation. In preview mode, it shows the value as if the updates were actually done.