IBM Streams 4.2.1

Job configuration overlays reference

You can specify or change configuration parameters when you use the submitjob command or the previewsubmitjob command. The settings control submission constraints, host constraints and locations, and operator status. Parameters are captured and updated in the configuration JSON file.

Submit job parameters format

The following categories classify parameters as input or output related:

Attribute	Type	Value
comment (optional)	String	Comment text.
jobConfigOverlays (optional)	Array `JobConfigOverlay`	Each element of the array contains the job configuration settings. Only the first jobConfigOverlay element is currently supported. Other elements in the array are ignored.
results (optional)	Results	Produced from the previewsubmitjob command. This parameter is output only and is ignored for job preview or submission. Results can be successful, partially successful, or failure.

jobConfigOverlays: Single element array of jobConfigOverlays.
Results: Array of strings that is produced from the previewsubmitjob command. This parameter is output only and is ignored for job preview or submission. Each element of the array describes a resulting PE.

JobConfigOverlay

The following parameter defines the contents of the job configuration overlay:

Attribute	Type	Value
jobConfig (optional)	Object `JobConfig`	Job configuration values
deploymentConfig	Object `DeploymentConfig`	Deployment configuration values.
operatorConfigs (optional) (ignored when fusionScheme=legacy)	Object`OperatorConfigs`	Operator configuration values.
configInstructions (optional)	Object`ConfigInstructions`	Configuration instructions.

JobConfig

The following parameter defines the contents of the job configuration:

Attribute	Type	Value
jobName (optional)	String	The name that is assigned to the job.
jobGroup (optional)	String	The job group to use to control permissions for the submitted job.
submissionParameters (optional)	Array`SubmissionParameters`	Job-specific key: value submission parameters.
dataDirectory (optional)	String	Specifies the location of the data directory.
preloadApplicationBundles (optional)	Boolean	Specifies whether to preload the job onto all resources in the instance, even if the job is not currently needed on each. The valid values are true and false; false is the default value. Preloading the job can improve the performance if the PEs are relocated to a new resource.
tracing (optional)	String	Specifies the trace setting for the PEs. The valid values are `error`, warn, `info`, `debug`, and `trace`.

SubmissionParameter

The following elements comprise the submission parameters:

Attribute	Type	Value
name	String	Parameter name.
value	String	Parameter value.

DeploymentConfig

The following elements comprise the contents of the deployment configuration:

Attribute	Type	Value
fusionScheme (optional)	String	`automatic`: (default) At submission time, IBM® Streams decides on a number of PEs that are appropriate to the job and the instance. `manual`: At submission time, the user must provide a target number of PEs for each job. `legacy`: At submission time, IBM Streams uses the fusion behavior from prior to Version 4.2.1.
fusionTargetPeCount (optional) (requires fusionScheme=manual orinstance.fusionScheme=manual)	Number	Use this option to set the number of PEs to create. fusionScheme must be set to `manual`.
placementScheme	String	`legacy`: The estimated load of PEs to be placed is fitted in with existing ldavg numbers of hosts, adjusted for the number of cores in hosts. `balancedJob`: PEs are distributed across candidate hosts proportionally to the number of cores on the hosts. PEs from previous jobs are ignored. `balancedInstance`: (default) PEs are distributed across candidate hosts proportionally to the number of cores on the hosts, taking into account PE placements from previous jobs. For both balanced schemes, the balancing pass selects the candidate host that has the lowest PE to core ratio. If there is a tie, the balancing process chooses the host with the lowest ldavg, normalized to number of cores.
threadingModel	String	`manual`: Threads are manually placed by the application developer. All threads come from either operators, explicitly requested threaded ports, or PE input ports. `automatic`: The SPL runtime decides on thread placement based on runtime information like necessary threads, number of operators that are available in the PE, and number of available logical processors. `dynamic`: Each operator input port can be executed by any thread. The assignment of threads to input ports can change at runtime as can the number of available threads. `dedicated`: Each operator input port has a dedicated thread that processes all of the tuples on that input port, regardless of whether or not those operators have the threadedPort configuration.
dynamicThreadingThreadCount (optional)	Number	Specifies the number of threads that scheduled ports should use. Inherits its value from instance.dynamicThreadingThreadCount, if set. Valid values are integers greater than or equal to 1. This setting can only be used if the configuration threadingModel is set to `dynamic`. If dynamicThreadingElastic is set to `true`, the number of threads might change at runtime.
dynamicThreadingElastic (optional)	Boolean	Specifies if elasticity is on or off for dynamically threaded ports. Valid values are `true` for on, and `false` for off. If elasticity is on, the number of dynamic threads can change at runtime to maximize throughput. This setting can only be used if the application configuration threadingModel is set to `dynamic`.
parallelRegionConfig (optional) (ignored if fusionScheme=legacy)	Object `parallelRegionConfig`	Configuration information for all the ParallelRegions in the job.

parallelRegionConfig

The following elements comprise the submission parameters:

Attribute	Type	Value
fusionType	String	`channelIsolation`: Operators within a channel are only fused with other operators within the channel. Whether this results in a single PE or multiple PEs depends on the fusionScheme PE calculation algorithms. A user can still explicitly co-locate an operator from outside the channel with one in the channel. If operators from different channel are explicitly co-located together, they are separated into their own PE. `channelExlocation`: Operators from different channels of this parallel region are not allowed to be fused together. But operators from outside of this channel, that are not in a different channel of this same region, are allowed to be fused with operators within the channel. A user can explicitly co-locate operators from different channels together, but PEs are separate from other operators. `noChannelInfluence`: (default) Inclusion in this parallelRegion has no impact on the fusion process.

OperatorConfigs

The following parameters define the contents of the operator configurations:

Attribute	Type	Value
operators (optional)	Array `Operator`	Configuration attributes for individual operators. Not all existing operators must be in the array, but only operators that are defined by the application are allowed, and they can exist only one time in this array.
hostColocationGroups (optional)	Array `LocationGroup`	Each element of the array represents a hostColocationGroup, where each member of the group must be placed onto the same host as other operators in this group. These groups are combined with groups that are defined in the application bundle.
hostExlocationGroups (optional)	Array`LocationGroup`	Each element of the array represents a hostExlocationGroup, where each member of the group cannot be placed onto the same host as any other operators in this group. These groups are combined with groups that are defined in the application bundle.
partitionColocationGroups (optional)	Array`LocationGroup`	Each element of the array represents a partitionColocationGroup, where each member of the group must be fused together with other operators in the group into the same PE. These groups are combined with groups that are defined in the application bundle.
partitionExlocationGroups (optional)	Array`LocationGroup`	Each element of the array represents a partitionExlocationGroup, where each member of this group cannot be fused together with any other operators from this group into the same PE. These groups are combined with groups that are defined in the application bundle.
poolPlacementGroups (optional)	Array `PoolPlacementGroup`	Each element of the array represents a hostpool definition.

LocationGroup

The following parameters define the location groups:

Attribute	Type	Value
groupName (optional)	String	Name of a location group. If this name matches the name of a group that is defined in the application bundle, this group replaces the group in the bundle.
members (optional)	Array `String`	Each element of the array is a nameSpec of the operator or operators that are included in this group. If this attribute is not specified, or the array is empty, then a group with a matching groupName that is defined in the application bundle is removed.

PoolPlacementGroup

The following parameters define the details for pool placement groups:

Attribute	Type	Value
name (optional)	String	Name of the hostpool.
operatorsInPool (optional)	Array `OperatorInPool`	A specific operator can be specified in only one PoolPlacementGroup or in the declaredHost attribute of the Operator.
membershipMode (optional)	String	`SHARED`: Hosts that are members of this pool are allowed to be in other pools as well. This setting is the default. `EXCLUSIVE`: Hosts that are members of this pool are not allowed to be in any other pools, whether this job or any other jobs. The value for size must be specified.
size (optional)	Number	Optional number. Defines the number of hosts that are allowed in the host pool. If specified, the number must be greater than 0. If declaredHosts size is greater than 0, then the value that is specified for size is ignored.
tags (optional) (exclusive with declaredHosts)	Array`String`	Defines what the tagging requirements are for hosts to be considered for membership of this pool.
declaredHosts (optional) (exclusive with tags) (exclusive with membershipMode=`Exclusive`)	Array`String`	Defined hosts that are members of this host pool.

OperatorInPool

The following parameters define the operator pools:

Attribute	Type	Value
nameSpec	String	nameSpec of the operators in the pool.
inPoolIndex (optional)	Number	Optional number. The index within the host pool, for example, the fourth host listed in the host pool. This value can be specified only as part of a PoolPlacementGroup that has a size specified. The index number must be smaller than the specified size.

Operator

The following parameters define the details for operator configuration:

Name	Type	Value
nameSpec	String	Specification of a name of an existing operator instance in the application after UDP transformations. The name might contain zero or more '' characters in the specification. The '' represents 0 or more occurrences of any legal name character. For example, "Main..MyComp2.Filter[]" '[ ]' are normal characters, and do not have any regular expression meaning.
restartable (optional)	Boolean	`true`: A PE that contains this operator is able to be restarted. (default) `false`: A PE that contains this operator is not able to be restarted.
relocatable (optional)	Boolean	`true`: A PE that contains this operator is able to be relocated. (default) `false`: A PE that contains this operator is not able to be relocated.
partitionIsolation (optional)	Boolean	`true`: A PE that contains this operator cannot be fused with any other operator. `false`: A PE that contains this operator can be fused with other operators. (default)
hostIsolation (optional)	Boolean	`true`: A PE that contains this operator cannot be placed on the same host as any other PE. `false`: A PE that contains this operator can be placed on the same host with other PEs. (default)
declaredHost (optional)	String	A PE that contains this operator must be placed on this specific host. If this attribute is specified, then the operator cannot be included in any of the PoolPlacement sections.

ConfigInstructions

The following parameters define the details for configuration instructions

Name	Type	Value
ignoreBundleRelationalPlacements(optional) (ignored when fusionScheme=legacy)	String	`PARTITION`: Clear all partition relational placements from bundle (partitionIsolation, partitionColocationGroups, partitionExlocationGroups). `HOST`: Clear all host relational placements from bundle (hostIsolation, hostColocationGroups, hostExlocationGroups). `ALL`: Clear partition and host relational placements. `NONE`: Bundle relational placements are left as is (default). Tip: Be careful when you use this keyword because the effects can counteract the intentions of the application developer.
ignoreBundleHostPlacements (optional) (ignored when fusionScheme=legacy)	Boolean	`true`: All operators are placed in the default pool. (default=false) Tip: Be careful when you use this keyword because the effects can counteract the intentions of the application developer.
convertTagSet (optional)	Array `tagConversion`	Replace tags in the poolPlacementGroups with tags in the convertTagSet. If poolPlacementGroup.tags is a superset of originalTagSet, then tags in originalTagSet are removed from poolPlacementGroup.tags, and tags in targetTagSet are inserted in its place. If the originalTagSet is not specified, then a poolPlacementGroup with an empty tag set, known as a default hostpool, has its tags changed to include targetTagSet. The parameter does not convert tags within the jobConfigOverlay.
changeThreadedPortSettings (optional)	Object `ThreadedPortSetting`	Change settings for all threaded ports that are defined in the application.

ThreadedPortSetting

The following parameter defines the threaded port settings:

Attribute	Type	Value
queueSize	Number	Size of threaded port queue. Must be greater than or equal to 1.

TagConversion

The following parameters define the tag conversion:

Attribute	Type	Value
originalTagSet	Array `String`	An array of strings that represent a list of tags to convert from.
targetTagSet	Array `String`	An array of strings that represent a list of tags to convert to.

For complex transformation for tagging definitions that go beyond a one-to-one transformation, you can define a convertTagSet:

OriginalTagSet	TargetTagSet	poolPlacementGroupTags Before	poolPlacementGroupTags After	Notes
"A"	"X"	"A,B"	"X,B"	One to one
"A"	"X,Y"	"A,B"	"X,Y,B"	One to many
"A,B"	"Y"	"A,B,C"	"Y,C"	Many to one
"A,B"	"W,Y,Z"	"A,B,C"	"W,Y,Z,C"	Many to many
"A,B"	"Z"	"A,C,D" Not a superset; missing B.	"A,C,D"	No change, since not all tags exist in the "Before" set
" "	"X,Y"	" "	"X,Y"	Adding tags to default hostpool.

Results

The following parameters define the submission results. These parameters are output only and are ignored on input:

Attribute	Type	Value
placementResults (optional)	Array `PlacementResult`	Each element of the array describes a resulting PE. This field is not specified if fusionErrors occur, but is specified if only placementErrors occur.
fusionErrors (optional)	Array `FusionError`	Each element of the array provides information on errors that occurred while the system is trying to fuse operators into a PE.
placementErrors (optional)	Array `PlacementError`	Each element of the array provides information on errors that occurred while the system is trying to place a PE onto a host.

PlacementResult

If the fusion phase completes successfully, a placementResults element is returned whether the placementPhase was successful or not. If the placement phase fails, then the hostPlacement field is not set. This partial return is useful if you want to use previewsubmitjob get feedback on the fusing phase before you have your test instance fully configured.

Attribute	Type	Value
peId (optional)	Number	PE identifier.
names	Array `String`	Names of the operators to fuse together into this PE.
tags (optional) (exclusive with declaredHosts)	Array `String`	The host placement tags that are associated with this fused PE.
declaredHosts (optional) (exclusive with tags)	Array `String`	Any declared host that this PE is associated with.
hostPlacement (optional)	String	The target host for PE placement if the job was submitted. If placementPhase is not successful, then this field is not set. Since the current runtime situation is temporal and affects the host selection, this hostPlacement might not be the same for subsequent `submitjob` operations.

FusionError

The FusionError information describes the set of operators that were intended to be fused together, as well as what factors might have prevented successful fusion.

Attribute	Type	Value
operatorNames	Array `String`	Names of operators that were not successfully fused.
detailedProblemMessages	Array `Message`	Messages that describe in detail the problems that were detected that prevented successful fusion. The messages are scoped to the collection of operators and not to a single operator. For example: CDISR3203E One or more of the operators are specified as partitionIsolate which prohibits from being fused with any other operator. CDISR3210E Different indexes [0, 1] within the 0 hostpool were specified by the operators.

Attribute

Type

Value

operatorNames

Array String

Names of operators that were not successfully fused.

detailedProblemMessages

Array Message

Messages that describe in detail the problems that were detected that prevented successful fusion. The messages are scoped to the collection of operators and not to a single operator. For example:

CDISR3203E One or more of the operators are specified as partitionIsolate which prohibits from being fused with any other operator.

CDISR3210E Different indexes [0, 1] within the 0 hostpool were specified by the operators.

PlacementError

The PE placement process is hierarchical in nature. A job contains one or more PE colocation groups. A PE colocation group contains one or more PEs. A PE contains one or more operators. The placement process cycles through various constraints to arrive at a candidate host set for each placementNode. If the host set is empty, that node in the placement tree cannot be placed.

The error messages are also hierarchical. In the previously described scenario, the reported error is not the last constraint that makes the set empty, but rather a composite message structure that describes the bigger picture of what causes the failure. At each node in the placement tree, there can be constraints that nominate or eliminate hosts to the candidate host set.

For example, each child node of a node nominates its candidate host set to its parent's candidate set. The parent node eliminates all hosts from its candidate set that are not included in each of its child's candidate set. That is, the parent set starts with the intersection of all child sets.

Attribute	Type	Value
placementNodeInfo	PlacementNodeInfo	Contains information about the overall status of the placement node, including whether the node is able to be successfully placed.
contributingMessages	Array `Message`	A set of messages that describe the contributions that produce the resulting placementNodeInfo status. Some contributing messages describe why a set of hosts is nominated to the candidate set, for example: CDISR3051I The following hosts were included in the scheduling process because the hosts are eligible to be included in the MyHostPool host pool by having the following host pool tags. The hosts are: 10.4.24.226, 10.4.40.236. The host pool tags that are shared by the hosts are: [Red,Blue]. Some contributing messages describe why a set of hosts is eliminated from the candidate set, for example: CDISR3066I: Because the following hosts are not supported by all the operators in the partition, these hosts cannot be included in the scheduling process: 10.4.24.226, 10.4.40.247.
childContributions	Array `PlacementError`	A set of placement errors that detail how the child placement nodes contribute to these placement node results.

Attribute

Type

Value

placementNodeInfo

PlacementNodeInfo

Contains information about the overall status of the placement node, including whether the node is able to be successfully placed.

contributingMessages

Array Message

A set of messages that describe the contributions that produce the resulting placementNodeInfo status. Some contributing messages describe why a set of hosts is nominated to the candidate set, for example:

CDISR3051I The following hosts were included in the scheduling process because the hosts are eligible to be included in the MyHostPool host pool by having the following host pool tags. The hosts are: 10.4.24.226, 10.4.40.236. The host pool tags that are shared by the hosts are: [Red,Blue].

Some contributing messages describe why a set of hosts is eliminated from the candidate set, for example:

CDISR3066I: Because the following hosts are not supported by all the operators in the partition, these hosts cannot be included in the scheduling process: 10.4.24.226, 10.4.40.247.

childContributions

Array PlacementError

A set of placement errors that detail how the child placement nodes contribute to these placement node results.

PlacementNodeInfo

Attribute	Type	Value
success	Boolean	If the value is `true`, this node has at least one valid candidate host. If the value is `false`, there are no candidate hosts remaining for this placementNode, and placement fails for this node.
name (optional) (exclusive with peOperatorSet and colocatedPeSet)	String	If present, the placement node is related to the operator that is named by this parameter.
peId (optional) (exclusive with name and colocatedPeSet	String	If present, the placement node is related to a PE. The peOperatorSet field contains the related operator name set for this PE.
peOperatorSet (optional) (exclusive with name and colocatedPeSet)	Array `String`	If present, the placement node is related to a PE, and this is the set of operator names that make up the PE. The ID for this PE is specified in the peID field.
colocatedPeSet (optional) (exclusive with name and peOperatorSet)	Array `String`	If present, this placement node is related to a colocated set of PEs. This parameter expresses the array of PE IDs as a string.
placementMessage	Message	A message that describes the placement status for this placement node. For example: CDISR3046E The processing element ID 3: op1 cannot be scheduled. For more information, see the messages that follow this one. CDISR3047I: A constraint on the op1 operator requires that it be scheduled from the following hosts: 10.4.24.226, 10.4.40.236.. For more information, see the messages that follow this one.

Message

Attribute	Type	Value
messageCode	String	The code for the message, for example, CDISR3200E .
substitutionTexts )	Array `String`	The substitution text string for the message.
displayMessage (optional)	String	Human readable display message with substitutionTexts inserted and instantiated by the client in the client locale. This field is transient and is included to improve the readability of error messages in the JSON file. The field is optional for the `previewsubmitjob` client.