IBM Streams 4.3.0

Job configuration overlays

A job configuration overlay file is a JSON file that contains name-value pairs for job configuration parameters. You can use a job configuration overlay file to define, save, and distribute the submission-time configuration or to change the configuration of a running job.

If you want to modify a few settings, you can create a job configuration overlay file that includes only the relevant sections and parameters. Only the settings that are specified in the job configuration overlay file are changed. The settings in the job configuration overlay file overwrite any previously set values.

Previewing or submitting a job with the job configuration overlay file

You can manually create a job configuration overlay file, or you can generate a job configuration overlay file when you preview how a job will be submitted to your instance. For more information about generating a job configuration overlay file from a preview, see Previewing a job before you deploy it.

The job configuration overlay file that is generated when you run a preview includes all of the settings that were configured for the job. For complex applications, the job configuration overlay file might be long and contain many entries.

A job configuration overlay file is one way that you can specify submission-time configuration parameters for a job before you deploy a stream processing application to your environment. (Alternatively, you can specify submission-time configuration parameters from the IBM® Streams Console, from the Streams Studio, or by using the streamtool submitjob command.) However, unlike manually specifying submission-time configuration parameters when you submit a job, a job configuration overlay file enables you to reuse or share a configuration. It also reduces the likelihood of entering information incorrectly if you want to specify submission-time configuration parameters from the interactive streamtool interface.

For information about how you can use submission-time configuration parameters to improve the runtime performance of the application without recompiling it, see Configuring submission-time parameters to optimize job deployment.

You can use a job configuration overlay file when you preview or submit a job in the following ways:

From IBM Streams Studio, specify the job configuration overlay file from the SPL Application tab when you run or preview a job.
From Streams Console, import the job configuration overlay file when you run or preview a job. You can import it by clicking the Import icon at the top of the page and selecting Import job configuration file.
From the interactive streamtool interface, choose the appropriate option:
- To preview a job with a job configuration overlay file, run the streamtool submitjob command with the --preview option and specify the name of your job configuration overlay file, for example:
```
streamtool submitjob --preview -g jobConfigOverlay.json myAppBundle.sab
```
  Note: When you preview the job, a new job configuration overlay file in JSON format is automatically generated. The default file name is appName_config.json, where appName is the name of your application. You can change the file name by using the --out_jobConfig option.
- To run a job with a job configuration overlay file, run the streamtool submitjob command and specify the name of your job configuration overlay file:
```
streamtool submitjob -g jobConfigOverlay.json myAppBundle.sab
```

Modifying a running job with a job configuration overlay file

You can use the job configuration overlay file with the streamtool updateoperators command to change the width of a parallel region in a running job. For example:

streamtool updateoperators myjob -g jobConfigOverlay.json

Important: Changes to the width of multiple parallel regions at once is supported only for a submitjob operation. For an updateoperators operation, only one width change at a time is allowed. (You cannot change multiple widths of a running job.)

Structure of a job configuration overlay file

The following outline shows the order of the sections in a job configuration overlay file:

comment
jobConfigOverlays
     jobConfig
          jobName
          jobGroup
          submissionParameters
          dataDirectory
          preloadApplicationBundles
          tracing
     deploymentConfig
          fusionScheme
          fusionTargetPeCount
          placementScheme
          threadingModel
          dynamicThreadingThreadCount
          dynamicThreadingElastic
          parallelRegionConfig
               fusionType
          jobResourceSharing
          proposedOperatorsPerResource
     operatorConfigs
          operators
               nameSpec
               restartable
               relocatable
               partitionIsolation
               partitionGroupIsolation
               hostIsolation
               declaredHost
          hostColocationGroups
               groupName
               members
          hostExlocationGroups
               groupName
               members
          partitionColocationGroups
               groupName
               members
          partitionExplocationGroups
               groupName
               members
          poolPlacementGroups
               name
               OperatorsInPool
                    nameSpec
                    inPoolIndex
               membershipMode
               size
               tags
               declaredHosts
               numberOfResourcesToRequest
     configInstructions
          ignoreBundleRelationalPlacements
          ignoreBundleHostPlacements
          convertTagSet
               originalTagSet
               targetTagSet
          changeThreadedPortSettings
               queueSize
          adjustmentSection
               targetParallelRegion                    
                    regionName
                    newWidth
               numberOfResourcesToBeAdded
     operationConfig
          overrideResourceLoadProtection 
          forcePeStopped 
          enableAutomaticAugmentation

Note: This outline does not include the results section. This content is generated when you preview how the job will be submitted to the instance.

For information about the parameters that you can specify in a job configuration overlay file, see Job configuration overlay parameters.

Syntax of a job configuration overlay file

If you create or edit a job configuration overlay file, you must use the correct syntax. Specifically, ensure that you include the commas in the correct locations and that you include opening and closing braces, brackets, and double quotation marks.

The following code shows an example of a job configuration overlay file with the correct syntax:

{
  "jobConfigOverlays": [
    {
      "jobConfig": {
        "jobName": "myJob",
        "jobGroup": "myJobGroup",
        "dataDirectory": "myDataDir",
        "submissionParameters": [          
          { "name": "streams::pe.display", "value": "host_name:port_number" }
        ]
      },
      "deploymentConfig": {
        "placementScheme": "balancedInstance",
        "fusionScheme": "automatic",
      },
      "operatorConfigs": {
        "operators": [
          { "nameSpec": "Op_1", "declaredHost": "hostB" },
          { "nameSpec": "Op_2", "restartable": "true" },
          { "nameSpec": "Op_3", "hostIsolation": "true" },
        ],
        "poolPlacementGroups": [
          {
            "name": "myPool".
            "size": 10,
            "tags": [ "tag1", "tag2" ]
            "operatorsInPool": [
              { "nameSpec": "Op_2" },
              { "nameSpec": "Op_3" },
            ]
          }
        ]
      }
    }
  ]
}