IBM Streams 4.2

Job configuration overlays

A job configuration overlay file is a JSON file that contains name-value pairs for the submission-time configuration parameters that you want to use. A job configuration overlay file enables you to define, save, and distribute the submission-time configuration.

A job configuration overlay file is one way that you can specify submission-time configuration parameters for a job before you deploy a streams processing application to your environment. (Alternatively, you can specify submission-time configuration parameters from the IBM® Streams Console, from the Streams Studio, or by using the streamtool submitjob command.)

However, unlike manually specifying submission-time configuration parameters when you submit a job, a job configuration overlay file enables you to easily reuse or share a configuration. It also reduces the likelihood of entering information incorrectly if you want to specify submission-time configuration parameters from the interactive streamtool interface.

You can manually create a job configuration overlay file, or you can generate a job configuration overlay file when you preview how a job will be submitted to your instance. For more information about generating a job configuration overlay file from a preview, see Previewing a job before you deploy it.

For more information about how you can use submission-time configuration parameters to improve the run-time performance of the application without recompiling it, see Configuring submission-time parameters to optimize job deployment.

Structure of a job configuration overlay file

The following outline shows the order of the sections in a job configuration overlay file:

comment
jobConfigOverlays
     jobConfig
          jobName
          jobGroup
          submissionParameters
          dataDirectory
          preloadApplicationBundles
          tracing
     deploymentConfig
          fusionScheme
          fusionTargetPeCount
          placementScheme
          threadingModel
          dynamicThreadingThreadCount
          dynamicThreadingElastic
          parallelRegionConfig
               fusionType
     operatorConfigs
          operators
               nameSpec
               restartable
               relocatable
               partitionIsolation
               hostIsolation
               declaredHost
          hostColocationGroups
               groupName
               members
          hostExlocationGroups
               groupName
               members
          partitionColocationGroups
               groupName
               members
          partitionExplocationGroups
               groupName
               members
          poolPlacementGroups
               name
               OperatorsInPool
                    nameSpec
                    inPoolIndex
               membershipMode
               size
               tags
               declaredHosts
     configInstructions
          ignoreBundleRelationalPlacements
          ignoreBundleHostPlacements
          convertTagSet
               originalTagSet
               targetTagSet
          changeThreadedPortSettings
               queueSize

Note: This outline does not include the results section. This content is generated when you preview how the job will be submitted to the instance.

For more information about the parameters that you can specify in a job configuration overlay file, see Job configuration overlay parameters

The job configuration overlay file that is generated when you run a preview includes all of the settings that were configured for the job. For complex applications, the job configuration overlay file might be very long and contain a large number of entries.

If you want to modify a small number of settings, you can create a job configuration overlay file that includes only the relevant sections and parameters. Only the settings that are specified in the job configuration overlay file are changed.

Syntax of a job configuration overlay file

If you create or edit a job configuration overlay file, you must ensure that you use the correct syntax. Specifically, ensure that you include the commas in the correct locations and that you include opening and closing braces, brackets, and double quotation marks.

The following code demonstrates the correct syntax for a job configuration overlay file:

{
  "jobConfigOverlays": [
    {
      "jobConfig": {
        "jobName": "myJob",
        "jobGroup": "myJobGroup",
        "dataDirectory": "myDataDir",
        "submissionParameters": [          
          { "name": "streams::pe.display", "value": "host_name:port_number" }
        ]
      },
      "deploymentConfig": {
        "placementScheme": "balancedInstance",
        "fusionScheme": "automatic",
      },
      "operatorConfigs": {
        "operators": [
          { "nameSpec": "Op_1", "declaredHost": "hostB" },
          { "nameSpec": "Op_2", "restartable": "true" },
          { "nameSpec": "Op_3", "hostIsolation": "true" },
        ],
        "poolPlacementGroups": [
          {
            "name": "myPool".
            "size": 10,
            "tags": [ "tag1", "tag2" ]
            "operatorsInPool": [
              { "nameSpec": "Op_2" },
              { "nameSpec": "Op_3" },
            ]
          }
        ]
      }
    }
  ]
}

Using a job configuration overlay file

You can use a job configuration overlay file when you preview a job or when you submit a job:

From IBM Streams Studio, specify the job configuration overlay file from the SPL Application tab when you run or preview a job.
From Streams Console, import the job configuration overlay file when you run or preview a job. You can import it by clicking the Import icon at the top of the page and selecting Import job configuration file.
From interactive streamtool interface, run the appropriate command:
- To preview a job with a job configuration overlay file, run the streamtool previewsubmitjob command and specify the name of your job configuration overlay file:
```
streamtool previewsubmitjob -g jobConfigOverlay.json myAppBundle.sab
```
- To run a job with a job configuration overlay file, run the streamtool submitjob command and specify the name of your job configuration overlay file:
```
streamtool submitjob -g jobConfigOverlay.json myAppBundle.sab
```