IBM Streams 4.2

Optimizing job deployment

When you create your IBM® Streams application, you make choices about PE placement, threading, and resource usage that lead to better application throughput and performance. You can also change or add settings to the application at deployment time to optimize job deployment.

About this task

For example, you can deploy a streams processing application on different instances that have different loads and different processing capabilities. The default settings from the SAB file might be optimized for a very specific deployment environment. If you deploy the application in a different environment, you might need to override the default behavior to:
  • Ensure that the application is less likely to overload a particular resource
  • Facilitate communication between related operators
  • Ensure that processing elements (PEs) are placed on appropriate resources

You can configure submission-time parameters from the IBM Streams Console, from the Streams Studio, or by using the streamtool submitjob command.

Additionally, you can create a job configuration overlay, a JSON file that contains name-value pairs for the submission-time parameters that you want to use. The job configuration overlay enables you to define, save, and distribute the submission-time parameters. For more information about creating and using job configuring overlay files, see Job configuration overlays.

There are several types of submission-time parameters that you can configure:
Job properties
Job properties enable you to specify information about the job, such as a job name or data directory to use for the application. These properties typically do not impact the performance of a job.
Deployment configuration properties
IBM Streams uses default deployment settings that should work for most applications. However, if you want to refine the behavior of your application or if the default settings don't work in your environment, you can specify the following information:
  • How operators are fused when you submit a job (fusion scheme)
  • How processing elements (PEs) are distributed to the resources in the instance when you run the application (placement scheme)
  • How to use threads in processing elements to execute operators in parallel (threading model)
Operator configuration properties
If you need to adjust how operators behave or how they are deployed in the runtime environment, you can specify the following information:
  • The properties of individual operators, such as whether the operator runs on a declared host or if the operator is restartable
  • The host pools that the operators are members of
  • The operators that should run on the same host at run time (host co-location)
  • The operators that cannot run on the same host (host ex-location)
  • The operators that should be placed in the same PE at run time (partition co-location)
  • The operators that cannot be place in the same PE at run time (partition ex-location)
Override instructions
You can optionally override application-wide settings in the application bundle file, such as ignoring relational constraints or overriding the threaded port settings.
Remember: You can specify these submission time parameters in a job configuration overlay file or from the interface from which you submit the job. Alternatively, you can also use a job configuration overlay file and then tweak settings manually.

Before you submit a job, you can preview how the submission-time configuration parameters that you specify will impact the runtime behavior of the application. For more information, see Previewing a job before you deploy it.

See the following topics for more information about the submission-time parameters:

  • Previewing a job before you deploy it
    Before you deploy an application to your run time environment, you can run a preview to see how the job will be deployed.
  • Specifying how operators are fused when you submit a job
    When you submit a job, you can optionally specify how the operators in the application are fused into processing elements (PE). The fusion scheme that you use can impact the runtime performance of the application.
  • Specifying how PEs are distributed in an application
    You can specify how the PEs in an application are distributed among the hosts in the instance. The placement scheme that you use can impact the runtime performance of the application.
  • Specifying how threads in a PE execute operators
    Within a processing element (PE), IBM Streams can use threads to execute operators in parallel to achieve better performance. The threading model of a PE determines how threads execute operators. When you submit a job, you can optionally change the threading model for all of the PEs in an application.
  • Overriding operator properties
    When you submit a job, you can optionally override the properties of the operators in your application bundle file. For example, if you plan to fuse a set operators into the same PE, you might need to change the partitionIsolation setting to false for all operators that you plan to fuse into the PE.
  • Creating host pools
    A host pool is set of resources that can be designated for a particular purpose. For example, you can specify a set of hosts that are optimized to run a specific type of analysis and assign operators that perform that analysis to the host pool.
  • Creating host co-location and ex-location groups
    You can specify how the operators in an application are grouped on the hosts in the instance. You can group operators together on a host, or you can prevent operators from being grouped with other operators.
  • Creating partition co-location or ex-location groups
    You can specify how the operators in an application are grouped together into processing elements (PEs). You can group operators together in a PE (also called a partition), or prevent operators from being grouped with other operators.
  • Overriding application-wide settings
    When you submit a job, you can optionally override several application-wide settings. For example, you might want to ignore all of the constraints that are specified in the bundle and define the appropriate constraints for your environment by specifying a job configuration overlay file or by configuring submission-time configuration parameters.
  • Job configuration overlays
    A job configuration overlay file is a JSON file that contains name-value pairs for the submission-time configuration parameters that you want to use. A job configuration overlay file enables you to define, save, and distribute the submission-time configuration.
Parent topic: Developing