IBM Streams 4.2

streamtool submitjob

The streamtool submitjob command submits one or more jobs.

Usage


submitjob

   .------------------------------------------------.   
   V                                                |   
>>---+--------------------------------------------+-+----------->
     '-+- -C-------+--config-setting=config-value-'     
       '- --config-'                                    

>--+-----------------------+--+------------------------+-------->
   '-+- -d----------+--did-'  +-+- -f-----+--file-name-+   
     '- --domain-id-'         | '- --file-'            |   
                              '-sab-pathname-----------'   

>--+----------------------------+--+-------------+-------------->
   '-+- -g---------+--file-name-'  '-+- -h-----+-'   
     '- --opConfig-'                 '- --help-'     

>--+-------------------------+---------------------------------->
   '-+- -i------------+--iid-'   
     '- --instance-id-'          

>--+--------------------------------+--------------------------->
   '-+- -J---------+--jobgroup-name-'   
     '- --jobgroup-'                    

>--+----------------------+--+-----------------------+---------->
   '- --jobname--job-name-'  '- --outfile--file-name-'   

>--+----------------------------+------------------------------->
   '- --override--override-list-'   

   .------------------------------.                        
   V                              |                        
>----+--------------------------+-+--+-----------------+-------->
     '-+- -P--+--parameter-name-'    '- --trace--level-'   
       '- --P-'                                            

>--+------------------+--+-----------------------+-------------->
   '-+- -U-----+--uid-'  '-+- -v--------+--level-'   
     '- --User-'           '- --verbose-'            

>--| Non-interactive tool options |----------------------------><

Non-interactive tool options

    (1)                                    
|--------+-----------------------------+------------------------|
         +- --embeddedzk---------------+   
         |               .-,---------. |   
         |               V           | |   
         '- --zkconnect----host:port-+-'

Notes:

The non-interactive tool options are not supported in the interactive streamtool interface.

Authority

You must have add authority for the appropriate jobgroup_name instance object or else submit permission for the job group. By default, the DomainAdministrator, InstanceAdministrator, and InstanceUser roles have this authority.

You must have add authority for the jobs-override instance object to use the --override HostLoadProtection option. By default, the DomainAdministrator and InstanceAdministrator roles have this authority.

For more information about access control lists, see the streamtool getacl and streamtool lsjobpermission commands.

Description

The job runs an application that is defined by an application bundle. Application bundles are created by the Stream Processing Language (SPL) compiler.

A submitted job consists of one or more processing elements (PEs). The PEs are placed on one or more of the application resources for the instance. The submission fails if the PE placement constraints cannot be met.

When you submit the job, it is assigned to the "default" job group unless you specify the -J or --jobgroup options. The job group that you specify in those options must exist before you run this command. For more information about creating job groups, see streamtool man mkjobgroup.

The command generates a job identifier (ID) for each job that you submit. You can also optionally specify a job name. The job name must satisfy the following requirements:

The name must be unique within the instance.
The name must contain alphanumeric characters. You cannot use the following alphanumeric characters: ^!#$%&'*+,/;<>=?@[]`{|}~(). You also cannot use the following Unicode and hexadecimal characters: u0000; u0001-u001F; u007F-u009F; ud800-uF8FF; uFFF0-uFFFF; x{10000}-x{10FFFF}.
The maximum length of the name is 1 KB.

If you do not specify a job name, IBM® Streams creates a unique job name with the following format: applicationName_jobID. You cannot change the job name after you submit the job.

An application can define optional or required submission parameters that can be used to control its behavior. If you do not provide required submission parameters when you submit the job, the submission fails. If you specify a parameter name that is not known to the application, the command generates a warning and the parameter is ignored.

You can optionally specify an application configuration setting by using the -C, --config option. For example, you can use this option to set the location of the data directory:

streamtool submitjob -C data-directory=/myDataDirectory myappl.sab

To submit more than one job in a single command, use the -f, --file option. Each job submission must be provided on a separate line in the file. Blank lines and lines that have a # character as the first non-white space character are ignored. In the file, job submission specifications consist of job parameter specifications and configuration settings followed by the path name for the application bundle. For example:

[-P,--P name=value]...[-C,--config name=value]... application-bundle-name

Use only double or single quotation marks within the file to deal with parameter values that contain white space characters. Put quotation marks around the combined name and value specification. For example: --P "myParm=myValue". The same advice applies to this situation when you submit a job from the command line.

Shell expansion processing does not occur for the contents of the file. For example, the command does not perform path name wildcard expansion or shell variable expansion.

If one of the jobs that are listed in a file fails, subsequent jobs are nonetheless parsed and can succeed.

Jobs remain in the system until they are canceled or the instance is stopped.

Options and arguments

-C, --config config-setting=config-value

Specifies the name of one of the following application configuration settings and its value:
data-directory

Specifies the location of the data directory. For example: /myDataDirectory.

fusionScheme

Specifies the scheme that is used to determine how to fuse the operators in a streams processing application into partitions. By default, the command uses the instance.fusionScheme instance property value.

Valid values are automatic (default), legacy, and manual. When you use the automatic value, the system determines the most efficient the number of PEs to assign to a job.

Using the legacy value prompts operator fusion to behave as it did before IBM Streams Version 4.2. Typically, each operator is fused into a separate PE.

With the manual fusion scheme, you can specify the target number of PEs to use.

fusionTargetPeCount

Specifies the target number of processing elements to use when fusing operators. This parameter can only be specified when fusionScheme is set to manual.

placementScheme

Specifies a scheme to determine which PEs get placed on which hosts. Valid values are balancedInstance (default), balancedJob, and legacy.

When you use the balancedInstance value, the system distributes PEs across candidate hosts proportionally to the number of cores on the hosts. This method takes into account PE placements from previous jobs.

Using the balancedJob value means that PEs are distributed across candidate hosts proportionally to the number of cores on the hosts, but PE placements from previous jobs are ignored.

When you use the legacy value, the estimated load of PEs to be placed is matched with existing ldavg numbers of hosts that are adjusted for the number of cores in hosts.

preloadApplicationBundles

Specifies whether to preload the application bundle files onto all resources in the instance, even if not currently needed there. Valid values are true and false. By default, the command uses the instance.preloadApplicationBundles instance property value. Preloading the application bundle files can improve the performance in situations where a PE is relocated to a new resource that was not already hosting a PE from that application.

dynamicThreadingElastic

Specifies if elasticity is on or off for dynamicThreading. Valid values are true for on, and false for off. This setting can only be used if instance.threadingModel is set to dynamic, or the application configuration threadingModel is set to dynamic.

dynamicThreadingThreadCount

Specifies the number of threads for scheduled ports to use. Inherits its value from instance.dynamicThreadingThreadCount, if set. Valid values are integers greater than or equal to 1. This setting can only be used if instance.threadingModel is set to dynamic, or the application configuration threadingModel is set to dynamic.

threadingModel

Specifies the threading model to use as the default for each PE in the job. Inherits its value from instance.threadingModel, if set. This default can be overridden by @threading annotations in the SPL code. Valid values are manual, automatic, scheduledPorts, allThreadedPorts and default.

tracing

Specifies the PE trace setting. The following valid levels are listed in order of increasing verbosity, which is to say that the first level in the list generates the least amount of information:
off

error

warn

info

debug

trace

By default, the trace setting is error unless you specify otherwise in the streams processing application.

For example: --config tracing=off,preloadApplicationBundles=true.

-d, --domain-id did

Specifies the domain identifier.
If you do not specify this option, IBM Streams uses the domain name that is set in the STREAMS_DOMAIN_ID environment variable. By default, that domain name is StreamsDomain. If you are using the interactive streamtool interface, it uses the name of the active domain for the current streamtool session or else it prompts you for the domain name.

The active domain for the current streamtool session is set every time that you successfully run a streamtool command with a -d or --domain-id option. Alternatively, you can run the streamtool domain command in the interactive interface.

--embeddedzk

Specifies to use the embedded copy of ZooKeeper. This option is not supported within the interactive streamtool interface.

If you are not using the interactive streamtool interface and you do not specify either this option or the --zkconnect option, IBM Streams uses the ZooKeeper connection that is associated with the active domain or the domain that is specified in the --domain-id option. IBM Streams determines which connection maps to the domain by using cached information about the domains. In this scenario, if the domain identifier is not unique in the IBM Streams configuration cache, the command fails.

-f, --file file-name

Specifies the name of a file that contains a list of job submission specifications.

-g, --opConfig file-name

Specifies the name of a file that contains job configuration information. For more information, see Job configuration overlays.

-h, --help

Specifies to show the command syntax.

-i, --instance-id iid

Specifies the instance identifier.
If you do not specify this option, IBM Streams uses the instance identifier that is set in the STREAMS_INSTANCE_ID environment variable. By default, that instance identifier is StreamsInstance. If you are using the interactive streamtool interface, it tries to use an instance ID that you specified in a previous command. If no such value is found, the command uses the STREAMS_INSTANCE_ID environment variable. Alternatively, you can run the streamtool instance command in the interactive interface.

-J,--jobgroup jobgroup-name

Specifies the job group. If you do not specify this option, the command uses the following job group: default.

--jobname job-name

Specifies the job name for the submitted job.

--outfile file-name

Specifies the name of the output file in which the command writes the list of submitted job IDs.

--override override-list

Specifies to override the instance.resourceLoadProtectionEnabled instance property for the job and treat it as false. The resource load protection property prevents the IBM Streams job scheduler from allocating processing elements (PEs) to a resource that already has a load average above the specified resource load threshold value. When you specify this option, resource load protection is disabled. For example:
streamtool submitjob --override HostLoadProtection myJob.sab

-P, --P parameter-name

Specifies a submission-time parameter and value for the job. You can specify this option multiple times in the command.

sab-pathname

Specifies the path name for the application bundle file. If you do not specify an absolute path, the command seeks the file in the directory where you ran the command. Alternatively, you can specify the path name for the application description language (ADL) file if the application bundle file exists in the same directory.

--trace level

Specifies the trace setting. The following valid levels are listed in order of increasing verbosity, which is to say that the first level in the list generates the least amount of information:
off

error

warn

info

debug

trace

The default value is off.

-U, --User userid

Specifies an IBM Streams user ID that has authority to run the command.

-v,--verbose level

Specifies to provide more detailed command output. The verbosity level can be 0-3, where 0 disables detailed reporting and each increment provides more detailed output.

--zkconnect host:port

The name of one or more host and port pairs that specify the configured ZooKeeper servers. This option is not supported within the interactive streamtool interface.

If you are not using the interactive streamtool interface and you do not specify this option, IBM Streams tries to use:
The --embeddedzk option

The value from the STREAMS_ZKCONNECT environment variable

A ZooKeeper connection string that is derived from cached information about the current domain.

Examples

The following command submits a job with submission time values for a distributed application:

[streamtool <bsmith@StreamsDomain.StreamsInstance>] submitjob -P students='[{name="Mary", grade=100},{name="John", grade=90}]' output/Main.sab