I'm new to Streams and I am trying to find out how the number of processing element containers (PEC) is defined in Streams.
Or is it totally dynamic, entirely controlled by Streams, spawning additional PECs as needed?
Is there a pointer to where in the documentation one can find a description on the configuration and life cycle of PECs?
Sorry if this information is in a rather obvious place and I couldn't find it.
Thanks for your help.
This topic has been locked.
Pinned topic Defining the number of PECs
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
SystemAdmin 110000D4XK1245 Posts
Re: Defining the number of PECs2012-10-05T05:23:40ZThis is the accepted answer. This is the accepted answer.Hi Julias - you have quite a bit if control over operator placement and fusion within a PE. See the operator config guide as a start
mendell 120000KPGC219 Posts
Re: Defining the number of PECs2012-10-05T12:22:49ZThis is the accepted answer. This is the accepted answer.There is one PEC for each PE (processing element) in the SPL program. By default, the SPL compiler puts each operator into its own PE, but you have control over that using config placement directives in your SPL program. You can also do a 'profiling run', and then the SPL compiler will use the results of that run to try to pick an optimal PE placement, minimizing data transfer time, while ensuring that CPU use isn't too high. You can then repeat as necessary.
PECs (and the associated PEs) are created a job submission time, and remain active until the job is cancelled.
jlerm 0600010T6X3 Posts
Re: Defining the number of PECs2012-10-05T21:10:20ZThis is the accepted answer. This is the accepted answer.Thanks a lot for your replies!