Topic
  • 5 replies
  • Latest Post - ‏2013-11-04T12:17:28Z by hnasgaard
AdiGazit
AdiGazit
3 Posts

Pinned topic Aggregate operator - time window fakes

‏2013-10-31T07:28:49Z |

Hello,

I have an issue with the aggregate operator time window.

It seems that it has some kind of divergence in the time when he triggers the window.

I'm using a tumbling, partitioned window with time parameter, but if I give him lets say 120 secs sometimes ge fire it much sooner.

I thought that the aggregate is start countiing for each group individualy from the first tuple that arrive to this group.

I've been told that time window starts to count from the moment the job is been uploaded, but how can it be true in parition mode?

help please.  thank you

  • hnasgaard
    hnasgaard
    200 Posts

    Re: Aggregate operator - time window fakes

    ‏2013-10-31T11:32:12Z  

    Time-based windows use a timer to control eviction.  The time is absolute in that it doesn't start when a tuple arrives. It starts when the operator starts running and triggers every n seconds, regardless of when tuples arrive, or how many have arrived.

  • AdiGazit
    AdiGazit
    3 Posts

    Re: Aggregate operator - time window fakes

    ‏2013-10-31T13:39:11Z  
    • hnasgaard
    • ‏2013-10-31T11:32:12Z

    Time-based windows use a timer to control eviction.  The time is absolute in that it doesn't start when a tuple arrives. It starts when the operator starts running and triggers every n seconds, regardless of when tuples arrive, or how many have arrived.

    I understand that but how it works with partition mode in this case?

    and thanks.

  • hnasgaard
    hnasgaard
    200 Posts

    Re: Aggregate operator - time window fakes

    ‏2013-10-31T16:46:57Z  
    • AdiGazit
    • ‏2013-10-31T13:39:11Z

    I understand that but how it works with partition mode in this case?

    and thanks.

    When the timer signals, all partitions are evicted at the same time.

  • AdiGazit
    AdiGazit
    3 Posts

    Re: Aggregate operator - time window fakes

    ‏2013-11-03T07:26:39Z  
    • hnasgaard
    • ‏2013-10-31T16:46:57Z

    When the timer signals, all partitions are evicted at the same time.

    so how the sub-windows expressed if this is the case?

    what is the deference between groupby and partitionby?

  • hnasgaard
    hnasgaard
    200 Posts

    Re: Aggregate operator - time window fakes

    ‏2013-11-04T12:17:28Z  
    • AdiGazit
    • ‏2013-11-03T07:26:39Z

    so how the sub-windows expressed if this is the case?

    what is the deference between groupby and partitionby?

    Normally, partitionBy behaves as if there are separate windows each behaving independently while groupBy is one window divided into groups.  In this case there probably is no difference.  Each eviction would give you one tuple per partition, or one per group.