Topic
5 replies Latest Post - ‏2013-11-04T12:17:28Z by hnasgaard
AdiGazit
AdiGazit
3 Posts
ACCEPTED ANSWER

Pinned topic Aggregate operator - time window fakes

‏2013-10-31T07:28:49Z |

Hello,

I have an issue with the aggregate operator time window.

It seems that it has some kind of divergence in the time when he triggers the window.

I'm using a tumbling, partitioned window with time parameter, but if I give him lets say 120 secs sometimes ge fire it much sooner.

I thought that the aggregate is start countiing for each group individualy from the first tuple that arrive to this group.

I've been told that time window starts to count from the moment the job is been uploaded, but how can it be true in parition mode?

help please.  thank you

  • hnasgaard
    hnasgaard
    200 Posts
    ACCEPTED ANSWER

    Re: Aggregate operator - time window fakes

    ‏2013-10-31T11:32:12Z  in response to AdiGazit

    Time-based windows use a timer to control eviction.  The time is absolute in that it doesn't start when a tuple arrives. It starts when the operator starts running and triggers every n seconds, regardless of when tuples arrive, or how many have arrived.

    • AdiGazit
      AdiGazit
      3 Posts
      ACCEPTED ANSWER

      Re: Aggregate operator - time window fakes

      ‏2013-10-31T13:39:11Z  in response to hnasgaard

      I understand that but how it works with partition mode in this case?

      and thanks.

      • hnasgaard
        hnasgaard
        200 Posts
        ACCEPTED ANSWER

        Re: Aggregate operator - time window fakes

        ‏2013-10-31T16:46:57Z  in response to AdiGazit

        When the timer signals, all partitions are evicted at the same time.

        • AdiGazit
          AdiGazit
          3 Posts
          ACCEPTED ANSWER

          Re: Aggregate operator - time window fakes

          ‏2013-11-03T07:26:39Z  in response to hnasgaard

          so how the sub-windows expressed if this is the case?

          what is the deference between groupby and partitionby?

          • hnasgaard
            hnasgaard
            200 Posts
            ACCEPTED ANSWER

            Re: Aggregate operator - time window fakes

            ‏2013-11-04T12:17:28Z  in response to AdiGazit

            Normally, partitionBy behaves as if there are separate windows each behaving independently while groupBy is one window divided into groups.  In this case there probably is no difference.  Each eviction would give you one tuple per partition, or one per group.