Topic
  • 8 replies
  • Latest Post - ‏2012-05-21T14:30:52Z by mendell
errah
errah
9 Posts

Pinned topic how to split a stream into component tuples?

‏2012-05-17T20:14:07Z |
I know this must be an absurdly easy question, but I can't figure it out.

I have a stream that contains multuple tuples. I want each tuple to be sent to be processed by a different operator, an absurdly basic problem. Yet I can't seem to make it work. So for example I want to do something like this


type inputTuple= tuple< tupleA, tupleB, tupleC>; Stream<inputTuple> inputStream= someInputOperator();   (stream<tupleA> tupleAStream; stream<tupleB> tupleBStream; stream<tupleC> tupleCStream) = functor(inputStream) 
{ output tupleAStream: tupleA; tupleBStream: tupleB; tupleCStream: tupleC;   
}   () as sinkA = TupleProcessA(tupleASteam); () as sinkB = TupleProcessB(tupleBSteam); () as sinkC = TupleProcessC(tupleCSteam);


Unfortunately the syntax as I wrote it is wrong, the output part of the functor is wrong. I need to write it as something = tupleA, but I don't know what to put. Since TupleAStream is suppose to get an exact copy of the contents of tupleA I don't have a specific variable within tupleAStream to set tupleA to.

This is so trivial I know there must be an easy way to do it. Anyone want to help me out with the syntax I'm missing?

PS I wrote this quickly, so please forgive me for any syntax errors in my sample code.
  • BruceGlassford
    BruceGlassford
    71 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-17T21:17:11Z  
    I think you need a Custom - to split an input tuple into three, try:

    (stream<tupleA> tupleAStream; stream<tupleB> tupleBStream; stream<tupleC> tupleCStream) = Custom(inputStream)
    {

    logic
    onTuple inputStream:
    {
    submit({tupleA = inputStream.tupleA}, tupleAStream);
    submit({tupleB = inputStream.tupleB}, tupleBStream);
    submit({tupleC = inputStream.tupleC}, tupleCStream);
    }
    }
  • errah
    errah
    9 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-17T22:15:58Z  
    I think you need a Custom - to split an input tuple into three, try:

    (stream<tupleA> tupleAStream; stream<tupleB> tupleBStream; stream<tupleC> tupleCStream) = Custom(inputStream)
    {

    logic
    onTuple inputStream:
    {
    submit({tupleA = inputStream.tupleA}, tupleAStream);
    submit({tupleB = inputStream.tupleB}, tupleBStream);
    submit({tupleC = inputStream.tupleC}, tupleCStream);
    }
    }
    I attempted to use the syntaxs you suggested, but it doesn't seem to work. In fact when trying to parse what you wrote I realized I didn't write my original sample correctly, which makes it unclear rather your refering to tuple type or a specific named tuple in your response. I should have named my inputTuple something like this:

    type inputTuple= tuple< tupleA A, tupleB B, tupleC C>;
    As i wrote things I sort of treatd the tuple type and tuple names as the same thing, my fault for writing something quickly.

    Anyways I believe you may be right that I need a Custom instead of a Functor since I want multuple output streams, but as far as I can tell the question of how to syntactically say "I want to treat the full contents of this tuple as the output" instead of individually setting each variable is still unanswered. If I'm wrong perhaps you could clearify your example now that i've actually given my inputs proper names to use?
  • errah
    errah
    9 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-17T22:33:21Z  
    • errah
    • ‏2012-05-17T22:15:58Z
    I attempted to use the syntaxs you suggested, but it doesn't seem to work. In fact when trying to parse what you wrote I realized I didn't write my original sample correctly, which makes it unclear rather your refering to tuple type or a specific named tuple in your response. I should have named my inputTuple something like this:

    type inputTuple= tuple< tupleA A, tupleB B, tupleC C>;
    As i wrote things I sort of treatd the tuple type and tuple names as the same thing, my fault for writing something quickly.

    Anyways I believe you may be right that I need a Custom instead of a Functor since I want multuple output streams, but as far as I can tell the question of how to syntactically say "I want to treat the full contents of this tuple as the output" instead of individually setting each variable is still unanswered. If I'm wrong perhaps you could clearify your example now that i've actually given my inputs proper names to use?
    Correction, I understand what you were saying. now that I've looked at the submit function closer I understand how it works. The actual calls I had to make were even simpler then I thought. Assuming tupleA is named A etc as I mentioned in my last post It worked using the syntax

    
    logic onTuple inputStrea: submit(A, tupleStreamA); submit(B, tupleStreamB); submit(C, tupleStreamC);
    

    Thank you for pointing me in the correct direction. I would mark the question answered, but I forgot to mark it as a question to begin with so...
  • mendell
    mendell
    219 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-18T00:17:43Z  
    • errah
    • ‏2012-05-17T22:33:21Z
    Correction, I understand what you were saying. now that I've looked at the submit function closer I understand how it works. The actual calls I had to make were even simpler then I thought. Assuming tupleA is named A etc as I mentioned in my last post It worked using the syntax

    <pre class="jive-pre"> logic onTuple inputStrea: submit(A, tupleStreamA); submit(B, tupleStreamB); submit(C, tupleStreamC); </pre>
    Thank you for pointing me in the correct direction. I would mark the question answered, but I forgot to mark it as a question to begin with so...
    You can do this with a Functor:
    
    type inputTuple = tuple<tupleA A, tupleB B, tupleC C>; ... (stream<tupleA> tupleAStream; stream<tupleB> tupleBStream; stream<tupleC> tupleCStream) = Functor(inputStream) 
    { output tupleAStream: A; tupleBStream: B; tupleCStream: C; 
    }
    


    Using a Functor will be (very) slightly more efficient than doing the same using a Custom.

    Mark
  • Jim Sharpe
    Jim Sharpe
    98 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-18T23:33:26Z  
    • mendell
    • ‏2012-05-18T00:17:43Z
    You can do this with a Functor:
    <pre class="jive-pre"> type inputTuple = tuple<tupleA A, tupleB B, tupleC C>; ... (stream<tupleA> tupleAStream; stream<tupleB> tupleBStream; stream<tupleC> tupleCStream) = Functor(inputStream) { output tupleAStream: A; tupleBStream: B; tupleCStream: C; } </pre>

    Using a Functor will be (very) slightly more efficient than doing the same using a Custom.

    Mark
    Hi Mark,

    You've peaked my interest. What is it that makes the Functor slightly more efficient? I'm guessing it has something to do with a difference in the way the values are copied but would like to learn more. Also would the performance behavior be different if a submit was used instead of doing it in the output clause?
  • mendell
    mendell
    219 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-21T12:55:34Z  
    Hi Mark,

    You've peaked my interest. What is it that makes the Functor slightly more efficient? I'm guessing it has something to do with a difference in the way the values are copied but would like to learn more. Also would the performance behavior be different if a submit was used instead of doing it in the output clause?
    The I probaby doesn't make too much difference, but the Custom operator has mutable inputs and outputs. This is relevant because it has to ensure that to the user, submitting a tuple doesn't change the contents of the tuple. To do this, if part of an input tuple is submitted downstream, and also referenced after it has been submitted, it will make a copy of that value, in order to ensure that it is unmodifed afterwards.

    A Functor has non-mutable outputs, and so it is safe to use the input tuple attributes directly without copying, as it is known that the contents won't change.

    This is only relevant for fused operators.

    Mark
  • Jim Sharpe
    Jim Sharpe
    98 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-21T13:05:11Z  
    • mendell
    • ‏2012-05-21T12:55:34Z
    The I probaby doesn't make too much difference, but the Custom operator has mutable inputs and outputs. This is relevant because it has to ensure that to the user, submitting a tuple doesn't change the contents of the tuple. To do this, if part of an input tuple is submitted downstream, and also referenced after it has been submitted, it will make a copy of that value, in order to ensure that it is unmodifed afterwards.

    A Functor has non-mutable outputs, and so it is safe to use the input tuple attributes directly without copying, as it is known that the contents won't change.

    This is only relevant for fused operators.

    Mark
    Hi Mark,

    Thanks for the clarification. I figured it was something along those lines but it's nice to have the confirmation. For some future version of the Redbook I think tidbits like that would make excellent content for a chapter on performance tuning. :)
  • mendell
    mendell
    219 Posts

    Re: how to split a stream into component tuples?

    ‏2012-05-21T14:30:52Z  
    Hi Mark,

    Thanks for the clarification. I figured it was something along those lines but it's nice to have the confirmation. For some future version of the Redbook I think tidbits like that would make excellent content for a chapter on performance tuning. :)
    Just as a note, this special handling (copying the input when referenced later) only applies to a submit in a Custom (which is the only operator that allows a submit )