Topic
  • 7 replies
  • Latest Post - ‏2013-02-01T21:27:11Z by hnasgaard
BruceGlassford
BruceGlassford
71 Posts

Pinned topic Custom literal as runtime parameter?

‏2013-02-01T15:04:17Z |
Got a situation where it would be extremely convenient to pass in a custom literal as a runtime parameter so as to avoid having to recompile the code to reuse a job for various purposes.

As a simple example, think of a TopN job where I get the top N tuples from the data stream passed through, and some times I want the highest 25 tuples of data through and sometimes the lowest 25 tuples. So I'd want to pass "ascending" or "descending" in to be used in the Sort operator.

Can this be done?
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Custom literal as runtime parameter?

    ‏2013-02-01T15:36:33Z  
    Bruce,

    The short answer is no, you can't do that. There are two reasons. First, assuming you could specify the value at submission time, the Sort's param value for order is used to generate order specific code at compile-time. The second is, at this time, you can't use a value passed in, at least where you would need it, to specify order. I think you would have to code something like:
    
    param order : (SortedOrder)getSubmissionTimeValue(...);
    

    That will not compile today, although I think you could argue that it should.
  • BruceGlassford
    BruceGlassford
    71 Posts

    Re: Custom literal as runtime parameter?

    ‏2013-02-01T17:18:37Z  
    • hnasgaard
    • ‏2013-02-01T15:36:33Z
    Bruce,

    The short answer is no, you can't do that. There are two reasons. First, assuming you could specify the value at submission time, the Sort's param value for order is used to generate order specific code at compile-time. The second is, at this time, you can't use a value passed in, at least where you would need it, to specify order. I think you would have to code something like:
    <pre class="jive-pre"> param order : (SortedOrder)getSubmissionTimeValue(...); </pre>
    That will not compile today, although I think you could argue that it should.
    Thanks for the confirmation. So if I'm building my own operators, I should avoid custom literals if I want to make it easier for the users to switch at run time and just pass in strings. And yes, I'd argue it should compile. Probably work if the literal was exposed as an enum, but that's not really feasible in general.
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Custom literal as runtime parameter?

    ‏2013-02-01T18:33:06Z  
    Thanks for the confirmation. So if I'm building my own operators, I should avoid custom literals if I want to make it easier for the users to switch at run time and just pass in strings. And yes, I'd argue it should compile. Probably work if the literal was exposed as an enum, but that's not really feasible in general.
    The literal actually is exposed as an enum, but the enum type (for the cast) isn't visible in that clause. Using a literal is the desirable way to do it. I'll track the desire to make the enum type visible in that scope.
  • BruceGlassford
    BruceGlassford
    71 Posts

    Re: Custom literal as runtime parameter?

    ‏2013-02-01T19:39:19Z  
    • hnasgaard
    • ‏2013-02-01T18:33:06Z
    The literal actually is exposed as an enum, but the enum type (for the cast) isn't visible in that clause. Using a literal is the desirable way to do it. I'll track the desire to make the enum type visible in that scope.
    Thanks. That would help. Of course that also means that for it to work the operator's author must rely on runtime logic instead of compile-time, so would have to be able to distinguish between runtime-changeable parameters and compile time only. We have that problem already, though - which raises the question of how to distinguish now? If I have a parameter I rely on in the perl code in a generic C++ operator, meaning it must be compile-time only, how do I indicate that in the operator model?
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Custom literal as runtime parameter?

    ‏2013-02-01T20:21:34Z  
    Thanks. That would help. Of course that also means that for it to work the operator's author must rely on runtime logic instead of compile-time, so would have to be able to distinguish between runtime-changeable parameters and compile time only. We have that problem already, though - which raises the question of how to distinguish now? If I have a parameter I rely on in the perl code in a generic C++ operator, meaning it must be compile-time only, how do I indicate that in the operator model?
    This is doable today, generally speaking. Consider the following:
    
    stream<int32 i> Beat = Beacon() 
    { param iterations : (int32)getSubmissionTimeValue(
    "count"); output Beat : i = (int32) IterationCount(); 
    }
    

    Here we are passing a submission time value to a param. When the Beacon is generated, the instance model contains transformed code that will use a literal that is supplied at submission time:
    
    <cppExpression>::SPL::spl_cast<SPL::int32, SPL::rstring >::cast(lit$0)</cppExpression> <splExpression>(int32)(getSubmissionTimeValue(
    "count"))</splExpression>
    

    Note the SPL expression and the equivalent cpp expression. If the code generator is correctly written then submission time values can be used.
    The operator model does not seem to be able to definitively express this characteristic. Perhaps it should. It would be good if this was specified in words in the toolkit reference.
  • BruceGlassford
    BruceGlassford
    71 Posts

    Re: Custom literal as runtime parameter?

    ‏2013-02-01T21:18:34Z  
    • hnasgaard
    • ‏2013-02-01T20:21:34Z
    This is doable today, generally speaking. Consider the following:
    <pre class="jive-pre"> stream<int32 i> Beat = Beacon() { param iterations : (int32)getSubmissionTimeValue( "count"); output Beat : i = (int32) IterationCount(); } </pre>
    Here we are passing a submission time value to a param. When the Beacon is generated, the instance model contains transformed code that will use a literal that is supplied at submission time:
    <pre class="jive-pre"> <cppExpression>::SPL::spl_cast<SPL::int32, SPL::rstring >::cast(lit$0)</cppExpression> <splExpression>(int32)(getSubmissionTimeValue( "count"))</splExpression> </pre>
    Note the SPL expression and the equivalent cpp expression. If the code generator is correctly written then submission time values can be used.
    The operator model does not seem to be able to definitively express this characteristic. Perhaps it should. It would be good if this was specified in words in the toolkit reference.
    Thanks for the details. I knew it was possible. What I'm wondering about is if I'm using an input parameter (like Sort apparently does) at compile-time to determine generated code, how to mark that in the operator model to prevent submission time parameters being used.
  • hnasgaard
    hnasgaard
    200 Posts

    Re: Custom literal as runtime parameter?

    ‏2013-02-01T21:27:11Z  
    Thanks for the details. I knew it was possible. What I'm wondering about is if I'm using an input parameter (like Sort apparently does) at compile-time to determine generated code, how to mark that in the operator model to prevent submission time parameters being used.
    In this particular case I would expect the operator's code generator to complain at compile time if it did not get a constant. I think that the compile doesn't get far enough, due to the fact that the SortedOrder CustomLiteral type is not recognized, to get to the point where the code generator could complain. It fails before the code generator is invoked.

    I suspect that the Sort could be changed so that order could be changed at runtime without any significant performance degradation. Then, if that cast would work, the order could be specified at submit time. Unfortunately, I don't expect to be able to change that in the near future.