Topic
  • 7 replies
  • Latest Post - ‏2010-06-03T22:05:43Z by SystemAdmin
ScottAJ
ScottAJ
26 Posts

Pinned topic Java Source Udop Help (I need somebody)

‏2010-06-02T18:23:19Z |
I am trying to make a Java Source Udop, but am running into issues. I finally got it to compile without any errors (which we all know doesn't really mean anything, but it's a nice plateau to reach), but nothing is getting sent to the Sink operator. I don't really know what I'm doing as I've patched my java file together from what was generated, what I found in 041_javaudop_at_work and what I've figured out from the included documentation (InfoSphereStreams/doc/spade/javadoc/api).

Basically, what I want to do is have a Java controlled operator that has one output stream, but doesn't have any streams coming in. Right now there's just some trivial code in my Udop as I just want to get it working before I actually start using it for what we need.

Here is my java file:



package source;   

import com.ibm.streams.spade.OperatorContext; 

import com.ibm.streams.spade.OutputTuple; 

import com.ibm.streams.spade.StreamingOutput; 

import com.ibm.streams.spade.Tuple; 

import com.ibm.streams.spade.samples.patterns.TupleProducer;   

public 

abstract 

class input 

extends TupleProducer 
{   @Override 

public 

void startProcessing() 
{ 

long i; OutputTuple tuple; StreamingOutput<OutputTuple> stream; tuple.setLong(
"Prime",0); 

for (i = 0; i < 1000000; i++) 
{ 

if (i % 7 == 0) 
{ tuple = stream.newTuple(); tuple.setLong(
"Prime",i); 

try 
{ stream.submit(tuple); 
} 

catch ( java.lang.Exception s) 
{
} 
//SPCDBG(L_DEBUG, "Sent " << i << " through", "myjop"); 
} 
} 

return; 
} 
}


and here is my dps file:

#Some Java Testing   [Application] javasourcetest [Program]     stream Primes(Prime: Long) := JavaUdop() [className: 
"source.input"; classLibrary: 
"CLASSLIB"]
{
}     Nil := Sink(Primes)[
"file:///out.dat",csvformat,nodelays]
{
}


So clearly something I'm doing isn't working, and I would appreciate ANY help I can get to get this working.

Thanks,
~Scott
  • ScottAJ
    ScottAJ
    26 Posts

    Re: Java Source Udop Help (I need somebody)

    ‏2010-06-02T18:28:19Z  
    Additional Question: What does "CLASSLIB" mean in terms of the classLibrary? Is it just whatever Streams defaults to based on the className?

    For instance, if className: "source.input" then would CLASSLIB be "src/source"?

    Thanks!
    ~Scott
  • kjerick
    kjerick
    227 Posts

    Re: Java Source Udop Help (I need somebody)

    ‏2010-06-03T00:15:33Z  
    • ScottAJ
    • ‏2010-06-02T18:28:19Z
    Additional Question: What does "CLASSLIB" mean in terms of the classLibrary? Is it just whatever Streams defaults to based on the className?

    For instance, if className: "source.input" then would CLASSLIB be "src/source"?

    Thanks!
    ~Scott
    Hi Scott,

    Regarding your second question first, I'm guessing you copied the "CLASSLIB" code from the java udop sample code. If you look in the .dps file there is a "#define CLASSLIB %1" that sets CLASSLIB to what is passed in to the .dps file when spadec is called on it. In the main Makefile for that sample you will see "CMD_LINE_ARGS = $(PWD)/bin", which is where the java class code is expected to be for that sample. In the main Makefile for that sample you will also see "make -C src -f Makefile.usr", and in Makefile.usr you will see where the ../bin directory is created and in which the results of the javac are placed. So, if you are using "CLASSLIB" in your code, it will have to be properly defined in a similar manner, or you can just hardcode the path.

    Regarding your first question, I will see if I can get someone in development that has more experience with java udops to look at your post. In the meantime, if you have any more information on what things you've done or seen, perhaps it will help. For example, do you get any errors or exceptions or anything else in terms of output in your terminal when you try running your application which you can add to your forum post?

    Thanks and best regards,
    Kevin
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Java Source Udop Help (I need somebody)

    ‏2010-06-03T01:11:38Z  
    • ScottAJ
    • ‏2010-06-02T18:28:19Z
    Additional Question: What does "CLASSLIB" mean in terms of the classLibrary? Is it just whatever Streams defaults to based on the className?

    For instance, if className: "source.input" then would CLASSLIB be "src/source"?

    Thanks!
    ~Scott
    The parameter classLibrary corresponds to the Java class path that your implementation of the Operator interface requires. Thus if you compiled your class into /home/scott/bin then SPADE source would look like:

    
    stream Primes(Prime: Long) := JavaUdop() [className: 
    "source.input"; classLibrary: 
    "/home/scott/bin"]
    {
    }
    


    Then say your Operator class depended on some other jar file, e.g. derby.jar for the Apache Derby database then the code would look something like:

    
    stream Primes(Prime: Long) := JavaUdop() [className: 
    "source.input"; classLibrary: 
    "/home/scott/bin", 
    "/home/scott/apache/derby/lib/derby.jar"]
    {
    }
    


    Note that classLibrary is a parameter that can take multiple values, each value corresponds to an element (in order) in the class path for your Operator implementation.
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Java Source Udop Help (I need somebody)

    ‏2010-06-03T01:27:57Z  
    There's a minor problem with the code, the general approach to sending tuples is correct but the startProcessing() method is only intended to kick-off the processing, not perform all the processing. Its javadoc tries to say this, but could have been clearer. Basically what is happening with TupleProcessor is that when its allPortsReady() is called it calls startProcessing(), but the InfoSphere Streams runtime is expecting allPortsReady() to return immediately.

    A simple re-write would be to kick-off the generation of tuples in a different method using the supplied ExecutorService (a standard JDK class), e.g.:

    
    @Override 
    
    public 
    
    void startProcessing() 
    {   
    // Asynchronously kick of the tuple generator getOperatorContext().getScheduledExecutorService().submit( 
    
    new Runnable() 
    { 
    
    public 
    
    void run() 
    { generateTuples(); 
    } 
    }); 
    }   
    
    private 
    
    void generateTuples() 
    { 
    
    long i; OutputTuple tuple; StreamingOutput<OutputTuple> stream; tuple.setLong(
    "Prime",0); 
    
    for (i = 0; i < 1000000; i++) 
    { 
    
    if (i % 7 == 0) 
    { tuple = stream.newTuple(); tuple.setLong(
    "Prime",i); 
    
    try 
    { stream.submit(tuple); 
    } 
    
    catch ( java.lang.Exception s) 
    {
    } 
    //SPCDBG(L_DEBUG, "Sent " << i << " through", "myjop"); 
    } 
    } 
    }
    


    If you look at the source for TupleProducer a more complete example of using the ExecutorService exists for when a starting delay is specified to TupleProducer. In that case the startProcessing() method is started asynchronously with a delay.

    Note the source code for the sample patterns is in the streams.spade.jar, if you are using Eclipse clicking on one of these samples should take you to the source code.
  • ScottAJ
    ScottAJ
    26 Posts

    Re: Java Source Udop Help (I need somebody)

    ‏2010-06-03T15:13:20Z  
    The parameter classLibrary corresponds to the Java class path that your implementation of the Operator interface requires. Thus if you compiled your class into /home/scott/bin then SPADE source would look like:

    <pre class="jive-pre"> stream Primes(Prime: Long) := JavaUdop() [className: "source.input"; classLibrary: "/home/scott/bin"] { } </pre>

    Then say your Operator class depended on some other jar file, e.g. derby.jar for the Apache Derby database then the code would look something like:

    <pre class="jive-pre"> stream Primes(Prime: Long) := JavaUdop() [className: "source.input"; classLibrary: "/home/scott/bin", "/home/scott/apache/derby/lib/derby.jar"] { } </pre>

    Note that classLibrary is a parameter that can take multiple values, each value corresponds to an element (in order) in the class path for your Operator implementation.
    So I would need to run "javac input.java" and then place the .class file wherever I have classLibrary equal to?

    I tried this, after changing my code to what you provided, and got a bunch of "package com.ibm.streams.spade does not exist"

    Here's the exact output:
    
    [sjones@drydock source]$ javac input.java input.java:3: 
    
    package com.ibm.streams.spade does not exist 
    
    import com.ibm.streams.spade.OperatorContext; ^ input.java:4: 
    
    package com.ibm.streams.spade does not exist 
    
    import com.ibm.streams.spade.OutputTuple; ^ input.java:5: 
    
    package com.ibm.streams.spade does not exist 
    
    import com.ibm.streams.spade.StreamingOutput; ^ input.java:6: 
    
    package com.ibm.streams.spade does not exist 
    
    import com.ibm.streams.spade.Tuple; ^ input.java:7: 
    
    package com.ibm.streams.spade.samples.patterns does not exist 
    
    import com.ibm.streams.spade.samples.patterns.TupleProducer; ^ input.java:9: cannot find symbol symbol: 
    
    class TupleProducer 
    
    public 
    
    abstract 
    
    class input 
    
    extends TupleProducer 
    { ^ input.java:15: cannot find symbol symbol  : method getOperatorContext() location: 
    
    class source.input getOperatorContext().getScheduledExecutorService().submit( ^ input.java:11: method does not override or implement a method from a supertype @Override ^ input.java:25: cannot find symbol symbol  : 
    
    class OutputTuple location: 
    
    class source.input OutputTuple tuple; ^ input.java:26: cannot find symbol symbol  : 
    
    class StreamingOutput location: 
    
    class source.input StreamingOutput<OutputTuple> stream; ^ input.java:26: cannot find symbol symbol  : 
    
    class OutputTuple location: 
    
    class source.input StreamingOutput<OutputTuple> stream; ^ 11 errors
    
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Java Source Udop Help (I need somebody)

    ‏2010-06-03T15:29:05Z  
    • ScottAJ
    • ‏2010-06-03T15:13:20Z
    So I would need to run "javac input.java" and then place the .class file wherever I have classLibrary equal to?

    I tried this, after changing my code to what you provided, and got a bunch of "package com.ibm.streams.spade does not exist"

    Here's the exact output:
    <pre class="jive-pre"> [sjones@drydock source]$ javac input.java input.java:3: package com.ibm.streams.spade does not exist import com.ibm.streams.spade.OperatorContext; ^ input.java:4: package com.ibm.streams.spade does not exist import com.ibm.streams.spade.OutputTuple; ^ input.java:5: package com.ibm.streams.spade does not exist import com.ibm.streams.spade.StreamingOutput; ^ input.java:6: package com.ibm.streams.spade does not exist import com.ibm.streams.spade.Tuple; ^ input.java:7: package com.ibm.streams.spade.samples.patterns does not exist import com.ibm.streams.spade.samples.patterns.TupleProducer; ^ input.java:9: cannot find symbol symbol: class TupleProducer public abstract class input extends TupleProducer { ^ input.java:15: cannot find symbol symbol : method getOperatorContext() location: class source.input getOperatorContext().getScheduledExecutorService().submit( ^ input.java:11: method does not override or implement a method from a supertype @Override ^ input.java:25: cannot find symbol symbol : class OutputTuple location: class source.input OutputTuple tuple; ^ input.java:26: cannot find symbol symbol : class StreamingOutput location: class source.input StreamingOutput<OutputTuple> stream; ^ input.java:26: cannot find symbol symbol : class OutputTuple location: class source.input StreamingOutput<OutputTuple> stream; ^ 11 errors </pre>
    Scott> So I would need to run "javac input.java" and then place the .class file wherever I have classLibrary equal to?

    This is just standard java class loading. Say you compiled your input.java and the destination/out directory was /home/scott/bin, then the compiler will produce /home/scott/bin/source/input.class since your class is in the 'source' package. You then put "/home/scott/bin" as a value of the classLibrary parameter for the JavaUdop invocation. If instead you wanted to copy the class files to a different location that matched classLibrary then you need to keep the same directory structure relative to the root directory set in classLibrary. So if classLibrary had /shared/classes then you would need to copy /home/scott/bin/source/input.class to /shared/classes/source/input.class.

    Scott> I tried this, after changing my code to what you provided, and got a bunch of "package com.ibm.streams.spade does not exist"

    You need to compile the classes with streams.spade.jar on the classpath. When using Streams Studio in Eclipse IDE then the class path is automatically setup for you.

    E.g.
    
    javac -d /home/scott/bin -classpath $STREAMS_INSTALL/lib/streams.spade.jar input.java
    


    You may want to consider using Apache Ant to manage your Java builds, it's the standard tool used by Java developers.
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Java Source Udop Help (I need somebody)

    ‏2010-06-03T22:05:43Z  
    I also just noticed your class is abstract, you need to change the definition to
    
    
    
    public 
    
    class input 
    
    extends TupleProducer
    

    if you want to use it as the implementation for JavaUdop.