Topic
  • 25 replies
  • Latest Post - ‏2013-10-28T11:46:40Z by V3R0_bhaskar_barman
SystemAdmin
SystemAdmin
1245 Posts

Pinned topic Consume Twitter Streaming API as a source

‏2011-09-22T14:40:42Z |
Hi,

I would like to use a stream of tweets from the Twitter Streaming API as the source. I just got a trial version of the Streams 2.0 installed on an Amazon EC2 instance. I've also started reading the Streams documentation but it's too much to start off. I wonder if there is any particular documentation, sample codes or tutorials that may be helpful for my problem? Thank you.
  • MikeBranson
    MikeBranson
    19 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-09-22T14:51:11Z  
    You might want to take a look at the Streams Exchange at:

    https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/communityview?communityUuid=d4e7dc8d-0efb-44ff-9a82-897202a3021e

    One of the recent contributed examples is an operator that can be used as a Twitter source.

    Mike B.
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-09-22T17:39:29Z  
    You might want to take a look at the Streams Exchange at:

    https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/communityview?communityUuid=d4e7dc8d-0efb-44ff-9a82-897202a3021e

    One of the recent contributed examples is an operator that can be used as a Twitter source.

    Mike B.
    Thanks, Mike. I just downloaded it and it looks promising for my need. But I couldn't find the JsonHelpers toolkit it uses. Have you ever seen it somewhere on developerworks?
  • MikeBranson
    MikeBranson
    19 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-09-22T17:51:59Z  
    Thanks, Mike. I just downloaded it and it looks promising for my need. But I couldn't find the JsonHelpers toolkit it uses. Have you ever seen it somewhere on developerworks?
    If you scroll down the main page for the Streams Exchange, you'll get to a section labeled "Browse Content". There's a link to a collection in that section called "Streams Operators (SPL)". Click on that and you should a number of contributions, including a package with JSON helpers in it.

    Here's a direct link to the collection if you can't locate it: https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0

    Mike B.
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-09-24T12:05:00Z  
    If you scroll down the main page for the Streams Exchange, you'll get to a section labeled "Browse Content". There's a link to a collection in that section called "Streams Operators (SPL)". Click on that and you should a number of contributions, including a package with JSON helpers in it.

    Here's a direct link to the collection if you can't locate it: https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0

    Mike B.
    Thanks.
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-03T09:42:58Z  
    hi,billhao88
    Are you working with "Twitter Streaming API"? I want to work with it! I dont know whether you would like sharing your experiences with me?

    peng chengzhang
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-03T09:44:59Z  
    hi,billhao88
    Are you working with "Twitter Streaming API"? I want to work with it! I dont know whether you would like sharing your experiences with me?

    peng chengzhang
    hi,billhao88
    Are you working with "Twitter Streaming API"? I want to work with it! I dont know whether you would like sharing your experiences with me? thanks a lot in advance!
    pengcz.nwpu@gmail.com
    peng chengzhang
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-03T19:06:14Z  
    hi,billhao88
    Are you working with "Twitter Streaming API"? I want to work with it! I dont know whether you would like sharing your experiences with me? thanks a lot in advance!
    pengcz.nwpu@gmail.com
    peng chengzhang
    I modified Cory's chunkSource operator to allow HTTP SSL connections. I used libcurl to do all the work. It is pretty simple right now. If you would like a copy of the code, I will try to submit it to Streams Exchange.
  • JimKlucar
    JimKlucar
    6 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-08T14:27:56Z  
    I modified Cory's chunkSource operator to allow HTTP SSL connections. I used libcurl to do all the work. It is pretty simple right now. If you would like a copy of the code, I will try to submit it to Streams Exchange.
    That would be nice of you, since twitter no longer allows http access. OAuth support will be needed next.
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-08T14:54:57Z  
    I modified Cory's chunkSource operator to allow HTTP SSL connections. I used libcurl to do all the work. It is pretty simple right now. If you would like a copy of the code, I will try to submit it to Streams Exchange.
    Hi,
    you can try this toolkit. Its http streaming read operator works with https

    https://www.ibm.com/developerworks/mydeveloperworks/files/app/person/060002871K/file/48f02f63-1f7b-401b-8cd4-11ed945dacfd?lang=en
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-19T00:40:02Z  
    I modified Cory's chunkSource operator to allow HTTP SSL connections. I used libcurl to do all the work. It is pretty simple right now. If you would like a copy of the code, I will try to submit it to Streams Exchange.
    of course, I hope you give me a copy of the code, or submit it to Streams Exchange. thanks a lot in advance!
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-22T06:28:41Z  
    hi,billhao88
    I am sorry for no replying you in time!
    Can you give me a copy of the code about consuming Twitter Streaming API, or submit it to Streams Exchange. thanks a lot in advance!

    BEST REGARDS
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-22T08:14:43Z  
    hi,billhao88
    I am sorry for no replying you in time!
    Can you give me a copy of the code about consuming Twitter Streaming API, or submit it to Streams Exchange. thanks a lot in advance!

    BEST REGARDS
    Hi pengchengzhang,
    Another user has already posted a SPL sample on how to connect to twitter.
    See here: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=395592&tstart=0

    You will need to download the HTTPUtils and JSONHelpers toolkits available at this location
    https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-23T05:54:42Z  
    Hi pengchengzhang,
    Another user has already posted a SPL sample on how to connect to twitter.
    See here: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=395592&tstart=0

    You will need to download the HTTPUtils and JSONHelpers toolkits available at this location
    https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0
    thanks,I will try it right now,
    thanks again!
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-26T04:56:38Z  
    thanks,I will try it right now,
    thanks again!
    Sorry for the delay. It was a few intense weeks of work towards a deadline.

    I have uploaded it to my files but I don't know how to add it to Streams Exchange. You can find the file here

    https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/person/06000018YF/file/b4d07454-2a4e-42c9-9296-0a8f4ad2358f
    Mike or Rohit, can you add it to the SPL operators collection? Thanks.

    Hao
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T00:00:22Z  
    Sorry for the delay. It was a few intense weeks of work towards a deadline.

    I have uploaded it to my files but I don't know how to add it to Streams Exchange. You can find the file here

    https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/person/06000018YF/file/b4d07454-2a4e-42c9-9296-0a8f4ad2358f
    Mike or Rohit, can you add it to the SPL operators collection? Thanks.

    Hao
    hi,wang hao
    thank you!
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T00:29:05Z  
    hi,wang hao
    thank you!
    hi,Hao Wang
    I am so sorry, maybe I have spelled your name in a wrong order in the previous reply. _

    BEST REGARDS
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T14:18:10Z  
    Hi pengchengzhang,
    Another user has already posted a SPL sample on how to connect to twitter.
    See here: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=395592&tstart=0

    You will need to download the HTTPUtils and JSONHelpers toolkits available at this location
    https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0
    I have installed infoSphere Streams 2.0.0.3 (Streams-2.0.0.3-x86-el5.trial.tar.gz) and tested some SPL examples in samples. But when I run the command 'make' in HTTPUtils(HTTPUtils V1.0.tgz), I got an error:
    Syntax error, annotations are only available if source level is 5.0

    can anyone help me? thanks in advance!

    BEST REGARDS
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T14:41:45Z  
    I have installed infoSphere Streams 2.0.0.3 (Streams-2.0.0.3-x86-el5.trial.tar.gz) and tested some SPL examples in samples. But when I run the command 'make' in HTTPUtils(HTTPUtils V1.0.tgz), I got an error:
    Syntax error, annotations are only available if source level is 5.0

    can anyone help me? thanks in advance!

    BEST REGARDS
    Hello, Please verify the version of Java installed on your system. Streams, and related toolkits have been tested on IBM java 6.
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T14:50:14Z  
    Hello, Please verify the version of Java installed on your system. Streams, and related toolkits have been tested on IBM java 6.
    I installed java ibm-java-i386-sdk-6.0-9.2.i386.rpm in Streams-2.0.0.3-x86-el5.trial.tar.gz, and when I run "gij --version" I got the messages
    as follows:

    java version "1.4.2"
    gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-51)

    Copyright (C) 2006 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    thanks a lot!
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T15:12:44Z  
    Hello, Please verify the version of Java installed on your system. Streams, and related toolkits have been tested on IBM java 6.
    the error "parameterized types are only available if source level is 5.0" indicate the java version is too low, but when I run
    abelard@dillon httputils$ javac, I got the messages as follows:
    Compliance options:
    -1.3 use 1.3 compliance level (implicit -source 1.3
    -target 1.1)
    -1.4 + use 1.4 compliance level (implicit -source 1.3
    -target 1.2)
    -1.5 use 1.5 compliance level (implicit -source 1.5
    -target 1.5)
    -1.6 use 1.6 compliance level (implicit -source 1.6
    -target 1.6)
    -source <version> set source level: 1.3 to 1.6 (or 5, 5.0, 6 or 6.0)
    -target <version> set classfile target level: 1.1 to 1.6 (or 5, 5.0, 6 or
    6.0)

    these messages tell me that it can support level 6?

    or please tell me your installed java package's name !

    thanks a lot!
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T15:55:09Z  
    I installed java ibm-java-i386-sdk-6.0-9.2.i386.rpm in Streams-2.0.0.3-x86-el5.trial.tar.gz, and when I run "gij --version" I got the messages
    as follows:

    java version "1.4.2"
    gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-51)

    Copyright (C) 2006 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions. There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

    thanks a lot!
    From the above post, it appears you still may not have the correct version of Java successfully installed on your system, or at least not in your PATH and/or JAVA_HOME environment variable(s) in the correct place. (See the install/admin guide for Streams for complete instructions on how to install and configure the IBM-provided rpm of Java 6.0 that's supplied with the product.)

    When you run "java -version" from the command line you should see:
    
    [~] >java -version java version 
    "1.6.0" Java(TM) SE Runtime Environment (build pxa6460sr9fp2-20110625_01(SR9 FP2)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr9-20110624_85526 (JIT enabled, AOT enabled) J9VM - 20110624_085526 JIT  - r9_20101028_17488ifx17 GC   - 20101027_AA) JCL  - 20110530_01
    


    and when you run "java -fullversion from the command line you should see:
    
    [~] >java -fullversion java full version 
    "JRE 1.6.0 IBM Linux build pxa6460sr9fp2-20110625_01 (SR9 FP2)"
    


    Regards...Randy
  • SystemAdmin
    SystemAdmin
    1245 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T17:20:17Z  
    the error "parameterized types are only available if source level is 5.0" indicate the java version is too low, but when I run
    abelard@dillon httputils$ javac, I got the messages as follows:
    Compliance options:
    -1.3 use 1.3 compliance level (implicit -source 1.3
    -target 1.1)
    -1.4 + use 1.4 compliance level (implicit -source 1.3
    -target 1.2)
    -1.5 use 1.5 compliance level (implicit -source 1.5
    -target 1.5)
    -1.6 use 1.6 compliance level (implicit -source 1.6
    -target 1.6)
    -source <version> set source level: 1.3 to 1.6 (or 5, 5.0, 6 or 6.0)
    -target <version> set classfile target level: 1.1 to 1.6 (or 5, 5.0, 6 or
    6.0)

    these messages tell me that it can support level 6?

    or please tell me your installed java package's name !

    thanks a lot!
    Randy is correct. I had the same problem before. You need to get Java/Javac 1.6 in your path and as the default Java version.
  • pengchengzhang
    pengchengzhang
    11 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-30T23:40:46Z  
    Randy is correct. I had the same problem before. You need to get Java/Javac 1.6 in your path and as the default Java version.
    Thank you both!
    Now I compiled httputil successfully!
    The reason for the previous error was that I installed a package gcj(GNU Compiler for Java) with the lower java version 1.4.2 when I run 'yum install ant'.
  • V3R0_bhaskar_barman
    V3R0_bhaskar_barman
    18 Posts

    Re: Consume Twitter Streaming API as a source

    ‏2013-10-28T11:22:22Z  
    Hi pengchengzhang,
    Another user has already posted a SPL sample on how to connect to twitter.
    See here: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=395592&tstart=0

    You will need to download the HTTPUtils and JSONHelpers toolkits available at this location
    https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0

    Hi,

    I have installed the above toolkits.However the toolkit.xml has not created in the respective toolkit directory.

     

    [streamsadmin@streamstrial Prashanjeet]$ cd JSONHelpers
    [streamsadmin@streamstrial JSONHelpers]$ ls -lrt
    total 32
    -rw-r--r-- 1 1056 1056 688 Jun 16 2011 info.xml
    drwxr-xr-x 3 1056 1056 4096 Jun 16 2011 impl
    drwxr-xr-x 4 1056 1056 4096 Jun 16 2011 com.ibm.ssb.parsers.json
    -rw-r--r-- 1 1056 1056 1653 Jun 16 2011 build.xml
    -rw-r--r-- 1 1056 1056 428 Jun 16 2011 Makefile
    drwxr-xr-x 2 1056 1056 4096 Jun 16 2011 data
    -rw-r--r-- 1 1056 1056 559 Jun 16 2011 README
    -rw-r--r-- 1 1056 1056 647 Nov 8 2011 Main.spl

     

     Could you please tell me what is the correct process to install HTTPUtils and JSONHelpers toolkits.

     

    Regards

    Bhaskar