Topic
25 replies Latest Post - ‏2013-10-28T11:46:40Z by V3R0_bhaskar_barman
SystemAdmin
SystemAdmin
1245 Posts
ACCEPTED ANSWER

Pinned topic Consume Twitter Streaming API as a source

‏2011-09-22T14:40:42Z |
Hi,

I would like to use a stream of tweets from the Twitter Streaming API as the source. I just got a trial version of the Streams 2.0 installed on an Amazon EC2 instance. I've also started reading the Streams documentation but it's too much to start off. I wonder if there is any particular documentation, sample codes or tutorials that may be helpful for my problem? Thank you.
  • MikeBranson
    MikeBranson
    19 Posts
    ACCEPTED ANSWER

    Re: Consume Twitter Streaming API as a source

    ‏2011-09-22T14:51:11Z  in response to SystemAdmin
    You might want to take a look at the Streams Exchange at:

    https://www.ibm.com/developerworks/mydeveloperworks/groups/service/html/communityview?communityUuid=d4e7dc8d-0efb-44ff-9a82-897202a3021e

    One of the recent contributed examples is an operator that can be used as a Twitter source.

    Mike B.
    • SystemAdmin
      SystemAdmin
      1245 Posts
      ACCEPTED ANSWER

      Re: Consume Twitter Streaming API as a source

      ‏2011-09-22T17:39:29Z  in response to MikeBranson
      Thanks, Mike. I just downloaded it and it looks promising for my need. But I couldn't find the JsonHelpers toolkit it uses. Have you ever seen it somewhere on developerworks?
      • MikeBranson
        MikeBranson
        19 Posts
        ACCEPTED ANSWER

        Re: Consume Twitter Streaming API as a source

        ‏2011-09-22T17:51:59Z  in response to SystemAdmin
        If you scroll down the main page for the Streams Exchange, you'll get to a section labeled "Browse Content". There's a link to a collection in that section called "Streams Operators (SPL)". Click on that and you should a number of contributions, including a package with JSON helpers in it.

        Here's a direct link to the collection if you can't locate it: https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0

        Mike B.
        • SystemAdmin
          SystemAdmin
          1245 Posts
          ACCEPTED ANSWER

          Re: Consume Twitter Streaming API as a source

          ‏2011-09-24T12:05:00Z  in response to MikeBranson
          Thanks.
  • pengchengzhang
    pengchengzhang
    11 Posts
    ACCEPTED ANSWER

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-03T09:42:58Z  in response to SystemAdmin
    hi,billhao88
    Are you working with "Twitter Streaming API"? I want to work with it! I dont know whether you would like sharing your experiences with me?

    peng chengzhang
    • pengchengzhang
      pengchengzhang
      11 Posts
      ACCEPTED ANSWER

      Re: Consume Twitter Streaming API as a source

      ‏2011-11-03T09:44:59Z  in response to pengchengzhang
      hi,billhao88
      Are you working with "Twitter Streaming API"? I want to work with it! I dont know whether you would like sharing your experiences with me? thanks a lot in advance!
      pengcz.nwpu@gmail.com
      peng chengzhang
      • SystemAdmin
        SystemAdmin
        1245 Posts
        ACCEPTED ANSWER

        Re: Consume Twitter Streaming API as a source

        ‏2011-11-03T19:06:14Z  in response to pengchengzhang
        I modified Cory's chunkSource operator to allow HTTP SSL connections. I used libcurl to do all the work. It is pretty simple right now. If you would like a copy of the code, I will try to submit it to Streams Exchange.
        • JimKlucar
          JimKlucar
          6 Posts
          ACCEPTED ANSWER

          Re: Consume Twitter Streaming API as a source

          ‏2011-11-08T14:27:56Z  in response to SystemAdmin
          That would be nice of you, since twitter no longer allows http access. OAuth support will be needed next.
        • SystemAdmin
          SystemAdmin
          1245 Posts
          ACCEPTED ANSWER

          Re: Consume Twitter Streaming API as a source

          ‏2011-11-08T14:54:57Z  in response to SystemAdmin
          Hi,
          you can try this toolkit. Its http streaming read operator works with https

          https://www.ibm.com/developerworks/mydeveloperworks/files/app/person/060002871K/file/48f02f63-1f7b-401b-8cd4-11ed945dacfd?lang=en
        • pengchengzhang
          pengchengzhang
          11 Posts
          ACCEPTED ANSWER

          Re: Consume Twitter Streaming API as a source

          ‏2011-11-19T00:40:02Z  in response to SystemAdmin
          of course, I hope you give me a copy of the code, or submit it to Streams Exchange. thanks a lot in advance!
  • pengchengzhang
    pengchengzhang
    11 Posts
    ACCEPTED ANSWER

    Re: Consume Twitter Streaming API as a source

    ‏2011-11-22T06:28:41Z  in response to SystemAdmin
    hi,billhao88
    I am sorry for no replying you in time!
    Can you give me a copy of the code about consuming Twitter Streaming API, or submit it to Streams Exchange. thanks a lot in advance!

    BEST REGARDS
    • SystemAdmin
      SystemAdmin
      1245 Posts
      ACCEPTED ANSWER

      Re: Consume Twitter Streaming API as a source

      ‏2011-11-22T08:14:43Z  in response to pengchengzhang
      Hi pengchengzhang,
      Another user has already posted a SPL sample on how to connect to twitter.
      See here: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=395592&tstart=0

      You will need to download the HTTPUtils and JSONHelpers toolkits available at this location
      https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/collection/09ddaa56-cd45-4e04-b880-d52a3ab630c0
      • pengchengzhang
        pengchengzhang
        11 Posts
        ACCEPTED ANSWER

        Re: Consume Twitter Streaming API as a source

        ‏2011-11-23T05:54:42Z  in response to SystemAdmin
        thanks,I will try it right now,
        thanks again!
        • SystemAdmin
          SystemAdmin
          1245 Posts
          ACCEPTED ANSWER

          Re: Consume Twitter Streaming API as a source

          ‏2011-11-26T04:56:38Z  in response to pengchengzhang
          Sorry for the delay. It was a few intense weeks of work towards a deadline.

          I have uploaded it to my files but I don't know how to add it to Streams Exchange. You can find the file here

          https://www.ibm.com/developerworks/mydeveloperworks/files/app?lang=en#/person/06000018YF/file/b4d07454-2a4e-42c9-9296-0a8f4ad2358f
          Mike or Rohit, can you add it to the SPL operators collection? Thanks.

          Hao
          • pengchengzhang
            pengchengzhang
            11 Posts
            ACCEPTED ANSWER

            Re: Consume Twitter Streaming API as a source

            ‏2011-11-30T00:00:22Z  in response to SystemAdmin
            hi,wang hao
            thank you!
            • pengchengzhang
              pengchengzhang
              11 Posts
              ACCEPTED ANSWER

              Re: Consume Twitter Streaming API as a source

              ‏2011-11-30T00:29:05Z  in response to pengchengzhang
              hi,Hao Wang
              I am so sorry, maybe I have spelled your name in a wrong order in the previous reply. _

              BEST REGARDS
      • pengchengzhang
        pengchengzhang
        11 Posts
        ACCEPTED ANSWER

        Re: Consume Twitter Streaming API as a source

        ‏2011-11-30T14:18:10Z  in response to SystemAdmin
        I have installed infoSphere Streams 2.0.0.3 (Streams-2.0.0.3-x86-el5.trial.tar.gz) and tested some SPL examples in samples. But when I run the command 'make' in HTTPUtils(HTTPUtils V1.0.tgz), I got an error:
        Syntax error, annotations are only available if source level is 5.0

        can anyone help me? thanks in advance!

        BEST REGARDS
        • SystemAdmin
          SystemAdmin
          1245 Posts
          ACCEPTED ANSWER

          Re: Consume Twitter Streaming API as a source

          ‏2011-11-30T14:41:45Z  in response to pengchengzhang
          Hello, Please verify the version of Java installed on your system. Streams, and related toolkits have been tested on IBM java 6.
          • pengchengzhang
            pengchengzhang
            11 Posts
            ACCEPTED ANSWER

            Re: Consume Twitter Streaming API as a source

            ‏2011-11-30T14:50:14Z  in response to SystemAdmin
            I installed java ibm-java-i386-sdk-6.0-9.2.i386.rpm in Streams-2.0.0.3-x86-el5.trial.tar.gz, and when I run "gij --version" I got the messages
            as follows:

            java version "1.4.2"
            gij (GNU libgcj) version 4.1.2 20080704 (Red Hat 4.1.2-51)

            Copyright (C) 2006 Free Software Foundation, Inc.
            This is free software; see the source for copying conditions. There is NO
            warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

            thanks a lot!
            • SystemAdmin
              SystemAdmin
              1245 Posts
              ACCEPTED ANSWER

              Re: Consume Twitter Streaming API as a source

              ‏2011-11-30T15:55:09Z  in response to pengchengzhang
              From the above post, it appears you still may not have the correct version of Java successfully installed on your system, or at least not in your PATH and/or JAVA_HOME environment variable(s) in the correct place. (See the install/admin guide for Streams for complete instructions on how to install and configure the IBM-provided rpm of Java 6.0 that's supplied with the product.)

              When you run "java -version" from the command line you should see:
              
              [~] >java -version java version 
              "1.6.0" Java(TM) SE Runtime Environment (build pxa6460sr9fp2-20110625_01(SR9 FP2)) IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr9-20110624_85526 (JIT enabled, AOT enabled) J9VM - 20110624_085526 JIT  - r9_20101028_17488ifx17 GC   - 20101027_AA) JCL  - 20110530_01
              


              and when you run "java -fullversion from the command line you should see:
              
              [~] >java -fullversion java full version 
              "JRE 1.6.0 IBM Linux build pxa6460sr9fp2-20110625_01 (SR9 FP2)"
              


              Regards...Randy
          • pengchengzhang
            pengchengzhang
            11 Posts
            ACCEPTED ANSWER

            Re: Consume Twitter Streaming API as a source

            ‏2011-11-30T15:12:44Z  in response to SystemAdmin
            the error "parameterized types are only available if source level is 5.0" indicate the java version is too low, but when I run
            abelard@dillon httputils$ javac, I got the messages as follows:
            Compliance options:
            -1.3 use 1.3 compliance level (implicit -source 1.3
            -target 1.1)
            -1.4 + use 1.4 compliance level (implicit -source 1.3
            -target 1.2)
            -1.5 use 1.5 compliance level (implicit -source 1.5
            -target 1.5)
            -1.6 use 1.6 compliance level (implicit -source 1.6
            -target 1.6)
            -source <version> set source level: 1.3 to 1.6 (or 5, 5.0, 6 or 6.0)
            -target <version> set classfile target level: 1.1 to 1.6 (or 5, 5.0, 6 or
            6.0)

            these messages tell me that it can support level 6?

            or please tell me your installed java package's name !

            thanks a lot!
            • SystemAdmin
              SystemAdmin
              1245 Posts
              ACCEPTED ANSWER

              Re: Consume Twitter Streaming API as a source

              ‏2011-11-30T17:20:17Z  in response to pengchengzhang
              Randy is correct. I had the same problem before. You need to get Java/Javac 1.6 in your path and as the default Java version.
              • pengchengzhang
                pengchengzhang
                11 Posts
                ACCEPTED ANSWER

                Re: Consume Twitter Streaming API as a source

                ‏2011-11-30T23:40:46Z  in response to SystemAdmin
                Thank you both!
                Now I compiled httputil successfully!
                The reason for the previous error was that I installed a package gcj(GNU Compiler for Java) with the lower java version 1.4.2 when I run 'yum install ant'.
      • V3R0_bhaskar_barman
        V3R0_bhaskar_barman
        18 Posts
        ACCEPTED ANSWER

        Re: Consume Twitter Streaming API as a source

        ‏2013-10-28T11:22:22Z  in response to SystemAdmin

        Hi,

        I have installed the above toolkits.However the toolkit.xml has not created in the respective toolkit directory.

         

        [streamsadmin@streamstrial Prashanjeet]$ cd JSONHelpers
        [streamsadmin@streamstrial JSONHelpers]$ ls -lrt
        total 32
        -rw-r--r-- 1 1056 1056 688 Jun 16 2011 info.xml
        drwxr-xr-x 3 1056 1056 4096 Jun 16 2011 impl
        drwxr-xr-x 4 1056 1056 4096 Jun 16 2011 com.ibm.ssb.parsers.json
        -rw-r--r-- 1 1056 1056 1653 Jun 16 2011 build.xml
        -rw-r--r-- 1 1056 1056 428 Jun 16 2011 Makefile
        drwxr-xr-x 2 1056 1056 4096 Jun 16 2011 data
        -rw-r--r-- 1 1056 1056 559 Jun 16 2011 README
        -rw-r--r-- 1 1056 1056 647 Nov 8 2011 Main.spl

         

         Could you please tell me what is the correct process to install HTTPUtils and JSONHelpers toolkits.

         

        Regards

        Bhaskar

      • V3R0_bhaskar_barman
        V3R0_bhaskar_barman
        18 Posts
        ACCEPTED ANSWER

        Re: Consume Twitter Streaming API as a source

        ‏2013-10-28T11:46:40Z  in response to SystemAdmin

        Hi,

        I have installed the above toolkits. However the toolkit.xml directory has not been created.

        [streamsadmin@streamstrial Prashanjeet]$ cd JSONHelpers
        [streamsadmin@streamstrial JSONHelpers]$ ls -lrt
        total 32
        -rw-r--r-- 1 1056 1056 688 Jun 16 2011 info.xml
        drwxr-xr-x 3 1056 1056 4096 Jun 16 2011 impl
        drwxr-xr-x 4 1056 1056 4096 Jun 16 2011 com.ibm.ssb.parsers.json
        -rw-r--r-- 1 1056 1056 1653 Jun 16 2011 build.xml
        -rw-r--r-- 1 1056 1056 428 Jun 16 2011 Makefile
        drwxr-xr-x 2 1056 1056 4096 Jun 16 2011 data
        -rw-r--r-- 1 1056 1056 559 Jun 16 2011 README
        -rw-r--r-- 1 1056 1056 647 Nov 8 2011 Main.spl

        Could you please tell me the correct installation for the the above toolkits.