Topic
  • 18 replies
  • Latest Post - ‏2014-01-15T09:53:01Z by AdrianL
DanDebrunner
DanDebrunner
25 Posts

Pinned topic Connectivity toolkit with HTTP REST APIs to streaming data added - Now at version 2.1.0

‏2012-06-22T15:32:44Z |

 This example toolkit com.ibm.ssb.inet provides a number of operators to provide connectivity to external systems using well-known standards.

 
A main use is the HTTPTupleView operator that provides HTTP GET access to the contents of a stream. This then allows browser-based visualization of streaming data using JavaScript widgets, such as Dojo Charting.
 
Also provided is the ability to inject tuples into a stream using HTTP POST, a telnet sink operator and the ability to invoke actions on local JMX objects.
 
 
Requires IBM InfoSphere Streams 3.0 or later
 
Also read this topic to see the additional functionality added over time, including the latest 2.1.0 version.
Updated on 2013-11-05T16:35:42Z at 2013-11-05T16:35:42Z by DanDebrunner
  • DanDebrunner
    DanDebrunner
    25 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added

    ‏2012-07-10T18:43:49Z  
     Version 1.0.4 has been uploaded which adds support for query parameters attribute and suppress for HTTPTupleView to allow control over what data is extracted from tuples for a GET request.
    Updated on 2012-07-10T18:43:49Z at 2012-07-10T18:43:49Z by DanDebrunner
  • DanDebrunner
    DanDebrunner
    25 Posts

    DOJO live chat and table example!

    ‏2012-07-25T22:28:54Z  
     The StreamsExchange wiki now has a page with an example of using a DOJO chart and table with HTTPTupleView to have live visualization of streaming data from a browser.
     
  • DanDebrunner
    DanDebrunner
    25 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added

    ‏2012-08-09T17:52:02Z  
     Version 1.0.6 has been uploaded which includes a workaround for PE's terminating due to SIGPIPE signals and the ability for a toolkit to provide resources (HTML, javascript etc.) from HTTPTupleView at a fixed URL context address regardless of the location of the application's data directory. See the com.ibm.ssb.inet_README.pdf contained in the toolkit for details.
  • danlopezv
    danlopezv
    4 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added

    ‏2012-10-09T21:20:26Z  
     Can you please provide a more in-depth step by step guide on how to run HTTPTupleView example? I created a new SPL Application project and copy pasted the main composite code into the project. Then added com.ibm.ssb.inet to the toolkit location in Streams Studio and added the dependency to the project. I also copied the data folder containing the HTML file and JS file into the Resources folder of the project replacing the data folder that already had. I built and launched the SPL App as Standalone but when I try to access 127.0.0.1:8080 in a browser it doesn't connect. Any suggestions?
  • danlopezv
    danlopezv
    4 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added

    ‏2012-10-10T21:08:52Z  
     I figured it out. All the jetty and servlet jar files from text toolkit were missing, that's why the web server was not running at all. I download it the missing files and put them at the right directory and now it's working like a charm.
  • DanDebrunner
    DanDebrunner
    25 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added

    ‏2012-10-17T15:58:39Z  
    • danlopezv
    • ‏2012-10-10T21:08:52Z
     I figured it out. All the jetty and servlet jar files from text toolkit were missing, that's why the web server was not running at all. I download it the missing files and put them at the right directory and now it's working like a charm.
     Which version of Streams are you using, as the first post indicates Streams 2.0.0.4 is required/
  • DanDebrunner
    DanDebrunner
    25 Posts

    Support for IBM InfoSphere Streams 3.0 added

    ‏2013-01-16T06:44:05Z  
     Version 1.0.10 has been uploaded which includes:
    • Support for IBM InfoSphere Streams 3.0 (and continued support for 2.0.0.4)
    • Support for loading web applications (war files) by the REST operators from the webapps directory relative to the context base
    • WebContext operator to allow a toolkit to provide web resources at a known context (URL path)
    • WebAppsPortsConnector operator to allow arbitrary servlets through web applications to interact with input ports and/or output ports
    See the com.ibm.ssb.inet_README.pdf contained in the toolkit for details.
  • DanDebrunner
    DanDebrunner
    25 Posts

    Support for Jetty 7.6.10

    ‏2013-04-12T15:58:31Z  
     The HTTPTupleView and HTTPTupleInject operators now can use Jetty 7.6.10.
     
    By default the toolkit will use Jetty 6 from the Streams install.

    To use  Jetty 7.6.10 create a symbolic link from com.ibm.ssb.inet/opt/jetty to a Jetty 7.6.10 install.

    For example
    1. Unpack Jetty 7.6.10 in the opt directory in the toolkit. This will create the directory  jetty-distribution-7.6.10.v20130312 in opt.
    2. Create a symbolic link jetty that links to jetty-distribution-7.6.10.v20130312
      cd com.ibm.ssb.inet/opt
      ln -s jetty-distribution-7.6.10.v20130312 jetty
    The next version of the toolkit will support arbitrary versions of Jetty 7 or later
  • danlopezv
    danlopezv
    4 Posts

    Re: Support for Jetty 7.6.10

    ‏2013-05-09T14:36:46Z  
     The HTTPTupleView and HTTPTupleInject operators now can use Jetty 7.6.10.
     
    By default the toolkit will use Jetty 6 from the Streams install.

    To use  Jetty 7.6.10 create a symbolic link from com.ibm.ssb.inet/opt/jetty to a Jetty 7.6.10 install.

    For example
    1. Unpack Jetty 7.6.10 in the opt directory in the toolkit. This will create the directory  jetty-distribution-7.6.10.v20130312 in opt.
    2. Create a symbolic link jetty that links to jetty-distribution-7.6.10.v20130312
      cd com.ibm.ssb.inet/opt
      ln -s jetty-distribution-7.6.10.v20130312 jetty
    The next version of the toolkit will support arbitrary versions of Jetty 7 or later

    Hi Dan,

    The HTTPTupleInjection operator is written to expect POST data encoded as key-value pairs

    i.e. request.getParameter("paramName")

    Is it possible that in future versions the operator can be able to get data with different encoding schema? e.g. If I send POST data as XML or JSON format that the operator might be able to read it and pass it to streams as a single string.

    public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException

    {

       StringBuffer sb = new StringBuffer();

       String line = null;

       try

       {

          BufferedReader reader = request.getReader();

          while((line = reader.readLine()) != null)

             sb.append(line);

       }

       catch(Exception e){}

       String toStreams = sb.toString();

    }

    Regards,

    Daniel Lopez

  • DanDebrunner
    DanDebrunner
    25 Posts

    Re: Support for Jetty 7.6.10

    ‏2013-05-11T16:01:16Z  
    • danlopezv
    • ‏2013-05-09T14:36:46Z

    Hi Dan,

    The HTTPTupleInjection operator is written to expect POST data encoded as key-value pairs

    i.e. request.getParameter("paramName")

    Is it possible that in future versions the operator can be able to get data with different encoding schema? e.g. If I send POST data as XML or JSON format that the operator might be able to read it and pass it to streams as a single string.

    public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException

    {

       StringBuffer sb = new StringBuffer();

       String line = null;

       try

       {

          BufferedReader reader = request.getReader();

          while((line = reader.readLine()) != null)

             sb.append(line);

       }

       catch(Exception e){}

       String toStreams = sb.toString();

    }

    Regards,

    Daniel Lopez

    That's a possibility for the future.

    For now you should be able to have the JSON in a single parameter and then using the JSONHelpers operators (on Streams Exchange) to convert them into a tuple. So the HTTPTupleInjection would just have a single attribute of type rstring.

    Similar for XML, but using the XMLParse operator.

  • danlopezv
    danlopezv
    4 Posts

    Re: Support for Jetty 7.6.10

    ‏2013-05-13T14:04:32Z  

    That's a possibility for the future.

    For now you should be able to have the JSON in a single parameter and then using the JSONHelpers operators (on Streams Exchange) to convert them into a tuple. So the HTTPTupleInjection would just have a single attribute of type rstring.

    Similar for XML, but using the XMLParse operator.

    Well parsing the data is not the problem in this case. The problem is that the HTTP post method is expecting content type: application/x-www-form-urlencoded and the appliance that is sending the data to the web server is sending it with content type: application/xml (parameters are never set).

  • DanDebrunner
    DanDebrunner
    25 Posts

    Version 2.1.0

    ‏2013-11-05T16:43:36Z  

    Version 2.1.0 provides these updates:

    • Support for InfoSphere Streams 3.0, 3.1 and 3.2.
    • Jetty 8.1.3 provided in the toolkit from Eclipse 4.2.2
    • Richer SPL type support for JSON output of HTTPTupleView
      • The JSON encoding added in Streams 3.0 is now used to convert tuples to JSON
      • Thus SPL maps, lists, sets and embedded tuples are supported by the operator

     Support for Streams 2.0 is dropped, please use the older versions if you need run on Streams 2.0.

  • madmanu
    madmanu
    2 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added - Now at version 2.1.0

    ‏2013-11-18T13:20:59Z  

    Dan,

    Thanks a lot for the Toolkit. Forgive my ignorance, would it be a good idea to use the Jetty server to expose a URL that an external application can push to Streams ? Would that be a possible use case for the toolkit.

    Thanks,

    Mano..

     

  • DanDebrunner
    DanDebrunner
    25 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added - Now at version 2.1.0

    ‏2013-11-18T18:08:42Z  
    • madmanu
    • ‏2013-11-18T13:20:59Z

    Dan,

    Thanks a lot for the Toolkit. Forgive my ignorance, would it be a good idea to use the Jetty server to expose a URL that an external application can push to Streams ? Would that be a possible use case for the toolkit.

    Thanks,

    Mano..

     

    That's what the HTTPTupleInject operator does. You can use HTTP POST to send data into an application as a tuple.

  • madmanu
    madmanu
    2 Posts

    Re: Connectivity toolkit with HTTP REST APIs to streaming data added - Now at version 2.1.0

    ‏2013-11-19T06:54:18Z  

    That's what the HTTPTupleInject operator does. You can use HTTP POST to send data into an application as a tuple.

    Thanks Dan.

    Mano.

  • AdrianL
    AdrianL
    2 Posts

    Cross origin access to HTTPTupleInjection

    ‏2013-12-17T16:13:55Z  

    How can I configure the Jetty server to allow cross origin access to HTTPTupleInjection?  I have a web application from which I want to sent tuples to Streams but see this error at the client:

    XMLHttpRequest cannot load http://localhost:8080/fromPsinapse/PriorityNodes/ports/output/inject/0?ipAddress=9.20.150.214. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8001' is therefore not allowed access.

    I've investigated using the instructions here: http://wiki.eclipse.org/Jetty/Feature/Cross_Origin_Filter but cannot work out where the web.xml file goes for the embedded server.

    Regards,
    Adrian
     

  • david.cyr
    david.cyr
    1 Post

    Re: Cross origin access to HTTPTupleInjection

    ‏2014-01-14T20:50:39Z  
    • AdrianL
    • ‏2013-12-17T16:13:55Z

    How can I configure the Jetty server to allow cross origin access to HTTPTupleInjection?  I have a web application from which I want to sent tuples to Streams but see this error at the client:

    XMLHttpRequest cannot load http://localhost:8080/fromPsinapse/PriorityNodes/ports/output/inject/0?ipAddress=9.20.150.214. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8001' is therefore not allowed access.

    I've investigated using the instructions here: http://wiki.eclipse.org/Jetty/Feature/Cross_Origin_Filter but cannot work out where the web.xml file goes for the embedded server.

    Regards,
    Adrian
     

    I wanted to check to see if there was a resolution to the 'Access-Control-Allow-Origin' header problem, as I am encountering the same issue.

     

    Thanks in advance for any assistance,

    d

  • AdrianL
    AdrianL
    2 Posts

    Re: Cross origin access to HTTPTupleInjection

    ‏2014-01-15T09:53:01Z  
    • david.cyr
    • ‏2014-01-14T20:50:39Z

    I wanted to check to see if there was a resolution to the 'Access-Control-Allow-Origin' header problem, as I am encountering the same issue.

     

    Thanks in advance for any assistance,

    d

    In the end the workaround I'm using is to put a proxy for HTTPTupleInjection on the same web server as the web application that sends tuples to HTTPTupleInjection, the proxy just forwards the request.  Using this technique Chrome does not see any cross origin access and so does not block the call to HTTPTupleInjection. 

    Regards,

    Adrian.