IBM Streams 4.3.0

Developing a simple stream application

Companies build stream processing applications when they need to extract information from large data sets so that they can analyze only the information that is relevant to them.

About this task

For example, you might want to know the highest and lowest trading price for IBM stocks for a specific day. To determine, you can write a stream processing application to filter the stock transaction data to process only the IBM transaction records. Then, process the IBM transaction records to find the highest and lowest trading price for that day.

Procedure

  1. Filtering large data sets
  2. Reading data from a source file
  3. Analyzing the subset of data
  4. Combining the operators into a graph
  5. Creating a graph by using Streams Studio
  6. Testing the stream processing application built from the graph
  1. Filtering large data sets
    In this example, the stream processing application needs to filter the stock transaction data for IBM transaction records. You use the Filter operator to extract relevant information from potentially large volumes of data. As shown, the input for the Filter operator is all the transactions; the output is only the IBM transactions.
  2. Reading data from a source file
    Suppose that the stock transaction data for a day contains the following sample data in a file named StockTrades.csv.
  3. Analyzing the subset of data
    To determine the highest and lowest trading price, you need to write the logic for determining those values. In addition, you need some local variables to track the highest and lowest price that is processed so far. You can use the Custom operator to accomplish these tasks. As shown, the input stream is only the IBM transactions; there is no output stream for the customized operator.
  4. Combining the operators into a graph
    A stream processing application that is written in SPL code is a directed graph where streams of data (edges in the graph) flow into and out of operators (nodes in the graph). Operators perform analysis on the incoming data; and produce new streams of data which then flow into subsequent operators. You can use this basic pattern, input data --> operator --> output data, to create an assembly line of analytics where downstream operators use data from upstream operators to perform analytics and produce new data.
  5. Creating a graph by using Streams Studio
    The stream processing application is created in a composite operator that includes a graph of all of the other operators. Instead of writing the SPL code for the operators, you can use the Graphical Editor of Streams Studio to create the graph of the operators and their streams; and definitions for those operators and streams. From the graph, Streams Studio can generate the corresponding SPL code for the operators. When you save the graph, the tool generates the SPL code.
  6. Testing the stream processing application built from the graph
    Now, test the stream processing application to verify that the code generated from the graph is correct. You must build (compile) the stream application before you can launch (run) it. There are two execution environments to run a stream application: distributed and stand-alone.
Parent topic: IBM Streams tutorials