Language overview

The following figure shows an example stream graph. The vertices of the stream graph are operator invocations, and the edges are streams. An operator invocation defines output streams by invoking a stream operator on input streams.

Figure 1. Stream graph for the sale-join application example

composite SaleJoin {                                                               
  graph                                                                            
    stream<rstring buyer, rstring item, decimal64 price> Bid = FileSource() {      
      param file: "BidSource.dat"; format: csv;                                    
    }                                                                              
    stream<rstring seller, rstring item, decimal64 price> Ask = FileSource() {     
      param file: "AskSource.dat"; format: csv;                                    
    }                                                                              
    stream<rstring buyer, rstring seller, rstring item> Sale = Join(Bid; Ask) {    
      window Bid   : sliding, time(30);                                            
             Ask   : sliding, count(50);                                           
      param  match : Bid.item == Ask.item && Bid.price >= Ask.price;               
      output Sale  : item = Bid.item;                                              
    }                                                                              
    () as Sink = FileSink(Sale) { param file: "Result.dat"; format: csv; }         
}         

An SPL program consists of one or more composite operators. A composite operator defines a stream graph, which consists of operator invocations. There is a one-to-one correspondence between the vertices in the figure and operator invocations in the example code. The body of an operator invocation customizes the way that the operator works. For example, the window, param, and output clauses in Lines 10-13 customize how the Join operator is invoked.

This language specification is written bottom-up, starting from types such as rstring or int32 (see topic Types). Values of these types are manipulated by expressions, such as Bid.item == Ask.item (see topic Expression language). But the core concept of SPL is the operator invocation, such as Sale = Join(Bid; Ask) (see topic Operator invocations). SPL allows users to define their own primitive or composite operators, such as SaleJoin (see topic Operator definitions). Finally, an SPL program specifies functions, operators, and types in namespaces (see topic Program structure). An extended example is listed in the topic VWAP example, and the overview of the grammar that SPL uses is listed in the topic Grammar overview.