GabrielaJS 270004FR6S Visits (2372)
When processing data, it is common to perform data enrichment. Enrichment is useful when the data source contains only partial information, but the analytics require additional information that is available only in other data sources. The InfoSphere Streams database toolkit contains two enrichment operators: ODBCEnrich and SolidDBEnrich. These operators require the data used for enrichment to be in a database.
In this post, we illustrate how to develop an SPL composite that serves as a generic file-based enricher. In this solution, we use a FileSource to scan the enrichment data from a file, and then store it in an in-memory map in a Custom operator. This map is keyed by the attributes used to correlate incoming tuples with the enrichment data. If the enrichment data fully fits in memory, this solution can be more efficient than querying the database every time a tuple must be enriched.
The code below shows a sample invocation of the file-based enricher (operator FileEnrich). In this example, the program generates a stream called Data, which has attributes id and city. Data is then consumed by FileEnrich, which outputs an enriched stream using the id attribute as a key. The output stream contains both Data attributes (id and city) and EnrichT attributes (id and name). Note that because EnrichT and Data share the id attribute, id appears only once in the EnrichedData stream. The FileEnrich operator receives the following parameters:
We now show the code for the generic FileEnrich composite operator. This composite is developed using 2 primitive operators and 1 Custom operator. The first operator is a FileSource (line 11). The FileSource uses the enrichmentFile parameter and produces a stream of type enrichmentType. Using a parameter to establish the type of the FileSource output stream gives users the option to use a CSV file with any set of attributes. The second operator is a Switch (line 16), which serves exclusively to control when the input stream (In) can start flowing into the downstream operator. By default, this operator has an initial status of false (i.e., blocking tuples). The status parameter indicates the action taken once a tuple arrives in the second input stream. In this case, a true value indicates that the switch will open when a tuple arrives. The third operator (lines 21-22), implemented as a Custom, is the one responsible for doing the data enrichment itself.
The Custom operator has two phases of execution. First, it builds a map based on EnrichmentData, the stream generated by the FileSource. To create this map, this operator uses the enrichmentKeyType and the enrichmentType itself (line 25). To populate the map, the operator uses the function getT
The C++ code below shows the implementation of this function. This function returns an error flag in two cases: (i) if the attribute provided as a string does not exist, and (ii) if the attribute type does not match the type of value.
The second stage of execution happens after the stream produced by the FileSource is fully processed, which is indicated by a final punctuation. At this point, the Custom operator notifies the Switch (line 38), and the data enrichment process starts. For enrichment, the Custom operator merges the attributes of Data and EnrichmentData using the assignFrom function (lines 45-46). This function assigns all matching fields from one tuple to the other, so be careful when naming the attributes of the enrichment type. If attribute names overlap, the last assigned value will prevail, which in this case is the value available in the enrichment tuple (line 46).
In summary, the FileEnrich composite has three characteristics that make it generic:
Thanks to Bugra Gedik for this example!
GabrielaJS 270004FR6S Visits (2745)
A new feature of InfoSphere Streams 3.0 allows dynamic filter expressions for applications that use
A recurring application sharing scenario is when different consuming applications are interested in processing different subsets of the exported stream. Prior to Streams 3.0, the importing application would receive all tuples available in the exported stream. This would result in waste of network resources, as the whole stream was transmitted but a filtering operation on the consuming application side would immediately discard many tuples. In a scenario where there are many different consumers, transferring the full stream multiple times wastes a significant amount of resources.
To reduce network transfers, developers can take advantage of dynamic filter expressions in Import operators. During application instantiation, the filtering expression is effectively shipped to the Export operator side. During runtime, the Export operator evaluates the filtering expression to decide which tuples should be transmitted to the consuming application.
The following figures show some SPL code using this new feature. All examples use a stream of type Schema declared as "int64 streamSubset, rstring stringSubset, uint32 random".
The segment below shows an Export operator that exports the stream produced by a Custom operator, which consumes a stream produced by a FileSource operator. In this example, the Custom operator forwards downstream all incoming tuples without doing any specific transformation. In reality, developers may substitute this operator with any arbitrary SPL topology. The invocation of the Export operator does not need to change from prior versions of Streams.
The file "sample.dat" has the following 10 lines:
In the Import side, one must now use the filter parameter, as in the example below. This instance of the Import operator receives only tuples where the streamSubset attribute has value 1 and the stringSubset attribute has value “streams”. This filtered, imported stream is processed by a Custom operator, which then sends the output directly to a FileSink. As in the example above, the Custom operator just illustrates a sample topology. The filter parameter in Import allows the construction of more complex expressions, similar to the subscription parameter.
The figure below shows the Streams instance graph when running the applications above. To illustrate the power of the dynamic filter expressions, we also run two other applications. The applications are similar to the importer application above, but use the following filtering expressions:stringSubset == "sources" || stringSubset == "sinks"
streamSubset == 2 && stringSubset == "operator"
As highlighted by the red rectangle, the Custom operator of one of the importing applications (right side) receives only 2 out of the 10 tuples submitted to the Export operator (10 lines in ‘sample.dat’). The filtering for the “streams” keyword allows 4 tuples to be transmitted and the filtering for “sources” or “sinks” keywords allows 3 tuples to be transmitted. The total number of tuples transmitted by the Export operator using filtering is 9, while a configuration without filtering would transfer 30 tuples.
Summary: When the application consumes only a subset of the tuples of an exported stream, use the filter parameter of the Import operator to save network resources.