scott.s 2700060BQD Visits (1036)
In a previous post, we discussed composites and the kinds of genericity they can have. A kind of genericity we did not cover is operator genericity, which is when at least one operator in a composite's stream graph is passed in as a parameter.
Building on the example code from composite genericity, suppose we have the following application:
The purpose of this application is to find "suspect" remote hosts in a log file, where we define a suspect as any remote host from which 10 or more failed login attempts have originated. The steps the application takes to do this are:
One problem with our application as it is written is that it assumes the source and the sink will always be on the filesystem. It's easy to imagine wanting to perform this operation on data sources that come from the network, and wanting to report the results over the network. However, we don't want to write another version of this composite that just has a different source and sink.
The solution to our problem is operator parameters:
Now, the FindSuspects composite is operator generic in its source and sink. However, we have one problem: most source and sink adapters require parameters to configure where they should send or receive data from. There is no way to make these parameters fully generic. Of course, if we have, say, a file name parameter, we can always parameterize the value to that parameter. But we cannot parameterize the parameter itself.
Wrapping operator invocations in a composite solves this problem. Given the above definition of FindSuspects, we could invoke it with:
The composite Find
When FindSuspectsFromTCP is used as the main composite for an application,
then it retrieves log data from logs.company.com on port 514, and sends
suspects to susp
scott.s 2700060BQD Visits (1546)
In a previous post, we looked at the practice of wrapping Custom operators in composites. We observed that the result was not generic:
Technically, however, the above code does have some type genericity, just not much. We will explore what that genericity is in a moment, but first let's go over the kinds of genericity there are in SPL, and which ones can apply to a composite operator.
Operators that can handle any number of input and output ports are port number generic. Composites cannot be port number generic; composites must define the exact number of input and output ports they provide. Primitive operators, however, can be port number generic. In our example, ParseMessages defines one input and one output port.
Composites can be type generic, which means that they can handle streams of any type. ParseMessages is not type generic, because the type of the stream Out is fully specified to have the type ServiceMessage.
The type for In, however, is partially type generic. The type is not fully specified, although we have made one assumption about it: that it contains an attribute named raw that is of type rstring. In the example application we previously developed, that attribute was the only attribute in the stream type, but that does not need to be true in general. For example, we could invoke ParseMessages in this way:
Even though ParseMessages does not know about the attributes processedTime and networkName, it can still handle the type AugmentedRawMessage on its input stream because it has an rstring attribute named raw.
However, we can still make ParseMessages attribute generic. We can do this by modifying the composite to take an attribute as a parameter:
Composites that take attributes as parameters are attribute generic because they make no assumptions about an attribute's name. The type of the attribute, however, cannot be generic. In the above version of ParseMessages, the attribute we provide upon invocation must have type rstring, like this example:
If we tried to provide an attribute that was not an rstring (such as processedTime), the compiler would raise an error the first time it tried to use the inappropriately typed attribute.
Using a similar idea, we can still make ParseMessages even more generic. While we want to ensure that the output stream has the specific attributes date, hostname, service and message, there is no reason for us to *limit* the output stream's type to those attributes. However, because we fully specified the type name, we have forced that to be the case. We can remove that restriction by not fully specifying the type:
Note that we can no longer create a tuple literal of our output tuple type - creating a tuple literal requires knowing the full type of a tuple, but we want to remain partially type generic. To do so, we only require the out the output's stream type contains date, hostname, service, message and that they are type rstring. We invoke this composite in this way:
The remaining kind of genericity is operator genericity, which is possible when a composite takes an operator as its parameter. We will cover operator genericity in a future post.