Operator NetflowMessageParser

Specialized toolkits - release 4.3.1.0-prod20190605 > com.ibm.streamsx.network 3.2.1 > com.ibm.streamsx.network.parse > NetflowMessageParser

Primitive operator image not displayed. Problem loading file: ../../image/tk$com.ibm.streamsx.network/op$com.ibm.streamsx.network.parse$NetflowMessageParser.svg

NetflowMessageParser is an operator for the IBM Streams product that parses individual fields of Netflow messages received in input tuples, and emits tuples containing message data. The operator may be configured with one or more output ports, and each port may be configured to emit different tuples, as specified by output filters. The tuples contain individual fields from the input message, as specified by output attribute assignments.

The NetflowMessageParser operator expects only Netflow messages in its input tuples, without any of the headers that precede them in network packets. The PacketLiveSource and PacketFileSource operators can produce tuples that contain Netflow messages with the PAYLOAD_DATA() output attribute assignment function.

The NetflowMessageParser operator consumes input tuples containing Netflow messages, parses indivdual fields in flow records within Netflow version 5 and version 9 messages, selects flows to emit as output tuples with filter expressions, and assigns values to them with output attribute assignment expressions.

Output filters and attribute assignments are SPL expressions. They may use any of the built-in SPL functions, and any of these functions, which are specific to the NetflowMessageParser operator:

For Netflow version 5 messages, there are result functions for each field in the message header and flow records. For Netflow version 9 messages, there are result functions for each field in the message header and the standardized fields in flow records. Nonstandardized fields in flow records can be accessed by field number.

The NetflowMessageParser operator emits a tuple on each output port for each flow record within a Netflow version 5 or 9 message, optionally filtered by the 'outputFilters' parameter. Specified fields from each flow record are assigned to output attributes with the Netflow parser result functions. All attributes of all output ports must be assigned values, either with explicit assignment expressions, or implicitly by copy from input tuples.

This operator is part of the network toolkit. To use it in an application, include this statement in the SPL source file: 


use com.ibm.streamsx.network.parse::*;

Sample Applications

The network toolkit includes several sample applications that illustrate how to use this operator.

References

Netflow version 5 messages and the fields they contain are described here:

Netflow version 9 messages and the fields they contain are described here:

The result functions that can be used in boolean expressions for the outputFilters parameter and in output attribute assignment expressions are described here:

Summary

Ports
This operator has 1 input port and 0 or more output ports.
Windowing
This operator does not accept any windowing configurations.
Parameters
This operator supports 4 parameters.

Required: messageAttribute, sourceAttribute

Optional: outputFilters, processorAffinity

Metrics
This operator does not report any metrics.

Properties

Implementation
C++
Threading
Always - Operator always provides a single threaded execution context.

Input Ports

Ports (0)

The NetflowMessageParser operator requires one input port. One input attribute must be of type blob and must contain a Netflow message, excluding the network headers that proceed them in network packets, as specified by the required parameter messageAttribute.

The PAYLOAD_DATA() output assignment function of the PacketLiveSource and PacketFileSource operators produces attributes that can be consumed by the NetflowMessageParser operator.

Properties

Output Ports

Assignments
This operator allows any SPL expression of the correct type to be assigned to output attributes.
Ports (0...)

The NetflowMessageParser operator requires one or more output ports.

Each output port will produce one output tuple for each input tuple if the corresponding expression in the outputFilters parameter evaluates true, or if no outputFilters parameter is specified.

Output attributes can be assigned values with any SPL expression that evaluates to the proper type, and the expressions may include any of the Netflow result functions. Output attributes that match input attributes in name and type are copied automatically.

Properties

Parameters

Required: messageAttribute, sourceAttribute

Optional: outputFilters, processorAffinity

messageAttribute

This required parameter specifies an input attribute of type blob that contains a Netflow message to be parsed by the operator.

Properties

outputFilters

This optional parameter takes a list of SPL expressions that specify which Netflow messages should be emitted by the corresponding output port. The number of expressions in the list must match the number of output ports, and each expression must evaluate to a boolean value. The output filter expressions may include any of the Netflow result functions.

The default value of the outputFilters parameter is an empty list, which causes all Netflow messages processed to be emitted by all output ports.

Properties

processorAffinity

This optional parameter takes one expression of type uint32 that specifies which processor core the operator's thread will run on. The maximum value is P-1, where P is the number of processors on the machine where the operator will run.

Where the operator runs on a thread of its own, this parameter applies to the operator's thread. This is the situation when the operator's input port is configured as a threaded input port, and when the operator has an @parallel annotation.

Where the operator runs on the thread of an upstream operator, this parameter affects the thread of the operator that sends tuples to it. This is the situation when the operator is fused with an upstream operator.

The default is to dispatch the operator's thread on any available processor.

Properties

sourceAttribute
Properties

Libraries

common code for toolkit operators
Include Path: ../../impl/include