Operator PacketLiveSource

Specialized toolkits - release 4.3.1.0-prod20190605 > com.ibm.streamsx.network 3.2.1 > com.ibm.streamsx.network.source > PacketLiveSource

Primitive operator image not displayed. Problem loading file: ../../image/tk$com.ibm.streamsx.network/op$com.ibm.streamsx.network.source$PacketLiveSource.svg

PacketLiveSource is an operator for the IBM Streams product that captures live network packets from an ethernet interface, parses their network headers, and emits tuples containing packet data. The operator may be configured with one or more output ports, and each port may be configured to emit different tuples, as specified by output filters. The tuples may contain the entire packet, the payload portion of the packet, or individual fields from the network headers, as specified by output attribute assignments.

The PacketLiveSource operator reads raw network packets from an ethernet adapter, selects packets to process with input filters, parses individual fields in the packet's network headers, selects messages to emit as output tuples with output filter expressions, and assigns values to them with output attribute assignment expressions.

Input filters are PCAP filter expressions, as described here:

Output filters and attribute assignments are SPL expressions. They may use any of the built-in SPL functions, and any of these functions, which are specific to the PacketFileSource operator:

The PacketLiveSource operator recognizes these types of packet encapsulation, and steps quietly over their headers:

  • Juniper Networks 'jmirror' encapsulation
  • Cisco Systems 'Encapsulated Remote Switch Port Analyzer (ERSPAN)' encapsulation

The PacketLiveSource operator is part of the network toolkit. To use it in an application, include this statement in the SPL source file: 


use com.ibm.streamsx.network.source::*;

Capabilities

The PacketLiveSource operator requires special privileges to receive 'raw' network packets from an ethernet interface.

On machines where 'extended file attributes' (see https://en.wikipedia.org/wiki/Extended_file_attributes) are enabled in the Linux kernel, these Linux capabilities should be granted to a PE that includes the PacketLiveSource operator:

  • CAP_NET_RAW+eip
  • CAP_NET_ADMIN+eip

Linux capabilities can only be granted to an executable that resides on a local disk volume such as /tmp.

For 'standalone' Streams applications (that is, SPL source files compiled with a 'standalone' configuration), the user must grant these capabilities to the program before executing it. This requires 'root' privileges. To do this:

  • the SPL source must be compiled with the '--static-link' option

  • the 'application bundle' must be unpacked with the 'spl-app-info' command into a directory on a local disk volume such as /tmp

  • Linux capabilities must be granted to the unbundled program with this command: 

    
sudo setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' .../bin/standalone.exe

For 'distributed' Streams applications, (that is, SPL source files compiled with a 'distributed' configuration), the Streams runtime must grant these capabilities to the program before executing it. This requires 'root' privileges. To do this:

  • the Streams runtime must be installed as 'root'

  • the Streams domain must be registered as a Linux system service

  • the Streams instance must have the 'instance.canSetPeOSCapabilities=true' property

  • the Streams instance must have the 'instance.runAsUser=$USER' property

  • the Streams instance must have the 'instance.applicationBundlesPath=/tmp/...' property

  • the SPL source must be compiled with the '--static-link' option

On machines where 'extended file attributes' are not enabled in the Linux kernel, 'standalone' PEs that include the PacketLiveSource operator can be executed with 'root' privileges like this: 


sudo STREAMS_INSTALL=$STREAMS_INSTALL .../bin/standalone.exe

Promiscuous Mode

Network interfaces normally ignore packets that are not addressed to them. However, when 'promiscious' mode is enabled on a network interface, it can capture all network packets on its ethernet segment, even those that are not addressed to it. This is sometimes referred to as "network sniffing". Modern ethernet switches normally send network interfaces only packets that are addressed to them; 'promiscious' mode is useful only when a switch has been specifically configured to send packets to a network interface that are not addressed to it. This is sometimes referred to as "mirroring".

The PacketLiveSource operator will enable 'promiscuous' mode in a ethernet interface when its 'promiscous' parameter is set to true.

Dependencies

The PacketLiveSource operator depends upon the Linux 'packet capture library (libpcap)'. The library must be installed on the machine where this operator executes. It is available as an installable 'repository package (RPM)' from the 'base' RHEL and CentOS repositories. It can be installed with administrator tools such as 'yum'. This requires root privileges, which can be acquired temporarily with administrator tools such as 'sudo'.

To install libpcap, enter this command at a Linux command prompt: 


sudo yum install libpcap-devel

Alternatively, you can download the source code for a newer version of libpcap and build the library yourself. The new library can then be installed in system directories, or used where built without being installed.

To do this, download the distribution package for the latest version of libpcap from this address:

To build libpcap from source code, open a 'terminal' window and type this at a command prompt: 


cd .../directory
tar -xvf .../libpcap-X.Y.Z.tar.gz
cd .../directory/libpcap-X.Y.Z
./configure
make

To instruct the Streams compiler (that is, the 'sc' command) to use your version of libpcap instead of the system version, set these environment variables before compiling an application that contains the PacketLiveSource operator: 


export STREAMS_ADAPTERS_LIBPCAP_INCLUDEPATH=.../directory/libpcap-X.Y.Z
export STREAMS_ADAPTERS_LIBPCAP_LIBPATH=.../directory/libpcap-X.Y.Z

For more information on configuring, building, and installing libpcap, refer to its 'INSTALL.txt' file.

This operator has been tested with these versions of libpcap:

  • libpcap 0.9.4, included in RHEL/CentOS 5.x
  • libpcap 1.0.0, included in RHEL/CentOS 6.2
  • libpcap 1.4.0, included in RHEL/CentOS 6.5
  • libpcap 1.5.3, included in RHEL/CentOS 7.1
  • libpcap 1.6.1
  • libpcap 1.6.2
  • libpcap 1.7.4
  • libpcap 1.8.1

Tuning

The PacketLiveSource operator is often used in applications that ingest packets at very high rates. In these situations, the rate may be limited by the size of the Linux network buffers. If so, you can increase the size of these buffers by setting Linux network configuration parameters. Note that root privileges are required to do this.

You can display the current values of the ethernet and IP configuration parameters by entering these commands at a Linux prompt: 


/sbin/sysctl net.core
/sbin/sysctl net.ipv4

For example, these are typical default values for Linux configuration parameters in RHEL/CentOS 5 and 6: 


net.core.wmem_default = 124928
net.core.rmem_default = 124928
net.core.rmem_max = 131071
net.core.wmem_max = 131071
net.core.netdev_max_backlog = 1000

net.ipv4.tcp_rmem = 4096        87380   4194304
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.tcp_window_scaling=1

You can change the current values of Linux configuration parameters by executing these commands at a Linux prompt. For example: 


# allow packet buffers to increase up to 64MB
sudo /sbin/sysctl -w net.core.rmem_max=67108864
sudo /sbin/sysctl -w net.core.wmem_max=67108864

# increase the length of the processor input queue to 50,000
sudo /sbin/sysctl -w net.core.netdev_max_backlog=50000

# increase autotuning TCP buffer limit to 32MB
sudo /sbin/sysctl -w net.ipv4.tcp_rmem=10240 87380 33554432
sudo /sbin/sysctl -w net.ipv4.tcp_wmem=10240 87380 33554432

Changes to Linux configuration parameters with /sbin/sysctl persist only for the current instance of Linux; they will revert to their default values when Linux is rebooted. To change the default values used when Linux boots, specify them in the Linux system file /etc/sysctl.cfg.

For more information, refer to http://fasterdata.es.net/host-tuning/linux/.

Threads

The PacketLiveSource operator contains a separate thread for running libpcap, which receives network packets from the operating system and produces tuples from them.

The PacketLiveOperator contains a separate thread for reporting its metrics to the Streams runtime if the metricsInterval parameter is greater than zero.

Exceptions

The PacketLiveSource operator will throw an exception and terminate in these situations:

  • The parameter 'networkInterface' does not specify a network interface defined on the machine where the operator executes.

  • The 'inputFilter' and 'outputFilters' parameters do not specify a valid PCAP filter expression.

Sample Applications

The network toolkit includes several sample applications that illustrate how to use this operator.

References

The ethernet frame format is described here:

The IPv4 header and the fields it contains are described here:

The IPv6 header and the fields it contains are described here:

The UDP header and the fields it contains are described here:

The TCP header and the fields it contains are described here:

The Juniper Networks 'jmirror' encapsulation header is described here:

The Cisco Systems ''Encapsulated Remote Switch Port Analyzer (ERSPAN)' encapsulation headers are described here:

The 'Generic Route Encapsulation (GRE)' headers used by Cisco ERSPAN encapsulation are described here:

Linux capabilities are described here:

For more information about using Linux capabilities with Streams applications, see:

libpcap filter expressions, which are used with the inputFilter parameter, are described here:

The result functions that can be used in boolean expressions for the outputFilters parameter and in output attribute assignment expressions are described here:

Summary

Ports
This operator has 0 input ports and 0 or more output ports.
Windowing
This operator does not accept any windowing configurations.
Parameters
This operator supports 13 parameters.

Required: networkInterface

Optional: bufferSize, initDelay, inputFilter, jMirrorCheck, maximumLength, metricsInterval, outputFilters, processorAffinity, promiscuous, rateLimit, timeout, timestampType

Metrics
This operator reports 4 metrics.

Properties

Implementation
C++
Capabilities
CAP_NET_RAW+eip
CAP_NET_ADMIN+eip
Threading
Always - Operator always provides a single threaded execution context.

Output Ports

Assignments
This operator allows any SPL expression of the correct type to be assigned to output attributes.
Ports (0...)

The PacketLiveSource operator requires one or more output ports:

Each output port will produce one output tuple for each packet received (and passed by the input filter, if the inputFilter parameter is specified) if the corresponding expression in the outputFilters parameter evaluates true, or if no outputFilters parameter is specified.

Output attributes can be assigned values with any SPL expression that evaluates to the proper type, and the expressions may include any of the PacketLiveSource result functions. Output attributes that match input attributes in name and type are copied automatically.

Properties

Parameters

Required: networkInterface

Optional: bufferSize, initDelay, inputFilter, jMirrorCheck, maximumLength, metricsInterval, outputFilters, processorAffinity, promiscuous, rateLimit, timeout, timestampType

bufferSize

This optional parameter takes an expression of type uint32 that specifies the size, in bytes, of the libpcap buffer used for receiving packets. The maximum value allowed is 2,147,483,647 (that is, 2**31-1).

The default value of the bufferSize parameter is determined by libpcap.

Properties

initDelay

This optional parameter takes an expression of type float64 that specifies the number of seconds the operator will wait before it begins to produce tuples. This parameter is allowed only when the pcapFilename parameter is also specified.

The default value is '0.0'.

Properties

inputFilter

This optional parameter takes an expression of type rstring that specifies which input packets should be processed. The value of this string must be a valid PCAP filter expression, as defined here:

The default value is an empty string, which causes all packets received to be processed.

Properties

jMirrorCheck

This optional parameter takes an expression of type 'boolean' that specifies whether or not the operator should check for Juniper Networks 'jMirror' headers on packets, and step over them when found.

The default value is 'false'.

Properties

maximumLength

This optional parameter takes an expression of type uint16 that specifies the maximum length in bytes of the packet data that will be produced by the operator. This length includes all network headers; packets longer than the specified length will be truncated.

The default value is 65,535 bytes.

Properties

metricsInterval

This optional parameter takes an expression of type float64 that specifies the interval, in seconds, for sending operator metrics to the Streams runtime. If the value is zero or less, the operator will not report metrics to the runtime, and the output assigment functions for libpcap statistics will be zero.

The default value is '10.0'.

Properties

networkInterface

This required parameter takes an expression of type rstring that specifies which network interface the operator will capture packets from. The value must be one of the network interface names configured in Linux on the machine where the operator will execute. To display a list of configured network interface names, type this at a Linux command prompt: 


/sbin/ifconfig
Properties

outputFilters

This optional parameter takes a list of SPL expressions that specify which packets should be emitted by the corresponding output port. The number of expressions in the list must match the number of output ports, and each expression must evaluate to a boolean value. The output filter expressions may include any of the PacketLiveSource result functions.

The default value of the outputFilters parameter is an empty list, which causes all packets processed to be emitted by all output ports.

Properties

processorAffinity

This optional parameter takes an expression of type uint32 that specifies which processor core the operator's thread will run on. The maximum value is P-1, where P is the number of processors on the machine where the operator will run. If this parameter is specified, then the operator's thread will be dispatched only on the specified processor.

The default is to dispatch the operator's thread on any available processor.

Properties

promiscuous

This optional parameter takes an expression of type boolean that specifies whether or not 'promiscuous' mode should be enabled on the network interface. If 'promiscious' mode is enabled, the operator must execute with special Linux capabilities, as described above.

The default value is false. However, some values of the inputFilter parameter, including the default value, require that 'promiscuous' mode be enabled on the network interface, even if the promiscous parameter is not specified, or set to false.

Properties

rateLimit

This optional parameter takes an expression of type 'float64' that specifies the maximum number of times per second the RATE_LIMITED() function returns false. This can be used to limit the rate of packets sent to an output port when used in a filter expression.

The default value is '1000.0'.

Properties

timeout

This optional parameter takes an expression of type float64 that specifies a timeout for the libpcap interface. If this value is specified, the operator will call the libpcap interface again whenever it times out, until the PE containing the operator receives a shutdown signal.

The default is no timeout. This may cause the thread that runs the libpcap interface to hang on shutdown until another packet is received.

Properties

timestampType

This optional parameter takes a value of 'host', 'adapter', or 'adapter_unsynced', which specifies which type of timestamp will be assigned to packets as they are received.

The default value is 'host'.

Properties

Metrics

nBytesProcessedCurrent - Counter

This metric counts number of bytes of packet data processed by the operator. When an input filter is specified, this includes only packets that pass the filter.

nPacketsDroppedCurrent - Counter

This metric counts the number of packets dropped because there was no room in the operating system's buffer when they arrived, as counted by libpcap.

nPacketsProcessedCurrent - Counter

This metric counts number of packets processed by the operator. When an input filter is specified, this includes only packets that pass the filter.

nPacketsReceivedCurrent - Counter

This metric counts the number of packets received by the network interface, as counted by libpcap

Libraries

Command: ../../impl/bin/libpcapPath.pl
Include Path: ../../impl/include