Operator DNSPacketDPDKSource

SPL standard and specialized toolkits > com.ibm.streamsx.network 3.0.4 > com.ibm.streamsx.network.dns > DNSPacketDPDKSource

introduction

DNSPacketDPDKSource is an operator for the IBM Streams product that captures live network packets from one of the network interfaces attached to the machine where it executes. The operator function and structure are similar to the PacketLiveSource and DNSMessageParser operators; see the documentation for those for details of common functions and general background. The primary difference between these related operators is that DNSPacketDPDKSource leverages the data plane development kit (DPDK) for higher performance.

Output filters and attribute assignments are SPL expressions. They may use any of the built-in SPL functions, and any of these functions, which are specific to the PacketFileSource operator:

The DNSPacketDPDKSource operator steps quietly over 'jmirror' headers prepended to packets by Juniper Networks 'mirror encapsulation'.

This operator is part of the network toolkit. To use it in an application, include this statement in the SPL source file:

use com.ibm.streamsx.network.dns::*;

promiscuous mode

Ethernet interfaces normally ignore packets that are not addressed to them. However, when 'promiscious' mode is enabled, they can capture all network packets on their ethernet segment, even those that are not addressed to them. This is sometimes referred to as 'network sniffing'. However, modern ethernet routers normally send only packets that are addressed to the ethernet device; 'promiscious' mode is useful only when a switch has been specifically configured to send packets to an ethernet interface that are not addressed to it. This is sometimes referred to as 'mirroring'.

The DNSPacketDPDKSource operator will enable 'promiscuous' mode in a ethernet interface when its 'promiscous' parameter is set to true.

hardware dependencies

DPDK supports a variety of ethernet devices. The DNSPacketDPDKSource operator has been tested with these adapters:

The DPDK libraries must be compiled from source code and installed on the machine where the network adapter is installed. Streams processing elements (PEs) must be placed on this machine.

software dependencies

DPDK depends upon additional Linux software packages, beyond those required for Streams. In particular, it requires libraries that match the specific version of the kernel your machine uses. To install these libraries, type these commands at a prompt:

bash> sudo yum install glibc-devel
bash> sudo yum install libibverbs-devel
bash> sudo yum install "kernel-devel-$(uname -r)"

installing and configuring DPDK

The DPDK user guide has detailed instructions for installing DPDK and configuring it for particular ethernet devices. These notes are a summary of those instructions. They include many Linux commands that should be entered at a command prompt in a Terminal window. In the instructions below, these commands are prefixed with 'bash>'. Many of the commands require 'root' privileges; they are prefixed with 'bash> sudo'.

identify ethernet device and network interface

Your machine probably has multiple ethernet interfaces. To configure the DPDK device driver, you will need to specify which ethernet device you intend to use with DPDK, and which Linux network interface is assigned to it.

First, display a list of ethernet devices installed in your machine by typing this at a command prompt:

bash> lspci | grep -i ethernet
0002:01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
0003:05:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0003:05:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0003:05:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0003:05:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0005:01:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

If none of the supported ethernet devices listed above under 'hardware dependencies' is displayed, the DNSPacketDPDKSource operator is not supported.

Otherwise, note the numbers preceding the word 'Ethernet' for the device you intend to use with DPDK. These numbers are the PCI 'bus:slot.function' address of the device: they identify the PCI bus the device uses, the physical slot the device is installed in, and the function number (the port number, for cards with multiple ethernet ports).

Then, display a list of Linux network interfaces, and the ethernet devices they are assigned to, by typing this at a command prompt:

bash> ls -la /sys/class/net
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP2p1s0 -> ../../devices/pci0002:00/0002:00:00.0/0002:01:00.0/net/enP2p1s0
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP2p1s0d1 -> ../../devices/pci0002:00/0002:00:00.0/0002:01:00.0/net/enP2p1s0d1
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP3p5s0f0 -> ../../devices/pci0003:00/0003:00:00.0/0003:01:00.0/0003:02:09.0/0003:05:00.0/net/enP3p5s0f0
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP3p5s0f1 -> ../../devices/pci0003:00/0003:00:00.0/0003:01:00.0/0003:02:09.0/0003:05:00.1/net/enP3p5s0f1
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP3p5s0f2 -> ../../devices/pci0003:00/0003:00:00.0/0003:01:00.0/0003:02:09.0/0003:05:00.2/net/enP3p5s0f2
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP3p5s0f3 -> ../../devices/pci0003:00/0003:00:00.0/0003:01:00.0/0003:02:09.0/0003:05:00.3/net/enP3p5s0f3
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP5p1s0 -> ../../devices/pci0005:00/0005:00:00.0/0005:01:00.0/net/enP5p1s0
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 enP5p1s0d1 -> ../../devices/pci0005:00/0005:00:00.0/0005:01:00.0/net/enP5p1s0d1
lrwxrwxrwx.  1 root root 0 Sep 28 09:49 lo -> ../../devices/virtual/net/lo

Find the Linux network interface of your ethernet device by matching its PCI 'bus:slot:function' address to those listed, preceding the string '/net/'.

For example, the 'lspci' list shows two Mellanox ethernet devices with PCI addresses '0002:01:00.0' and '0005:01:00.0'. The 'ls' list shows the adapters are assigned to network interfaces 'enP2p1s0' and 'enP5p1s0', respectively.

Select the Linux network interface and PCI 'bus:slot.function' address of the ethernet device you intend to use with DPDK, and remember them for use below under 'prepare Linux user account for DPDK'. To verify your selection, type this command at a prompt, substituting your network interface name, and verify that 'bus-info:' shows the PCI address of your ethernet device:

bash> ethtool -i enP2p1s0
driver: mlx4_en
version: 3.3-2.0.0 (28 Jul 2016)
firmware-version: 2.36.5150
bus-info: 0002:01:00.0

create Linux user group for DPDK

DPDK will lock the memory pages containing its buffers to prevent the kernel from swapping them out to disk while it is using them. Users who run the DNSPacketDPDKSource operator will need permission to do this.

To grant this permission, create a Linux user group for DPDK and add the user accounts that will run the DNSPacketDPDKSource operator to the group.

For example, to create a Linux user group 'dpdk' and add your own user account to it, open a Terminal window and type these commands at the prompt:

bash> sudo groupadd dpdk
bash> sudo usermod -a -G dpdk $USER

To verify that your user account is in the 'dpdk' group, type this at a command prompt:

bash> groups $USER
streamsdpdk : streamsdpdk dpdk

To grant permission to lock memory pages, edit the '/etc/security/limits.conf' system file:

bash> sudo vi /etc/security/limits.conf

and insert this line into the 'limits.conf' file:

@dpdk - memlock unlimited

prepare Linux user account for DPDK

You will need some environment variables whenever you use DPDK. To set these variables permanently, edit the '$HOME/.bashrc' script by typing this at a command prompt:

vi $HOME/.bashrc

and insert these lines into the '.bashrc' file, substituting the Linux network interface and PCI 'bus:slot.function' address of the ethernet device you intend to use with DPDK:

export RTE_SDK=$HOME/dpdk-2.2.0
[[ $( uname -m ) == "x86_64" ]] && export RTE_TARGET=x86_64-native-linuxapp-gcc
[[ $( uname -m ) == "ppc64le" ]] && export RTE_TARGET=ppc_64-power8-linuxapp-gcc
export DPDK_INTERFACE=network-interface-name
export DPDK_DEVICE=$( ethtool -i $DPDK_INTERFACE | gawk '/bus-info:/ { print $2 }' )

log off and log back on

After creating the 'dpdk' user group and making the changes to $HOME/.bashrc described above, log off your user account and log back in to put the changes into effect.

configure Linux 'hugepages'

DPDK uses Linux kernel 'hugepages' support to allocate large buffers with minimal 'translation lookaside buffer (TLB)' overhead. The size and number of 'hugepages' available for DPDK must be configured in the Linux kernel. This requires 'root' access, and may require rebooting the machine.

Note that the memory allocated for hugepages will reduce the memory available for Streams and other applications.

To display the amount of memory installed in your machine, and the amount of memory available, type this at a a command prompt:

bash> cat /proc/meminfo | grep -i mem
MemTotal:       263636096 kB
MemFree:        226797632 kB
MemAvailable:   249287680 kB
Shmem:            131776 kB

To display the size and number of hugepages currently allocated, type this at a command prompt:

bash> cat /proc/meminfo | grep -i hugepages
AnonHugePages:         0 kB
HugePages_Total:      10
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:      16384 kB

The supported sizes of hugepages depend upon the machine architecture. Typical sizes for the architectures supported by IBM Streams include:

The size of hugepages on your machine is set by Linux kernel parameters when the machine is booted. If you want to change the size of hugepages on your machine, please refer to these documents for details:

The number of hugepages is determined by the 'vm.nr_hugepages' system control parameter. This parameter can be set when the system is booted, and it can be changed later while the system is running.

For example, to set the 'vm.nr_hugepages' parameter to 1000 when the system boots, edit the '/etc/sysctl.conf' system file and set the parameter like this:

vm.nr_hugepages=1000

To set the 'vm.nr_hugepages' parameter to 1000 while the system is running, type this at a command prompt:

bash> sudo sysctl -w vm.nr_hugepages=1000

To verify that the 'vm.nr_hugepages' parameter is set to 1000, type this at a command prompt:

bash> sysctl vm.nr_hugepages
vm.nr_hugepages = 1000

To verify that the number of hugepages available is actually 1000, type this at a command prompt:

bash> cat /proc/meminfo | grep -i hugepages
AnonHugePages:         0 kB
HugePages_Total:    1000
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:      16384 kB

If the number of hugepages has not 1000, memory may be too fragemented. In this case, you will need to reboot the machine and then set the number of hugepages.

To make the hugepages accessible to the user accounts that will run the DNSPacketDPDKSource operator, create and mount a directory for them by typing these commands at a prompt:

bash> sudo mkdir /dev/hugepages
bash> sudo mount -t hugetlbfs nodev /dev/hugepages -o gid=dpdk,mode=1770

To ensure the hugepages are mounted in the '/dev/hugepages' directory when the machine is rebooted, edit the '/etc/fstab' system file by typing this command at a prompt:

bash> sudo vi /etc/fstab

and inserting this line into the file:

nodev  /dev/hugepages  hugetlbfs  gid=dpdk,mode=1770  0  0

get DPDK library and utilities

The DPDK source code needs to be downloaded, configured for your machine's architecture and ethernet device, and compiled into libraries and utilities.

First, download the DPDK source code and configure it for your machine's architecture by typing these commands at a prompt:

bash> wget http://fast.dpdk.org/rel/dpdk-2.2.0.tar.xz
bash> tar -xvf dpdk-2.2.0.tar.xz
bash> cd $RTE_SDK
bash> make config T=$RTE_TARGET

Then, edit the DPDK configuration file by typing this command:

bash> vi ./build/.config

and change these parameters in the '.config' file:

CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_MAX_MEMSEG=1024

and, if your machine has a Mellanox ethernet device, change this parameter in the '.config' file, too:

CONFIG_RTE_LIBRTE_MLX4_PMD=y 

Then, compile DPDK by typing this command at a prompt:

bash> EXTRA_CFLAGS=-fPIC make                
bash> EXTRA_CFLAGS=-fPIC make install T=$RTE_TARGET

re-compile Streams network toolkit

If you don't already have the Streams network toolkit installed in the $STREAMS_INSTALL/toolkits directory, get it by typing these commands at a prompt:

bash> cd $HOME/git
bash> git clone git@github.com:IBMStreams/streamsx.network.git

In any case, you will need to compile the Streams DPDK glue library. To do this, type these commands at a prompt:

bash> .../streamsx.network
bash> ant

bind 'e1000' ethernet device to DPDK

Before Streams can use an ethernet device with DPDK, it must be bound to a DPDK device driver.

For the 'e1000' family of devices, load the DPDK device driver into the Linux kernel by typing these commands at a prompt:

bash> sudo modprobe uio
bash> sudo insmod $RTE_SDK/build/kmod/igb_uio.ko || true

The DPDK device driver requires exclusive use of the ethernet device. To stop the Linux device driver, type these commands at a prompt:

bash> sudo ip link set dev $DPDK_INTERFACE down
bash> ip link show dev $DPDK_INTERFACE

To bind an ethernet device of the 'e1000' family to the DPDK device drier, type these commands at a prompt:

bash> sudo $RTE_SDK/tools/dpdk_nic_bind.py --bind=igb_uio $DPDK_DEVICE
bash> sudo $RTE_SDK/tools/dpdk_nic_bind.py --status

Then grant read/write access to the DPDK device driver's files to the 'dpdk' user group by typing these commands at a prompt:

bash> sudo chown root:dpdk /dev/uio0 /sys/class/uio/uio0/device/config /sys/class/uio/uio0/device/resource*
bash> sudo chmod 664 /dev/uio0 /sys/class/uio/uio0/device/config /sys/class/uio/uio0/device/resource*

You can now run one of the sample applications that use DPDK, or run your own SPL application.

unbind 'e1000' ethernet device from DPDK

After Streams has finished using an ethernet device with DPDK, it should be rebound to the Linux device driver.

To bind an ethernet device of the 'e1000' family to the Linux device driver, type these commands at a prompt:

bash> sudo $RTE_SDK/tools/dpdk_nic_bind.py --bind=igb $DPDK_DEVICE
bash> sudo $RTE_SDK/tools/dpdk_nic_bind.py --status

Then start the Linux device driver by typing these commands at a prompt:

bash> sudo ip link set dev $DPDK_INTERFACE up
bash> ip link show dev $DPDK_INTERFACE

sample applications

The network toolkit includes several sample applications that illustrate how to use this operator.

references

The ethernet frame format is described here:

The ethernet header and the fields it contains are described here:

The IPv4 header and the fields it contains are described here:

The IPv6 header and the fields it contains are described here:

The UDP header and the fields it contains are described here:

DNS messages and the fields they contain are described here:

The Juniper Networks 'jmirror' encapsulation header is described here:

The Cisco Systems ''Encapsulated Remote Switch Port Analyzer (ERSPAN)' encapsulation headers are described here:

The 'Generic Route Encapsulation (GRE)' headers used by Cisco ERSPAN encapsulation are described here:

The Linux tcpdump command is described here:

The Wireshark tools are described here:

The result functions that can be used in boolean expressions for the outputFilters parameter and in output attribute assignment expressions are described here:

Summary

Ports

This operator has 0 input ports and 1 or more output ports.

Windowing

This operator does not accept any windowing configurations.

Parameters

This operator supports 7 parameters.

Required: lcore

Optional: lcoreMaster, metricsInterval, nicPort, nicQueue, outputFilters, promiscuous

Metrics

This operator reports 5 metrics.

Properties

Implementation

C++

Threading

Always - Operator always provides a single threaded execution context.

Output Ports

Assignments

This operator allows any SPL expression of the correct type to be assigned to output attributes.

Ports (0...)

The DNSPacketDPDKSource operator requires one or more output ports:

Each output port will produce one output tuple for each packet received if the corresponding expression in the outputFilters parameter evaluates true, or if no outputFilters parameter is specified.

Output attributes can be assigned values with any SPL expression that evaluates to the proper type, and the expressions may include any of the DNSPacketDPDKSource result functions. Output attributes that match input attributes in name and type are copied automatically.

Properties

• TupleMutationAllowed: false
• WindowPunctuationOutputMode: Generating

Parameters

Required: lcore

Optional: lcoreMaster, metricsInterval, nicPort, nicQueue, outputFilters, promiscuous

lcore

Logical core number on which this operator will run.

Properties

• Type: int32
• Optional: false
• ExpressionMode: Expression

lcoreMaster

Logical core number to be used for the master DPDK thread.

Properties

• Type: int32
• Optional: true
• ExpressionMode: Expression

metricsInterval

This optional parameter takes an expression of type float64 that specifies the interval, in seconds, for sending operator metrics to the Streams runtime. If the value is zero or less, the operator will not report metrics to the runtime, and the output assigment functions for libpcap statistics will be zero.

The default value is '10.0'.

Properties

• Type: float64
• Cardinality: 1
• Optional: true
• ExpressionMode: Expression

nicPort

NIC port of type uint32 which sources packets for this operator instance. Obtaining the proper mapping may require use of multiple tools, and varies by NIC.

Suggested tools to help identify NIC ports:

- ifconfig

- dpdk-*/tools/dpdk_nic_bind.py --status

- ibstat (for Mellanox NICs)

- lspci | grep -i net

Properties

• Type: int32
• Optional: true
• ExpressionMode: Expression

nicQueue

NIC queue of type int32 which sources packets for this operator instance.

Properties

• Type: int32
• Optional: true
• ExpressionMode: Expression

outputFilters

This optional parameter takes a list of SPL expressions that specify which packets should be emitted by the corresponding output port. The number of expressions in the list must match the number of output ports, and each expression must evaluate to a boolean value. The output filter expressions may include any of the DNSPacketDPDKSource result functions.

The default value of the outputFilters parameter is an empty list, which causes all packets processed to be emitted by all output ports.

Properties

• Type: boolean
• Optional: true
• ExpressionMode: Expression

promiscuous

This optional parameter takes an expression of type 'boolean' that specifies whether or not 'promiscuous' mode should be enabled on the network interface.

Properties

• Type: boolean
• Optional: true
• ExpressionMode: Expression

Metrics

nPacketsReceivedCurrent - Counter

This metric counts the number of packets received by the network interface, as counted by DPDK.

nPacketsDroppedCurrent - Counter

This metric counts the number of packets dropped by the network interface, as counted by DPDK.

nBytesReceivedCurrent - Counter

This metric counts the number of bytes received by the network interface, as counted by DPDK.

nPacketsProcessedCurrent - Counter

This metric counts number of bytes of packet data processed by the operator.

nBytesProcessedCurrent - Counter

This metric counts number of bytes of packet data processed by the operator.

Libraries

General support functions.

Include Path: ../../impl/include

streams_source interface library to dpdk.

Library Name: streams_source

Library Path: ../../impl/src/source/dpdk/build/lib

Include Path: ../../impl/src/source/dpdk