Operator PacketDPDKSource

• network header parser result functions

use com.ibm.streamsx.network.source::*;

The PacketDPDKSource operator depends upon the DPDK libraries available from http://dpdk.org. The library must be installed on the machine where this operator is built. It has been tested with the following versions:

• DPDK 2.2

The base DPDK library supports a wide range of NICs as described at http://dpdk.org. For IBM Streams, we have tested the following:

• Mellanox CX-3
• Mellanox CX-3 Pro
• Intel e1000

• Download and untar the dpdk.tgz file from http://dpdk.org.

• Set environment variables (these are needed for DPDK and Streams builds and depend on your specific platform):

  export EXTRA_CFLAGS=-fPIC
  export RTE_SDK=/home/<userid>/dpdk-2.2.0   # (point to install directory)
  
  export RTE_TARGET=x86_64-native-linuxapp-gcc
     OR
  export RTE_TARGET=ppc_64-default-linuxapp-gcc

• Configure the DPDK build from the top directory

  cd $RTE_SDK
  make config T=x86_64-native-linuxapp-gcc
     OR
  make config T=ppc_64-power8-linuxapp-gcc

• Change configuration as needed. For example:

  # Edit the .config files
  vi ./build/.config
  
  # Set key parameters
  CONFIG_RTE_BUILD_COMBINE_LIBS=y
  CONFIG_RTE_LIBRTE_MLX4_PMD=y (MLX4 NIC)
  CONFIG_RTE_BUILD_SHARED_LIB=y (e1000 NIC)
     OR
  CONFIG_RTE_BUILD_SHARED_LIB=n (MLX4 NIC)
  
  # Build the library
  make
  
  # Complete the build
  make install T=x86_64-native-linuxapp-gcc
     OR
  make install T=ppc_64-power8-linuxapp-gcc

• There is also a Streams to DPDK glue library that must be built. To do this, make sure the RTE environment variables noted above are set and then do the following:

  cd <toolkit-directory>/com.ibm.streamsx.network/impl/src/source/dpdk
  make

• Build the Streams example application with the following steps:

  cd <toolkit-directory>/samples/SamplePacketDPDKSource
  
  # Update the sample Makefile to use the appropriate libraries for your adapter type
  vi Makefile
  
  # Modify the operator parameters as needed for your NIC and environment
  vi TestPacketDPDKSource.spl
  make

• Create a dpdk group for permisssions.

  groupadd dpdk
  usermod -a -G dpdk <userid>
  groups <userid>
  
  # You may need to log out and back in to make sure your user is actively in the group.
  # Alternatively, you could use newgrp to change the current default group for the current shell session:
  newgrp dpdk

• Configure and enable access to hugepages (You'll need to be root to do most of these):

  # Make sure hugetlbfs is mounted, and the appropriate group owner and permissions are set:
  # (May need to umount /dev/hugepages first, to make sure the right gid and mode settings take.)
  mount -t hugetlbfs nodev /dev/hugepages -o gid=dpdk,mode=1770
  
  # Enable a suitable number of huge pages of the appropriate size, on the appropriate NUMA nodes
  # The exact number and size of hugepages will depend on your workload and packet rates.
  # Note: Fewer, larger hugepages can be used if available, such as on POWER systems.
  # The exact NUMA nodes that need huge pages will depend on adapter location and which CPUs the
  # DPDK application threads will be running on.
  echo 1000 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
  
  # Ensure all users in the dpdk group can lock an unlimited amount of memory.
  # Add a line to /etc/security/limits.conf like the following:
  @dpdk - memlock unlimited
  # This will take effect at next login.
  
  # An alternative to setting up huge pages after booting each time, you could add
  # the mount options to /etc/fstab or a systemd unit file, and add a line to
  # /etc/sysctl.conf with the number of huge pages to reserve:
  vm.nr_hugepages = 1000
  # These will be spread over the NUMA nodes, and be of the default size for the system.
  # If you change these files, you will not need to run the explicit mount or echo 1000
  # commands, above, before running the DPDK applications (provided you have rebooted
  # the system after making the changes).

• Additional Items for e1000 Adapters (RHEL7):

  # Bring the interface down
  ifconfig <eth-interface> down
    OR
  ip link set dev <eth-interface> down
  
  # Load the intel DPDK uio driver:
  modprobe uio
  insmod <dpdk-path>/dpdk-2.2.0/build/kmod/igb_uio.ko
  
  # Bind the interface to the intel DPDK uio driver:
  <dpdk-path>/dpdk-2.2.0/tools/dpdk_nic_bind.py --bind=igb_uio <eth-interface>
  <dpdk-path>/dpdk-2.2.0/tools/dpdk_nic_bind.py --status
  
  # Ensure the dpdk group has full access to the new uio device:
  chown root:dpdk /dev/uio0
  chmod 660 /dev/uio0
  
  # Ensure the dpdk group has full access to config and resources for the new uio device:
  chown root:dpdk /sys/class/uio/uio0/device/config
  cdmod 660 /sys/class/uio/uio0/device/config
  chown root:dpdk /sys/class/uio/uio0/device/resource*
  chmod 660 /sys/class/uio/uio0/device/resource*
  
  # Alternatively, a system-level udev rule could be used to ensure the dpdk group had access to
  # any new uio device by adding a file at /etc/udev/rules.d/10-dpdk.rules with content like:
  SUBSYSTEM=="uio", GROUP="dpdk", MODE="0660"
  # but note that this will not give access to the config and resource files, so that step would still
  # need to be done manually (or in a script).  If you do add a udev rule, it will take effect at the
  # next reboot, or, after forcing udev to reload rules (but then only at the time of the next uio bind):
  udevadm control --reload
  
  # If you need a particular interface to always be bound to the uio driver, at boot
  # time, the easiest way is to add the ifconfig down, modprobe/insmod,
  # dpdk_nic_bind.py, and chown/chmod commands to a simple startup script.

• Additional Items for Mellanox Adapters (RHEL7). Assumes OFED packages have been installed. (See Mellanox Poll Mode Driver for DPDK and Mellanox OFED for more information.)

  # Configure IB module.
  # Add a line to (or create) /etc/modprobe.d/mlnx.conf:
  options ib_uverbs disable_raw_qp_enforcement=1
  
  # Make sure the mlx4 driver is re-loaded and uses the above settings:
  modprobe mlx4_en
  /etc/init.d/openibd restart
  # "ibstat" and "/etc/init.d/openibd status" are useful commands to verify things worked.

• Other Adapters may require other additional items. See the DPDK documentation for more details.

• Running the Streams application

  # Submit normally:
  streamtool submitjob output/TestPacketDPDKSource.sab
  
  # Packet output can be found in the "data" directory.