Boost IBM InfoSphere Streams performance with Linux channel bonding

Have you ever wondered if Linux® channel bonding would allow you to get faster throughput using IBM® InfoSphere® Streams? We have answered that question when running InfoSphere Streams release 2.0.0.2 on Red Hat Enterprise Linux release 5.5. In this article, we describe what channel bonding is at a high level, how we set up our test environment, and the results observed. In our experiments, channel bonding increased bandwidth by as much as 68 percent.

Denton Hatzenbihler (drhvm@us.ibm.com), Advisory Software Engineer, IBM

Denton HatzenbihlerDenton (Denny) Hatzenbihler has over 34 years' experience at IBM. He started his career at IBM in Manufacturing Engineering, where he worked on hardware and microcode design for various pieces of test equipment and developing testing software and processes for System 36 and 38. He then moved to the development lab, where his assignments have included developing of TCP/IP communications code on Facsimile Support 400; developing C++ code on AS/400 System License Internal Code in I/O device drivers; and porting various Lotus products to iSeries, including Domino HTTP server, Sametime server, and Lotus Workplace. He then consulted on and developed code for System i Lab Services Team on various projects, primarily relating to security and single sign-on and application design solutions for a variety of customers. He is currently working on the InfoSphere Streams development team and holds a bachelor's degree in electrical and electronics engineering from North Dakota State University.



Richard P. King (rpk@us.ibm.com), Senior Programmer, IBM

Richarc King's photoRichard King has been a part of the InfoSphere Streams team for a number of years. When he joined IBM in 1977, it was as an associate programmer working on the development of System/38. Since transferring to IBM's Research Division at IBM Thomas J. Watson Research Center in Yorktown in 1981, he has worked on a wide variety of projects, including the design and development of what became the IBM Sysplex Coupling Facility. He has a bacholor's degree in industrial engineering and operations research from Cornell University and a master's in operations research and industrial engineering from Northwestern University.



Wesley Most (wmost@us.ibm.com), Software Engineer, IBM

Wesley MostWesley Most joined IBM in 2004 as a software engineer, supporting the InfoSphere Streams cluster. He is a member of the Watson IS department at the Thomas J. Watson Research Center. He has played a lead role in the architecture, administration, and performance tuning of the systems, networking and storage to support the development and research work. He holds a bachelor's degree in computer science and engineering from the University of Connecticut.



Mike Oberholtzer (oberholt@us.ibm.com), Senior Software Engineer, IBM

Mike OberholtzerMike Oberholtzer joined IBM in 1984 as a software programmer working on testing tape devices on IBM System/36. In 1986, he moved to the IBM OS/400 operating system, working on device support and in 1989 moved on to backup and recovery support. In 1993, he began work on the IBM OS/400 Integrated File System with various development and project lead roles, and worked there until moving to InfoSphere Streams in 2008 working on various development and test-related roles. He holds a bachelor's degree in computer science from Michigan Technological University.



16 February 2012

Also available in Chinese Portuguese

Overview

This article explores the effects of channel bonding on Red Hat Enterprise Linux to IBM InfoSphere Streams throughput and latency. It describes how to set up and configure a channel-bonded environment using Red Hat Enterprise Linux systems and what kind of performance improvements can be expected from InfoSphere Streams applications running in this environment. Target readers are those familiar and skilled with InfoSphere Streams and Linux network interface configuration.


What is channel bonding?

Channel bonding is a term used to describe the practice of splitting a single stream of data between two network interface cards while presenting a single interface to the application layer. By utilizing this technique, higher data throughput can be achieved, since the data can flow on both network interface cards at the same time. Channel bonding is available on all Red Hat Enterprise Linux kernels supported by InfoSphere Streams. We have tested one channel-bonded setup and will explain the results observed.

See the Resources section for a link to Red Hat Enterprise Linux documentation on channel bonding.


Setting up channel bonding

Hardware

To achieve the best performance results with channel bonding over 1Gbps connections, consider the following hardware requirements:

  • 1Gbps switch or a line card on a larger switch with a high over-subscription ratio
  • Two Linux-capable systems with two 1Gbps Ethernet connections

    Note: If you're using a blade, use pass-through modules in the chassis. Switch modules may not obtain the best performance due to algorithms in the switch for mediating traffic.

In our testing environment, Cisco 6509 switches form the 1Gbps backbone for the private cluster network. To achieve the maximum performance for individual systems, WS-X6748-GE-TX line cards are used. These line cards are high-performance cards (40Gbps per slot), providing near 1Gbps for all 48 ports on the card. For this test, the switch is running Version 12 of Cisco's IOS and the line card is current with its firmware. If your Cisco switch is running NX-OS or you're using another vendor's products, these commands may need modification or may be different altogether. Consult with your network administrator for assistance.

For the performance numbers listed below, no special configuration was utilized in the network switch. Although some vendor switches support Link Aggregation Control Protocol (LACP), higher performance results can be obtained by an operating system-only bonding configuration. Since today's modern hardware can easily handle issues like reordering packets that are not sequential without any significant penalty, reliance on the switch hardware to do the work is not as important.

Figure 1. InfoSphere Streams channel bonding test configuration
Image shows InfoSphere Streams test set up with Streams App A on left, Streams App B on right, connected through Cisco 6509 network

Software

From a software perspective, everything you will need is already part of the Red Hat Enterprise Linux 5 distribution. To set up a channel bond in your environment, for each machine, you need to adjust the networking configuration and aggregate the link in the switch. To streamline these directions, we will assume that any server you're about to channel bond has been provisioned with Red Hat Enterprise Linux 5 and has one of two network interfaces that have been defined at build time. In our example, eth0 was set up initially, and /etc/sysconfig/network-scripts/ifcfg-eth0 looks like Listing 1.

Listing 1. /etc/sysconfig/network-scripts/ifcfg-eth0
              DEVICE=eth0
            
HWADDR=E4:1F:13:78:FE:98
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.6.24.43
NETMASK=255.255.252.0
DNS1=10.6.24.11

There are four steps involved in setting up the software for channel bonding:

  1. Alter ifcfg-eth0 in /etc/sysconfig/network-scripts
  2. Alter ifcfg-eth1 in /etc/sysconfig/network-scripts
  3. Create a new ifcfg-bond0 in /etc/sysconfig/network-scripts
  4. Describe the "bond" to the system

Note: It is best to do this work with the interface down. If you do it while up, the system will issue a warning since eth0 is now a slave, but you will get the correct networking configuration. You will find that the restart will take longer.

Step 1: For ifcfg-eth0, change the BOOTPROTO line to none. We will remove the lines pertaining to IPADDR, NETMASK and DNS. This information will be in the ifcfg-bond0 file, so don't lose the important details. The new lines for ifcfg-eth0 are below:

              MASTER=bond0
            
SLAVE=yes
USERCTL=no

Step 2: For ifcfg-eth1, we will edit the default unused configuration that the system established at provisioning:

              DEVICE=eth1
            
BOOTPROTO=dhcp
ONBOOT=no
HWADDR=e4:1f:13:78:fe:9a

Again, we will change the BOOTPROTO line as we did for the eth0 interface and add the three lines beginning with MASTER, SLAVE, and USERCTL. We also need to change ONBOOT to yes.

Step 3: We will now define the bond via the new ifcfg-bond0 file. Using the same networking information, this file will look like this:

              DEVICE=bond0
            
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.6.24.43
NETMASK=255.255.252.0
DNS1=10.6.24.11
USERCTL=no

Step 4: Since the bond will be considered a new network device with it's own kernel module, we need to provide Linux some information about this new "device." We will create a file in /etc/modprobe.d called bonding.conf. In the file, we add these two lines:

             alias bond0 bonding
             
options bond0 mode=0 miimon=100

See Resources for Red Hat Enterprise Linux documentation on channel bonding.


Testing results

Environment

We implemented a channel-bonded environment using two machines running Red Hat Enterprise Linux release 5.5. Each machine has 8 Intel Xeon CPU X5570 processors running at 2.93 GHz. We took two 1Gbps NICs and channel-bonded them together through a Cisco 6509 switch. See Figure 1.

Since our goal was to test performance, we chose the bonding method called Balance Round Robin (balance-rr). One way to improve throughput using channel-bond mode balance-rr is to change TCP's tolerance for out-of-order delivery. By increasing the tolerance for out-of-order delivery, you can reduce how quickly TCP asks for a re-transmittal of packets. The tolerance for out-of-order packets can be adjusted by setting the tcp_reordering sysctl parameter. For this test environment, we set the value to 127, utilizing the following Linux command: sysctl -w net.ipv4.tcp_reordering=127.

Setting the value to 127 effectively disables re-transmission entirely. The drawbacks of doing this are increased CPU cost of TCP, and the TCP connections will generally stay in a congested state.

Throughput

The throughput test is a simple test with a user-defined operator on one machine sending tuples to a receiving operator on a different machine. The receiving operator simply counts the incoming tuples. In addition to counting, it stores away the current timestamp when the first and last tuples are received. Determining the difference between the first and last timestamps gives the duration of time taken to receive all the tuples.

Throughput = (Number of tuples) / (duration of time taken to receive the of tuples)

IBM achieved the following test results using TCP transport configuration.

Figure 2. InfoSphere Streams throughput for TCP transport
Image shows that InfoSphere Streams throughput for TCP transport shows 60-percent increase

The same test was then run with the InfoSphere Streams applications configured to utilize LLMRUM transport over TCP.

Figure 3. InfoSphere Streams throughput with LLMRUMTCP transport
Image shows that InfoSphere Streams throughput with LLMRUMTCP transport shows almost 40-percent increase

As indicated by the previous graphs, throughput increased approximately 60 percent utilizing TCP. Throughput utilizing LLMRUM over TCP increased almost 40 percent. Throughput is lower with LLM because LLM's own scheduling may be interfering with TCP's scheduling of the traffic to the two NICs.

Latency results

The latency test measures round trip latency by using a source operator and a sink operator on the same machine and a functor operator on a different machine that merely passes to the sink operator the tuples coming to it from the source operator. The latency is calculated as the time when the tuple comes into the sink operator minus the time the tuple left the source operator. The difference in these times is accurate because the same machine's clock is used for both readings, thereby eliminating any relative clock skew.

Round-trip Latency = (End time of tuple — Start time of tuple on same machine)

IBM achieved the following latency results during testing when sending at an approximate rate of 5,000 tuples per second.

Figure 4. InfoSphere Streams latency using TCP transport
Image shows that InfoSphere Streams latency using TCP transport shows little change with small tuple sizes
Figure 5. InfoSphere Streams latency using LLMRUMTCP transport
Image shows that InfoSphere Streams latency using LLMRUMTCP transport shows little change with smaller tuple sizes

As indicated by the graphs, channel bonding has little effect on latency with smaller tuple sizes. With larger tuple sizes, latency increases, possibly due to out-of-order packet re-transmission.


Conclusion

As indicated from the experiments above, the key factors that seem to affect performance are tuple size and transport configuration. However, in many circumstances, channel bonding can be a useful tool to increase throughput for applications on IBM InfoSphere Streams.

Resources

Learn

Get products and technologies

  • Build your next development project with IBM trial software, available for download directly from developerWorks.
  • Now you can use DB2 for free. Download DB2 Express-C, a no-charge version of DB2 Express Edition for the community that offers the same core data features as DB2 Express Edition and provides a solid base to build and deploy applications.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Big data and analytics on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Big data and analytics, Information Management, Linux
ArticleID=793304
ArticleTitle=Boost IBM InfoSphere Streams performance with Linux channel bonding
publish-date=02162012