Testing methodology

Edit online

This topic describes the testing methodology that was used in this paper.

Testing was performed by running uperf with the 20 unique tests defined in the topic Workload tests of Network workload, that scaled the simulated users for each of the five workloads. The execution of each of the 20 workload tests invoked sequentially, one after the other, are called a run. Runs were performed for 1, 2, 4 and 8 uperf pairs. The KVM host LPARs were freshly rebooted at the start of each run.

A separate run was conducted for each LPAR, network configuration, uperf pair count and when a different value was used for one of the many Linux® operating system, network, KVM, or uperf settings / tunables.

Running uperf

To run uperf, the server must first be started with the following command:
[root@kvmguest ~] # uperf -s

When the server starts, it enters listening mode waiting for one or more clients to connect.

Once the server is listening, the client can then be started with the following command:
[root@kvmguest ~] # uperf -m profile.xml -a -i 30

The -m parameter specifies the XML which contains the workload test definition or profile that uperf will use for this invocation. Two additional command line options were used. First, the -a option to tells uperf to collect all statistics which is useful for analysis purposes. Second, the -i option specifies the interval (in seconds) until the next real-time update of throughput and operations per second statistics are reported by uperf to the user.

There are a variety of additional command line parameters available. For the complete list of options, refer to the www.uperf.org Web page.

uperf Output

An example of the data generated by the command # uperf -m profile.xml -i 30 is listed below. Additionally, specifying the -a option includes the details from Group, Strand, Transaction (Txn), andFlowop sections in the results report.
# uperf -m profile.xml -a 
Starting 5 processes running profile:tcp_rr ...   0.00 seconds 
Txn1          0 /   1.00(s) =            0           5op/s
Txn2    18.49MB / 300.50(s) =   516.11Kb/s       64513op/s
Txn3          0 /   0.00(s) =            0           0op/s
------------------------------------------------------------------------
          
Total   18.49MB / 302.60(s) =   512.52Kb/s       64065op/s 

Group Details
------------------------------------------------------------------------
Group0        0 / 302.50(s) =            0           0op/s

Strand Details
------------------------------------------------------------------------
Thr0     3.70MB / 302.50(s) =   102.55Kb/s       12818op/s 
Thr0     3.70MB / 302.50(s) =   102.55Kb/s       12818op/s 
Thr0     3.70MB / 302.50(s) =   102.55Kb/s       12818op/s 
Thr0     3.70MB / 302.50(s) =   102.55Kb/s       12818op/s 
Thr0     3.70MB / 302.50(s) =   102.55Kb/s       12818op/s

Txn                Count         avg         cpu         max         min
------------------------------------------------------------------------
Txn0                   5    453.42us      0.00ns    513.58us    383.42us
Txn1             9693177    154.87us      0.00ns    200.06ms     32.28us
Txn2                   5     10.65us      0.00ns     12.38us      8.63us

Flowop             Count         avg         cpu         max         min
------------------------------------------------------------------------
accept                 5    453.10us      0.00ns    513.18us    383.18us
write            9693177      3.53us      0.00ns    232.93us      1.49us
read             9693172    151.25us      0.00ns    200.06ms    867.00ns
disconnect             5     10.50us      0.00ns     12.30us      8.47us
          
Netstat statistics for this run          

------------------------------------------------------------------------
          
Nic       opkts/s     ipkts/s      obits/s      ibits/s 
10gb2           0           0            0     24.75b/s 
1gb             1           1     2.42Kb/s    617.18b/s        
10gb1       32033       32033    17.17Mb/s    13.58Mb/s       
-------------------------------------------------------------------------------------------------     

Run Statistics 
Hostname            Time       Data   Throughput   Operations   Errors   xferbytes          nsec 
-------------------------------------------------------------------------------------------------
          
10.12.38.254     302.60s    18.49MB   512.52Kb/s   19386369     0.00      19386354   302603373999 
master           302.60s    18.49MB   512.52Kb/s   19386364     0.00      19386349   302603147820
-------------------------------------------------------------------------------------------------
          
Difference(%)     -0.00%     -0.00%        0.00%     -0.00%     0.00%

The uperf output data is separated into three sections.

The first section provides real-time updates of the execution progress of the transactions defined in the profile. In the example profile above, you will see the three transactions that were defined.
  • The first transaction was to connect each process from the client to the server.
  • The second transaction sends a 1 byte Request and reads a 1 byte Response. The throughput rate and operations per second for each transaction is updated based on the specified interval (-i) parameter (which defaults to 1 second).
  • The third transaction disconnects each client process from the server. When the transaction has completed, a final update is displayed.1

If the -a parameter is specified, four additional detail sections (group, strand, transaction and flowop) will be displayed. Each section will show totals for each element type relevant to that section.

The next section reports the Netstat statistics for the run, showing the packets and bits sent and received by each network interface in the system. Typically only one or two interfaces will be relevant to the test that was run.

The final section reports the Run statistics which is a high level summary of the completed test, reporting the total test duration, data transferred, operations and errors across the hostname or IP of all the members involved in the test. The data in this section will be a primary source of the results used to conduct our comparisons.

For the measurement results used in this paper, the workload test throughputs and transaction times were extracted from the uperf output report.

1 Private network: Wikipedia