The common standard benchmarksBy: Bill Buros.
After we have a new Power7 system installed and configured, we generally will run a set of performance benchmarks on the system to confirm that it's running things as we expect it to.
In a performance lab like ours, we have the luxury of access to SPEC.org benchmarks, the latest IBM XL C/C++ and Fortran Compilers, the latest gcc-based Advance Toolchain libraries, and the full suite of performance tools and profilers installed on our systems.
On a new system, we will regularly install, build, and run the following "standard suite" of benchmarks.
- Stream - measures memory throughput - takes advantage of OpenMP and 16MB large pages - gcc used
- Linpack - measures floating point performance - takes advantage of IBM's ESSL math libraries - gcc often used with IBM XL run-time libraries
- SPECcpu2006 - a broad suite of C, C++, and Fortran applications - we work with both IBM XL compilers and GCC compilers
- SPECjbb2005 - checks the Java performance on a server - we work with the various versions of IBM Java
These four benchmarks are classic compute-intensive workloads which can be run on a single server. Linpack can be extended to run on clusters, and is the benchmark which underlies the Top500 list of supercomputers.
For us, all four benchmarks are intended to run on all of the processor cores and processor hardware threads. We use all cores and hardware threads to focus on the overall capabilities of each system.
Since we take advantage of all of the processor cores and hardware threads, we use the SPECcpu2006 benchmark in a particular way - focusing on what the SPEC.org committee calls the "rate" runs. The SPEC team has a description of this on their website. To save you some time, I copied some of the applicable text here.
There are several different ways to measure computer performance. One way is to measure how fast the computer completes a single task; this is a speed measure. Another way is to measure how many tasks a computer can accomplish in a certain amount of time; this is called a throughput, capacity or rate measure.
- The SPECspeed metrics (e.g., SPECint2006) are used for comparing the ability of a computer to complete single tasks.
- The SPECrate metrics (e.g., SPECint_rate2006) measure the throughput or rate of a machine carrying out a number of tasks. For the SPECrate metrics, multiple copies of the benchmarks are run simultaneously.