By: Bill Buros.
Continuing our journey into benchmarks, we recently were working on the SPECcpu2006 set of benchmarks again, and returned to look at some of the dependent products which we build and run with.
One of the products which we link with on Power Linux is MicroQuil's Smartheap library. This is a run-time library which can be purchased from MicroQuil. One of the benchmark components of the SPECcpu2006 suite - xalancbmk - is a classic example of a questionable choice as a system and compiler measurement benchmark. The benchmark - across hardware architectures - uses malloc() calls in an unusual fashion with significant malloc call overhead. Many companies use Smartheap in SPECcpu2006 submissions to dramatically improve the component's performance. I'm not a big fan of these special case implementations, but they are often a reality in the world of comparative benchmarks.
Recently we repeated some SPECcpu2006 comparisons on a Power 730 system running Linux, comparing previously published results with essentially the same runs but without Smartheap linked in. As expected, xalancbmk dropped about a third in performance without the Smartheap specialized malloc implementation.
Specifically, if you look at the SPECint_rate2006 run on the Power 730 done a while ago,
Look for the xalancbmk result. You'll see that the published peak result was 610. In our engineering runs without Smartheap, that result (in a single run - so not publish'able and not SPEC reviewed), was around 416. An impressive drop! or vice-versa, quite the impressive gain for the benchmark component when you use Smartheap.
- For what it's worth, the nice thing about SPECcpu2006 benchmarks is all of the compile, build, and link flags and settings have to be specified in the disclosure, so you can search for which components linked with Smartheap in our runs. The other listed components got minor improvements with Smartheap, nothing really to write home about.
We'd love to see alternative malloc's get more traction in the Linux community. We've seen some promising implementations in the Google examples and one or two others. Perhaps we'll see some tunable malloc implementations emerge in the GCC realm or the Toolchain worlds. The notion is you may have a unique application with malloc behaviors like xalancbmk, in which case you should obviously look at alternative malloc implementations.
Anyway, this is simply an example of one of the benchmarks that can be considered when you are looking at servers, operating systems, and compilers. We recommend that you get familiar with what the benchmark is really measuring, and the pros and cons of each benchmark, so that you understand best what you are measuring. Better yet, if you have questions, ask here on Think Power Linux, we'd love to hunt down answers for you.
(ps.. thanks to Rafael for getting the results being shared here)