The press release is IBM Software Smashes Java Performance Record, which discusses the SPECjAppServer2004 Results, which include the Fourth Quarter 2005 SPECjAppServer2004 Results.
In the results, the best time came from WAS 6.0 ND running on a IBM eServer p5 550 with SUSE LINUX Enterprise Server 9 and DB2 Universal Database 8.2 (details). This configuration clocked in at 2921.48 JOPS (jAppServer Operations Per Second). By comparison, the other configuration benchmarked this quarter was BEA WebLogic Server 9.0 running on a Sun Fire X4100 Cluster, which produced 1781.47 JOPS. (2921.48 - 1781.47) / 1781.47 = 63.99%, so WAS did 64% better.
So what is SPECjAppServer2004? From SPEC's home page:
The Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops benchmark suites and also reviews and publishes submitted results from our member organizations and other benchmark licensees.
Members include IBM, BEA Systems, Sun Microsystems, Oracle, Hewlett-Packard, JBoss, and Microsoft. You can learn more about the benchmark itself from the SPECjAppServer2004 User's Guide.
Dec 16, 2005 Update: IBM, at BEA's request, has asked me to modify this blog posting from its original content to no longer discuss the BEA press release. The use of and comparison of the CPUs/1000JOPS was a violation of the SPEC organization's SPECjAppServer2004 benchmark rules as outlined in SPECjAppServer2004 Run and Reporting Rules V1.02. IBM wishes to emphasize their desire to support the rules of the benchmarking organizations it belongs to. Although I am hesitant to make major changes to historical blog postings, it is important that IBM cooperate with other industry organizations and it is always my intention to support that; this includes remedying statements that are later found to be inconsistent with agreed upon rules. Accordingly, as per my employer's request, I have struck out the paragraphs in question.
Incredibly, the BEA table actually shows that their results are getting worse!?! Whereas they achieved 7.21 CPUs per 1,000 JOPS (CPU/KJ) back in the Aug 05 benchmark, they've now fallen to 11.23 CPU/KJ in Oct 05. They didn't include the latest WAS benchmark in their table, but the figure is (32 / 2921.48) * 1000 = 10.95 CPU/KJ. So WAS in Oct 05 actually did better than WLS in Oct 05, though neither did as well as WLS in Aug or Sept. At least WAS is better now than the figures listed for Jan 05 and Dec 04 (11.99 and 14.89, according to BEA).
So BEA isn't really bragging that they're more efficient now, but that they were two months ago; except their efficiency is getting worse whereas WAS's is getting better. Is this really what BEA is saying?! Gotta love statistics!
For more commentary, see:
- IBM's WebSphere Application Server Breaks Java Performance Record (Sys-Con) (NEW)
- WebSphere vs WebLogic: IBM and BEA Spar Over SPEC Results (Sys-Con)
- IBM posts new SPECjAppServer2004 score (TSS)