Java web services: CXF performance comparison

Find out how CXF performance compares with the latest versions of Axis2 and Metro

Apache CXF shares certain underlying components with both Apache Axis2 and Metro but combines the components within an entirely different architecture. Dennis Sosnoski continues his Java web services column series by comparing how the CXF, Metro, and Axis2 stacks perform both with and without WS-Security.

Share:

Dennis Sosnoski, Architecture Consultant and Trainer, Sosnoski Software Solutions, Inc.

Author photoDennis Sosnoski is a consultant and trainer specializing in Java-based XML and Web services. His professional software development experience spans more than 30 years, with the last 10 focused on server-side XML and Java technologies. Dennis is the lead developer of the open source JiBX XML Data Binding framework and the associated JiBX/WS Web services framework, as well as a committer on the Apache Axis2 Web services framework. He was also one of the Expert Group members for the JAX-WS 2.0 and JAXB 2.0 specifications. The material for the Java Web Services series is based on Dennis' training classes.



27 April 2010

Also available in Chinese Japanese

The Apache CXF web services stack builds on some of the same technologies as the Apache Axis2 and Metro stacks discussed in earlier articles of this series. Like Axis2, it uses the Apache WSS4J WS-Security implementation. Like Metro, it predominantly uses JAX-WS 2.x web services configuration and JAXB 2.x data binding (even using the same reference implementation of JAXB as Metro, though versions may differ between the two stacks). Beyond these common components, however, the stacks differ in many ways, including their processing engines and WS-SecurityPolicy configuration handling.

Past articles in this series have compared Axis2's performance with Metro's, both for simple message exchanges and with different WS-Security configurations. In this article, you'll see how CXF performance compares with the most-recent releases of both Axis2 and Metro.

About this series

Web services are a crucial part of Java technology's role in enterprise computing. In this series of articles, XML and web services consultant Dennis Sosnoski covers the major frameworks and technologies that are important to Java developers using web services. Follow the series to stay informed of the latest developments in the field and aware of how you can use them to aid your programming projects.

Looking at performance

Just like the earlier articles on web services performance — "The high cost of (WS-)Security " and "Metro vs. Axis2 performance" — this article takes the approach of measuring the time required to execute a particular sequence of requests when both the client and the server run on a single system. This approach does a great job of comparing web services processing overhead, because it eliminates the impact of network latencies and overhead from timing results. Assuming the client code isn't significantly slower than the server, the figures are also good representatations of actual server performance under load.

This article also uses the same test application as the earlier articles, a seismic data retrieval service. The service uses a database of more than 93,000 earthquakes that occurred worldwide over a period of years. Requests to the service specify a time range and a geographic coordinate range, and the service returns all earthquakes within the specified range. See "The high cost of (WS-)Security" for full details of the test application and a sample request/response message pair.

As in the previous articles, two sets of request sequences are used for the performance tests. The first set uses 1,000 requests with query parameters adjusted to match a small portion of the entire earthquake database (returning only 816 matching earthquakes for the 1,000 requests). The second set uses 100 requests adjusted to match a larger portion of the database (returning 176,745 matching earthquakes for the 100 requests). These two request sequences emphasize different performance characteristics of web services stacks. The first one shows how quickly stacks process requests with little data, and the second emphasizes the speed of processing data volumes. Each request sequence was run multiple times in different security configurations with only the best time for each configuration kept in the results.

The tests were run on a Mandriva 2009.1 64-bit Linux system with an Athlon X2 5400+ processor and 4GB of RAM, using a Sun (Oracle) Java 1.6.0_18 32-bit JVM (which gave slightly better performance than the 64-bit JVM for a given heap size). The server code was run on Tomcat 6.0.20, configured to use 1024MB of heap, with the client code using 512MB of heap. The web service stack versions were:

  • CXF 2.1.7
  • Metro 2.0
  • Axis2 1.5.1 with the 1.5 release of Rampart

If you want to try the tests on your own hardware and JVM, download the code.


Performance without WS-Security

Figure 1 shows the measured test times for Axis2, Metro, and CXF without any WS-Security usage. As you can see from the chart, Metro is significantly faster than Axis2 and CXF (about 25 percent faster) for the case of many requests with small responses, and about the same speed as the Apache stacks for the lower number of requests with larger responses. (In all of this article's charts, shorter bars are better because they indicate faster times.)

Figure 1. Test times without security
Bar graph illustration of test times without security

These results show some interesting differences from the earlier comparison between Metro 1.5 and Axis2 1.5.1. The timings suggest that, at least for the data used by the test application, Metro 2.0 is faster than Axis2 and CXF in per-request processing. In data conversions to and from XML, the three stacks all run at about the same speed. This is to be expected in the case of Metro and CXF, since they both use the JAXB reference implementation for conversions. Judging from these results, the Axis2 Databinding Framework (ADB) binding implementation used by default with Axis2 runs about as fast as JAXB.


Performance with WS-Security

The next figures show test times for the following security configurations:

  • plain: No security (the same values as in Figure 1)
  • username: WS-Security plain-text UsernameToken on requests
  • sign: WS-Security signing of body and headers, with timestamp
  • signencr: WS-Security signing of body and headers, with timestamp, and encryption of body

Figure 2 shows the measured test times for 1,000 requests with small responses:

Figure 2. Small-response measured times
Bar graph illustration of small response measured times

Figure 3 shows the relative times (normalized to the CXF results) for the same 1,000 requests with small responses:

Figure 3. Small-response normalized times
Bar graph illustration of small response normalized times

Axis2 is consistently the slowest of the stacks across all security configurations for this test case. Metro is consistently the fastest, though the difference between Metro and CXF is much smaller than the difference between CXF and Axis2: Metro is about 10 percent faster than CXF across the different security configurations, whereas Axis2 is more than twice as slow as CXF.

Figure 4 shows the measured test times for 100 requests with larger responses:

Figure 4. Large-response measured times
Bar graph illustration of large response measured times

Figure 5 shows the relative times (normalized to the CXF results) for the same 100 requests with larger responses:

Figure 5. Large-response normalized times
Bar graph illustration of large response normalized times

For this second test case, Axis2 is again much slower than Metro and CXF (again about half the speed of CXF), and the difference between Metro and CXF in the smaller-response messages is more than reversed, with CXF about 15 percent faster overall.

CXF 2.1.7 vs. 2.1.6

These test times reflect a noticeable performance improvement for CXF between versions 2.1.6 and 2.1.7. Code tuning is partly responsible for this, but a more significant portion of the improvement is due to the fix for a problem discussed in "WS-Security with CXF." CXF 2.1.6 did not process a UsernameToken WS-SecurityPolicy configuration unless a <sp:TransportBinding> policy or some form of encryption or signing policy was present. Normally such an added policy would be present when UsernameToken is used — especially a plain-text UsernameToken because without either transport- or message-level encryption the username and password can be observed in transit. But it's perfectly valid from a policy standpoint to use UsernameToken on its own (as in this article's username configuration), and the fix implemented for CXF 2.1.7 handles this case. As part of the fix, CXF 2.1.7 skips adding WS-Security handling to the response message flow in this particular case.

Removal of WS-Security from the message flow in the username configuration improved CXF's results by several percentage points overall compared to the same tests run with CXF 2.1.6. That improvement between versions is, unfortunately, somewhat artificial. If the usernaeme test case used either transport- or message-level encryption, WS-Security handling would be present in the response message flow and cause the CXF timing results for that configuration to be noticeably slower. Hopefully, future CXF versions will extend the policy analysis so that WS-Security is configured in either request or response message flows only when it is needed, thereby extending the performance benefit to a wider range of use cases (including those in which signing or encryption is needed in only one direction).


Conclusions

Based on the test timings reported in this article, it looks like Metro 2.0 is faster in basic request/response processing than either Axis2 1.5.1 or CXF 2.1.7. When it comes to WS-Security processing, though, Metro's XWSS library is no faster overall than the WSS4J library used by both Axis2 and CXF.

Probably the most interesting aspect of these results is the implication for Axis2. Earlier tests showed that Axis2 was much slower than Metro in WS-Security processing. These test results show that the difference is not due to the WS-Security implementation code, so it must be due to the way Axis2 (and the Rampart security module) handle messages being passed to and from WSS4J. This confirms the results discussed in "Metro vs. Axis2 performance." Axis2 also requires significantly more memory to run the tests than the other stacks, most noticably for the signenecr configuration (Axis2 needed about 80MB of memory on the client; CXF ran fine with 48MB). These issues reinforce the earlier impression that Axis2 wastes processing time and memory doing unnecessary conversions between different message representations as part of the WS-Security handling.

The performance overhead of using WS-Security signing and encryption is substantial for all the stacks tested. This overhead can be problematic for services with heavy usage. In the next article, you'll learn about the WS-Trust and WS-SecureConversation add-on technologies, which can reduce the WS-Security overhead when clients make extensive use of a particular service.


Download

DescriptionNameSize
Source code for this articlej-jws14-src.zip4.86MB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Java technology on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java technology, SOA and web services, Open source
ArticleID=484864
ArticleTitle=Java web services: CXF performance comparison
publish-date=04272010