Java web services
CXF performance comparison
Find out how CXF performance compares with the latest versions of Axis2 and Metro
This content is part # of # in the series: Java web services
This content is part of the series:Java web services
Stay tuned for additional content in this series.
The Apache CXF web services stack builds on some of the same technologies as the Apache Axis2 and Metro stacks discussed in earlier articles of this series. Like Axis2, it uses the Apache WSS4J WS-Security implementation. Like Metro, it predominantly uses JAX-WS 2.x web services configuration and JAXB 2.x data binding (even using the same reference implementation of JAXB as Metro, though versions may differ between the two stacks). Beyond these common components, however, the stacks differ in many ways, including their processing engines and WS-SecurityPolicy configuration handling.
Past articles in this series have compared Axis2's performance with Metro's, both for simple message exchanges and with different WS-Security configurations. In this article, you'll see how CXF performance compares with the most-recent releases of both Axis2 and Metro.
Looking at performance
Just like the earlier articles on web services performance — "The high cost of (WS-)Security " and "Metro vs. Axis2 performance" — this article takes the approach of measuring the time required to execute a particular sequence of requests when both the client and the server run on a single system. This approach does a great job of comparing web services processing overhead, because it eliminates the impact of network latencies and overhead from timing results. Assuming the client code isn't significantly slower than the server, the figures are also good representatations of actual server performance under load.
This article also uses the same test application as the earlier articles, a seismic data retrieval service. The service uses a database of more than 93,000 earthquakes that occurred worldwide over a period of years. Requests to the service specify a time range and a geographic coordinate range, and the service returns all earthquakes within the specified range. See "The high cost of (WS-)Security" for full details of the test application and a sample request/response message pair.
As in the previous articles, two sets of request sequences are used for the performance tests. The first set uses 1,000 requests with query parameters adjusted to match a small portion of the entire earthquake database (returning only 816 matching earthquakes for the 1,000 requests). The second set uses 100 requests adjusted to match a larger portion of the database (returning 176,745 matching earthquakes for the 100 requests). These two request sequences emphasize different performance characteristics of web services stacks. The first one shows how quickly stacks process requests with little data, and the second emphasizes the speed of processing data volumes. Each request sequence was run multiple times in different security configurations with only the best time for each configuration kept in the results.
The tests were run on a Mandriva 2009.1 64-bit Linux system with an Athlon X2 5400+ processor and 4GB of RAM, using a Sun (Oracle) Java 1.6.0_18 32-bit JVM (which gave slightly better performance than the 64-bit JVM for a given heap size). The server code was run on Tomcat 6.0.20, configured to use 1024MB of heap, with the client code using 512MB of heap. The web service stack versions were:
- CXF 2.1.7
- Metro 2.0
- Axis2 1.5.1 with the 1.5 release of Rampart
If you want to try the tests on your own hardware and JVM, download the code.
Performance without WS-Security
Figure 1 shows the measured test times for Axis2, Metro, and CXF without any WS-Security usage. As you can see from the chart, Metro is significantly faster than Axis2 and CXF (about 25 percent faster) for the case of many requests with small responses, and about the same speed as the Apache stacks for the lower number of requests with larger responses. (In all of this article's charts, shorter bars are better because they indicate faster times.)
Figure 1. Test times without security
These results show some interesting differences from the earlier comparison between Metro 1.5 and Axis2 1.5.1. The timings suggest that, at least for the data used by the test application, Metro 2.0 is faster than Axis2 and CXF in per-request processing. In data conversions to and from XML, the three stacks all run at about the same speed. This is to be expected in the case of Metro and CXF, since they both use the JAXB reference implementation for conversions. Judging from these results, the Axis2 Databinding Framework (ADB) binding implementation used by default with Axis2 runs about as fast as JAXB.
Performance with WS-Security
The next figures show test times for the following security configurations:
- plain: No security (the same values as in Figure 1)
- username: WS-Security plain-text
- sign: WS-Security signing of body and headers, with timestamp
- signencr: WS-Security signing of body and headers, with timestamp, and encryption of body
Figure 2 shows the measured test times for 1,000 requests with small responses:
Figure 2. Small-response measured times
Figure 3 shows the relative times (normalized to the CXF results) for the same 1,000 requests with small responses:
Figure 3. Small-response normalized times
Axis2 is consistently the slowest of the stacks across all security configurations for this test case. Metro is consistently the fastest, though the difference between Metro and CXF is much smaller than the difference between CXF and Axis2: Metro is about 10 percent faster than CXF across the different security configurations, whereas Axis2 is more than twice as slow as CXF.
Figure 4 shows the measured test times for 100 requests with larger responses:
Figure 4. Large-response measured times
Figure 5 shows the relative times (normalized to the CXF results) for the same 100 requests with larger responses:
Figure 5. Large-response normalized times
For this second test case, Axis2 is again much slower than Metro and CXF (again about half the speed of CXF), and the difference between Metro and CXF in the smaller-response messages is more than reversed, with CXF about 15 percent faster overall.
CXF 2.1.7 vs. 2.1.6
These test times reflect a noticeable performance improvement for CXF between versions
2.1.6 and 2.1.7. Code tuning is partly responsible for this, but a more significant
portion of the improvement is due to the fix for a problem discussed in "WS-Security
with CXF." CXF 2.1.6 did not process a
UsernameToken WS-SecurityPolicy configuration unless a
<sp:TransportBinding> policy or some form of encryption or signing
policy was present. Normally such an added policy would be present when
UsernameToken is used — especially a plain-text
UsernameToken because without either transport- or message-level encryption the username and password can be observed in transit. But it's perfectly valid from a policy standpoint to use
UsernameToken on its own (as in this article's username configuration), and the fix implemented for CXF 2.1.7 handles this case. As part of the fix, CXF 2.1.7 skips adding WS-Security handling to the response message flow in this particular case.
Removal of WS-Security from the message flow in the username configuration improved CXF's results by several percentage points overall compared to the same tests run with CXF 2.1.6. That improvement between versions is, unfortunately, somewhat artificial. If the usernaeme test case used either transport- or message-level encryption, WS-Security handling would be present in the response message flow and cause the CXF timing results for that configuration to be noticeably slower. Hopefully, future CXF versions will extend the policy analysis so that WS-Security is configured in either request or response message flows only when it is needed, thereby extending the performance benefit to a wider range of use cases (including those in which signing or encryption is needed in only one direction).
Based on the test timings reported in this article, it looks like Metro 2.0 is faster in basic request/response processing than either Axis2 1.5.1 or CXF 2.1.7. When it comes to WS-Security processing, though, Metro's XWSS library is no faster overall than the WSS4J library used by both Axis2 and CXF.
Probably the most interesting aspect of these results is the implication for Axis2. Earlier tests showed that Axis2 was much slower than Metro in WS-Security processing. These test results show that the difference is not due to the WS-Security implementation code, so it must be due to the way Axis2 (and the Rampart security module) handle messages being passed to and from WSS4J. This confirms the results discussed in "Metro vs. Axis2 performance." Axis2 also requires significantly more memory to run the tests than the other stacks, most noticably for the signenecr configuration (Axis2 needed about 80MB of memory on the client; CXF ran fine with 48MB). These issues reinforce the earlier impression that Axis2 wastes processing time and memory doing unnecessary conversions between different message representations as part of the WS-Security handling.
The performance overhead of using WS-Security signing and encryption is substantial for all the stacks tested. This overhead can be problematic for services with heavy usage. In the next article, you'll learn about the WS-Trust and WS-SecureConversation add-on technologies, which can reduce the WS-Security overhead when clients make extensive use of a particular service.
- CXF: CXF is an open source web services stack from the Apache Software Foundation.
- Axis2: Axis 2 is another open source web services stack from Apache.
- WSS4J: WSS4J is the open source implementation of WS-Security from the Apache Software Foundation, used by both Axis2 and CXF for WS-Security processing.
- Understanding Web Services specifications: This series of tutorials introduces many of the important Web services standards, including:
- "Understanding Web Services specifications: Web Services Description Language (WSDL)" (Nicholas Chase, developerWorks, July 2006)
- "Understanding Web Services specifications: WS-Security" (Nicholas Chase, developerWorks, August 2006)
- "Understanding Web Services specifications: WS-Policy" (Tyler Anderson, developerWorks, February 2007).
- OASIS Web Services Security (WSS) TC: This is the organization responsible for the WS-Security specification and token profiles. You can find links here to all versions of these standards.
- The W3C Web Services Policy Working Group: This group defines the WS-Policy specification.
- OASIS Web Services Secure Exchange (WS-SX) TC: This organization is responsible for WS-SecurityPolicy and related specifications.