Java Web services
Metro vs. Axis2 performance
See how Metro stacks up against Axis2 on performance, both with and without WS-Security
This content is part # of # in the series: Java Web services
This content is part of the series:Java Web services
Stay tuned for additional content in this series.
The Metro Web services stack is based on the reference implementations of the JAXB 2.x data-binding and JAX-WS 2.x Web services standards, but it uses additional components to provide features beyond the basic support defined by JAX-WS. WS-Security and other SOAP extension technologies are implemented by the Web Services Interoperability Technologies (WSIT) project, with the actual WS-Security processing implemented by another added component: the XML and WebServices Security Project (XWSS).
Axis2 is based on completely different technologies, including the default Axis2 Data Binding (ADB) data-binding implementation, the Axis2 engine itself, and the Rampart module combined with the Web Services Security for Java (WSS4J) for WS-Security support. An earlier article in this series, "The high cost of (WS-)Security," showed the impact of WS-Security processing on performance in the Axis2 Web services stack.
"Introducing Metro" and "WS-Security with Metro" showed you how the two stacks differ in terms of installation, configuration, and actual usage. This article looks at the performance differences between the two, including the differences when WS-Security is used.
Looking at performance
Just like "The high cost of (WS-)Security," this article takes the approach of measuring the time required to execute a particular sequence of requests when both the client and the server are running on a single system. This approach does a great job of comparing Web services processing overhead, since the impact of network latencies and overhead are eliminated from timing results. Assuming the client code isn't significantly slower than the server, the figures are also good representations of actual server performance under load.
This article uses the same test application as the earlier article, a seismic data retrieval service. The service uses an actual database of more than 93,000 earthquakes that occurred worldwide over a period of years. Requests to the service specify a time range and a geographic coordinate range, and the service returns all earthquakes within the specified range. See "The high cost of (WS-)Security " for full details and a sample request-response message pair.
As in the previous article, two sets of request sequences were used for the performance tests. The first set used 1,000 requests, with query parameters adjusted to match a small portion of the entire earthquake database (returning only 816 matching earthquakes for the 1,000 requests). The second set used 100 requests, adjusted to match a larger portion of the database (returning 176,745 matching earthquakes for the 100 requests). Each request sequence was run multiple times in different security configurations, with only the best time for each configuration kept in the results.
The tests were run on a Mandriva 2009.1 64-bit Linux system with an Athlon X2 5400+ processor and 4 GB of RAM, using a Sun Java 1.6.0_13 32-bit JVM (which gave slightly better performance than the 64-bit JVM for a given heap size). The server code was run on Tomcat 6.0.20, configured to use 1024 MB of heap, with the client code using 512 MB of heap. The Web service stack versions were Metro 1.5 (which includes WSIT and XWSS) and Axis2 1.5.1 with a current build of the Rampart code (since there's still no release of Rampart matching the Axis2 1.5.x code). If you want to try the tests on your own hardware and JVM, download the code.
The earlier article looked only at Axis2 performance, and included plain text, SSL, and various WS-Security/WS-SecureConversation configurations. This one uses a more-limited set of configurations but directly compares the Axis2 and Metro performance for each configuration.
Performance without WS-Security
Figure 1 shows the measured test times for both Axis2 and Metro without any WS-Security usage. The chart shows that there's only a small difference between the two stacks. In the test with 1,000 requests and small responses, Metro ran half a second faster than Axis2. In the test with only 100 requests but larger responses, the two stacks ran equally fast (to within .1 seconds).
Figure 1. Test times without security
These timing results show that (for the data used by the test application) Metro is probably slightly faster than Axis2 in per-request processing, but in actual data conversions the two run neck-and-neck (when using the default ADB data binding with Axis2 — other data bindings may give different results, and XMLBeans binding, in particular, is known to be considerably slower).
Performance with WS-Security
The next two figures show the relative test times for the following security configurations:
- plain— No security (the same values as in Figure 1, but normalized to the Axis2 time)
- username— WS-Security plain-text
- sign— WS-Security signing of body and headers, with timestamp
- signencr— WS-Security signing of body and headers, with timestamp, and encryption of body
Figure 2. Small response test
Figure 3 shows the times for 100 requests with large responses:
Figure 3. Large response test
Metro is consistently much faster than Axis2 for the WS-Security configurations — more than twice as fast overall, and more than three times as fast in the case of the username and sign configurations for large messages. This is a surprising result for two Web services stacks at about the same level of maturity.
Rampart has rudimentary time logging built-in using the
org.apache.rampart.TIME logger. By enabling this logger at
DEBUG level, you can find out the amount of clock time required for various parts of the Rampart processing. Strangely, when I did this for the signatures example, I found that the Rampart processing times made up less than half of the total time taken for the test. This implies that the main performance problems with Axis2/Rampart WS-Security handling lie outside Rampart and the underlying WSS4J security implementation.
Rampart certainly has a lot of room for improvement. As mentioned in "The high cost of (WS-)Security," Rampart does build a complete in-memory model of the message any time WS-Security processing is involved. The overhead from building the in-memory model is the apparent cause of Axis2/Rampart's poor performance in the
UsernameToken case. It's possible that the poor performance by Axis2/Rampart in other WS-Security scenarios also relates to this same type of conversion issue.
Wrapping up Metro
Metro can be awkward to configure for standalone use, especially given the limited documentation available. Metro also only works with JAXB 2.x data binding and JAX-WS 2.x Web services configuration, as opposed to the wider range of data bindings and alternative configuration supported by Axis2. But Metro offers performance equal to Axis2 for plain-text message exchanges and much better performance than Axis2 when WS-Security is involved. If you're using WS-Security and concerned about performance, you should definitely look into using Metro for your application.
The next article moves on to looking at the CXF Web services stack, another Apache Foundation project. CXF uses some of the same underlying components as Axis2 but a very different style for configuring and deploying Web services. You'll see the basics of CXF usage and how it differs from Axis2 and Metro then.