A performance benchmark method for comparing open source Java application servers

It's easy and free

More and more companies are changing their IT environments from expensive commercial software to free, open source software to cut entry cost and survive in today's difficult economy. An open source Java application server, as the core of a business application, plays an important role in this trend.

Companies can spend a lot of time and effort when choosing a Java application server that advances performance, is easy to administer, and is cost-effective. There are many choices in the open source Java application server arena. Geronimo, GlassFish, JBoss, JonAS, IBM® WebSphere® Application Server Community Edition, Apache Tomcat, and Jetty are a few examples. One reason it is so difficult to make the right decision about a suitable solution is the lack of performance benchmark data for comparison.

There is an industry-standard, SPECjAppServer2004 (Java Application Server), which is a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) technology-based application servers. SPECjAppServer2004 is an end-to-end application that exercises all major J2EE technologies implemented by compliant application servers. Some commercial application server vendors publish their performance benchmark data periodically, such as BEA WebLogic and IBM WebSphere Application Server. However, there is no data for open source Java application servers. Why? Perhaps because it's not free.

In this article, learn about an easy open source method for doing performance benchmark testing.

Open source method for a performance benchmark

Open source Java application servers use open source performance benchmark methods. You can use a straightforward, easy method for Java application servers performance benchmark testing with the following steps:

  1. Determine the requirements of the performance benchmark.
  2. Determine the tools needed to measure performance.
  3. Prepare for performance testing, including the test environment construction and a smoke test.
  4. Run tests and record raw data.
  5. Visually display the results for report and analysis.

The steps are not linear. You might need to repeat them several times during performance benchmark testing. The rest of this article walks you though the steps.

Determine the requirements

First, you need to determine the requirements for your performance benchmark. You must have some performance indicators to measure. Some general questions can yield the performance indicators. For example:

  • How much throughput does the application server have?
  • What is the average response time for application server response to client requests?
  • How much time does it take to boot the application server?
  • How much memory is required to run the application server?

From these questions, you can create a matrix of application server performance indicators and use it to record the performance benchmark results.

You must also consider several things about the test environment. For example:

  • What is the typical hardware configuration for running the application server? Is it PC? Low-end server? What about the CPU and memory configuration?
  • What is the typical software configuration? What version of JRE is used? What JVM heap size is allocated? What is the configuration for the database software?
  • Does it require an isolated network environment?

Considering these questions will help you form a realistic test topology for the performance benchmark.

Determine the tools

To do a performance benchmark of a Java application server, you need a set of well-designed sample Java EE programs. By using the programs and recording data, you can get quantified results for the performance indicators. A benchmark is about comparison, so the set of sample Java EE programs should be able to run on your selected different application servers. It's hard to start this from scratch. Fortunately, in the open source community there is a sample application: Apache DayTrader.

Sample application

DayTrader, an end-to-end Web application, is at the core of our example method. DayTrader is a benchmark application modeled after an online stock brokerage. The application lets users log in, view their portfolios, look up stock quotes, and buy or sell stock shares. With the aid of a Web-based load driver, the real-world workload provided by DayTrader can be used to measure and compare the performance of Java EE application servers offered by a variety of vendors. The application also contains a set of primitives used for functional and performance testing of various Java EE components and common design patterns.

Figure 1. DayTrader interface
 Figure 1 shows the DayTrader login screen
Figure 1 shows the DayTrader login screen

DayTrader leverages J2EE components, such as servlets, JSP files, enterprise beans, message-driven beans (MDBs), and Java database connectivity (JDBC) to provide a set of user services. Services include login/logout, stock quotes, buy, sell, account details, etc. using standards-based HTTP and Web services protocols. (For more details about DayTrader, check out the FAQ tab in the application.)

As of the writing of this article, DayTrader was updated to be compatible with Java EE 5 features, such as EJB V3.0 with Java Persistence API. You can use it freely to measure the following aspects of a Java EE 5-compliant application server:

  • JSP/Servlet container
  • EJB container
  • DB connectivity
  • JPA capability
  • JTA capability

You can also use DayTrader for non-Java EE 5-compliant application servers, such as Tomcat.

Assume you have a Java EE 5-compliant application server composed of two major containers: a Web container and an EJB container. Java EE artifacts, such as JSP and Servlet, are running in the Web container. Enterprise JavaBeans (EJBs) are running in the EJB container. These artifacts are the major components of Java Web applications.

DayTrader provides a set of workload primitives to measure Java application server capability. Each primitive has its specific target on a Java EE programming model or artifact. They are divided into the two categories: Web container primitives and EJB container primitives.

Table 1 describes the Web container ping suite.

Table 1. Web container primitives
PingHtmlThe most basic operation providing access to a simple "Hello World" page of static HTML.
Explicit GCInvokes garbage collection on AppServer. Reports heap statistics after the GC has completed.
PingServletTests fundamental dynamic HTML creation through server-side servlet processing.
PingServletWriterExtends PingServlet by using a PrintWriter for formatted output vs. the output stream used by PingServlet.
PingServlet2Include *Tests response inclusion. Servlet 1 includes the response of Servlet 2.
PingServlet2ServletTests request dispatching. Servlet 1, the controller, creates a new JavaBean object and forwards the request with the JavaBean added to Servlet 2. Servlet 2 obtains access to the JavaBean through the Servlet request object and provides dynamic HTML output based on the JavaBean data.
PingJSPTests a direct call to the JSP providing server-side dynamic HTML through JSP scripting.
PingJSPELTests a direct call to a JSP providing server-side dynamic HTML through JSP scripting and usage of the new JSP V2.0 Expression Language.
PingServlet2JSPTests a commonly used design pattern, where a request is issued to servlet providing server-side control processing. The servlet creates a JavaBean object with dynamically set attributes and forwards the bean to the JSP through a RequestDispatcher. The JSP obtains access to the JavaBean and provides formatted display with dynamic HTML output based on the JavaBean data.
PingHTTPSession1Tests fundamental HTTP session function by creating a unique session ID for each individual user. The ID is stored in the user's session and is accessed and displayed on each user request.
PingHTTPSession2Further extends the previous test by invalidating the HTTP session on every fifth user access. This results in testing HTTPSession create and destroy.

* The primitive can be adjusted by a load repetition parameter. The DayTrader application has more detail.

Table 2 describes the EJB 3 container ping suite.

Table 2. EJB V3 container ping suite
PingServlet2Session *Tests key function of a servlet call to a stateless Session EJB. The Session EJB performs a simple calculation and returns the result.
PingServlet2Entity *Tests key function of a servlet call to an EJB V3.0 container managed entity. In this test, the EJB entity represents a single row in the database table.
PingServlet2Session2Entity *Tests the full servlet to Session EJB to Entity EJB path to retrieve a single row from the database.
PingServlet2Session2EntityCollection *Test extends the previous EJB Entity test by calling a Session EJB that uses a finder method on the Entity that returns a collection of Entity objects. Each object is displayed by the servlet.
PingServlet2Session2CMROne2One *Test that drives an Entity EJB to get another Entity EJB's data through an EJB V3.0 CMR one to one relationship.
PingServlet2Session2CMROne2Many *Test that drives an Entity EJB to get another Entity EJB's data through an EJB V3.0 CMR one to many relationship.
PingServlet2Session2JDBC *Tests the full servlet to Session EJB to JDBC path to retrieve a single row from the database.
PingServlet2Session2JDBCCollection *Test that extends the previous JDBC test by calling a Session EJB to JDBC path which returns multiple rows from the database.
PingServlet2MDBQueue *Drives messages to a queue based message-driven EJB (MDB). Each request to the servlet posts a message to the queue. The MDB receives the message asynchronously and prints message delivery statistics on each 100th message.

Note: Not intended for performance testing.

PingServlet2MDBTopic *Drives messages to a topic-based publish/subscribe MDB. Each request to the servlet posts a message to the Topic. The TradeStreamMDB receives the message asynchronously and prints message delivery statistics on each 100th message. Other subscribers to the topic will also receive the messages.

Note: Not intended for performance testing.

PingServlet2TwoPhase *Drives a Session EJB that invokes an Entity EJB with findByPrimaryKey (DB access), followed by posting a message to an MDB through a JMS queue (message access). These operations are wrapped in a global two-phase transaction and commit.

* The primitive can be adjusted by a load repetition parameter. The DayTrader application has more detail.

In addition to these workload primitives, DayTrader also provides a complex trade scenario to measure overall performance of a Java EE application server. The scenario includes a typical Web application model, from Web container to EJB container to DB connectivity. Due to differences in database connections, the complex trade scenario is differentiated in two ways:

  • Trade scenario – JDBC Direct
  • Trade scenario – Full EJB

A link to Test DayTrader Scenario appears in your browser. Click Reload to manually step through a DayTrader scenario.

You can easily access some or all of the primitives, and complex scenarios, to do your performance benchmark on Java application servers.

Load-generation tool

To test performance, you will need a load-generation tool. Fortunately, there are several choices of free load-generation tools, such as Apache JMeter, Grinder, and OpenSTA. The example in this article uses Apache JMeter. You can choose whatever tool you're comfortable with, as long as it meets your requirements.

Build a test environment

At this point, you apply your plan on paper to real running systems. For the following method, it is suggested that you have at least three machines: one for the application server (or system under test (SUT)), one for a load-generation tool, and one for a database server. Testing for the example used three common desktop PCs with the same hardware configuration:

  • CPU: Intel® Core 2 Duo E6750 2.66 GHz
  • Memory: 2 G
  • Hard disk: Seagate SATA 250G

To avoid effects from unexpected network traffic, the example uses an isolated network environment for dedicated performance benchmarking purposes. A 10/100M switch was used to connect the machines.

Once the hardware is ready, you need to install the operating system. The example uses SuSE Linux® Enterprise Server 10 SP2 as the OS of the application server. Most of the open source Java application servers currently are running on Linux platforms.

After installing the OS, you might need to adjust some kernel settings so the application server works well under a big load. A typical parameter to tweak is the process handler number that the kernel allows. By default the value is 1024, but it is not enough for an application server performance benchmark. You can change it to a bigger value: $ ulimit -n 10000.

Based on your selected OS and installation type, you might want to adjust other settings, such as firewall settings, to ensure no port blocking on target application servers; and network settings, to ensure that it can operate under your target load level.

After finalizing the OS settings, you'll start to install the target application server and the database software. Following the manual for the target application servers, install JDK and application servers. Once the application servers are up and running, you may need to adjust some settings for large load testing. In a performance benchmark, the load-generation tool will generate large concurrent clients to send HTTP requests to the application servers. You need to extend maxThreads and acceptCount to a bigger value.

Using WebSphere Application Server CE and JBoss as an example, shut down the server and change the tomcat6 module in $WASCE_HOME/var/config/config.xml, as shown in Listing 1.

Listing 1. Change Tomcat module
<module name="org.apache.geronimo.configs/tomcat6/2.1.4/car"> 
        <gbean name="TomcatWebConnector"> 
            <attribute name="host">${ServerHostname}</attribute> 
            <attribute name="port">${HTTPPort + PortOffset}</attribute> 
            <attribute name="redirectPort">${HTTPSPort + PortOffset}</attribute> 
            <attribute name="maxHttpHeaderSize">8192</attribute> 
            <attribute name="maxThreads">3000</attribute> 
            <attribute name="minSpareThreads">25</attribute> 
            <attribute name="maxSpareThreads">75</attribute> 
            <attribute name="enableLookups">false</attribute> 
            <attribute name="acceptCount">4000</attribute> 
            <attribute name="connectionTimeout">${WebConnectorConTimeout}</attribute> 
            <attribute name="disableUploadTimeout">true</attribute> 

To keep the same level for JBoss, you need to change $JBOSS_HOME/server/standard/deploy/jbossweb.sar/server.xml, as shown in Listing 2.

Listing 2. Keep the same level
<Service name="jboss.web"> 

      <!-- A HTTP/1.1 Connector on port 8080 --> 
      <Connector protocol="HTTP/1.1" port="8080" address="${jboss.bind.address}" 
        connectionTimeout="20000" redirectPort="8443" maxThreads="3000"
       minSpareThreads="100" maxSpareThreads="700" 
enableLookups="false" acceptCount="4000" disableUploadTimeout="true"/>

The changes shown above are basic settings. Depending on your test goals, you might need to adjust other settings. For example, you may need to change the max database connection number on the database server. For an out-of-box environment, these settings should be enough.

Now you can boot the servers and deploy DayTrader. By default, DayTrader is built to run on WebSphere Application Server CE/Geronimo, but there is a read-me file with instructions for running it in JBoss. To run it in other application servers, you can do some resource configuration on your own. Of course, it would be appreciated if you contribute your settings back to the open source community to help DayTrader become more broadly used.

After DayTrader is deployed and running successfully, you need to follow its instructions to configure the runtime parameters and initialize the databases. DayTrader supports three types of databases: DB2, Oracle, and Derby. The example used DB2® as the back-end database.

Smoke test

The first test run after first assembly to provide some assurance that the system under test will not catastrophically fail. After a smoke test proves that the software will not crash outright, the assembly is ready for more stressful testing.

Once DayTrader is up and running, you can have your load-generation tool put some load on it. Generally, a smoke test is required. Through the smoke test, you can:

  • Prepare load scripts
  • Expose potential configuration problems
  • Learn the limits of your environment

It's important to understand these factors when executing effective performance benchmark testing. The previously mentioned kernel and application server configuration are found through the smoke test.

When preparing to load scripts, keep in mind that different load-generation tools have different syntax and formats. In general, for load generation scripts you should consider:

  • Simulated concurrent client number
  • Repeat number
  • Repeat intervals
  • Log content and format

You might need several repetitions to debug and enrich your load scripts before you start the formal execution. It could take a good bit of time, depending on your test goal and the selected load-generation tool. (Personally, I think Apache JMeter is very easy to get started. The GUI can save you a lot of time on script preparation.)

Another benefit of smoke testing is that you learn the limits of your SUT. You can set a confidence level for the test results when doing the formal performance benchmark.

Run the test and record raw data

After the smoke test, you should get all load scripts ready so the formal performance benchmark execution will be easy. There are a few things you need to keep in mind:

  • For each load script, do a warm-up run to get target primitives compiled or loaded. After that, run a sustained period of time to get a stable result. For the example, each load script was run for 20 minutes.
  • To get objective results, you should consider repeating the same load script several times. For the example, for each DayTrader primitive it was repeated three times to get an average value.
  • Every time you execute load scripts on TradeScenario, you need to reinitialize the database. Executing TradeScenario will increase the data scale, which causes a slowdown on the next database-related execution.

You can prepare a data sheet to record the test results to easily turn them into charts for comparison. Figure 2 shows a sample data sheet.

Figure 2. Sample data sheet for recording raw data
Figure 2 shows a sample data                 sheet
Figure 2 shows a sample data sheet

Display the results for reporting and analysis

Once you finish running the test and have the results recorded, you need to do turn the data into a visible display (spreadsheet, chart, graph) for reporting and analysis. For example, WebSphere Application Server CE and JBoss were benchmarked and yielded the results shown in Figure 3.

Figure 3. Sample throughput chart
WebSphere Application Server CE and JBoss were benchmarked and yielded the results shown in the                 charts
WebSphere Application Server CE and JBoss were benchmarked and yielded the results shown in the charts
Table 3. Sample throughput data sheet
PrimitiveWebSphere Application Server CE (Pages/second)JBoss (Pages/second)
Web PingServlet 22241.191 21353.691
PingSession1 19300.566 17323.521
PingJSP 8160.188 7253.987
PingServletWriter 22232.547 20823.371
PingServlet2Servlet 21341.996 19742.348
PingJspEL 4008.242 3823.502
PingJDBCRead 4854.584 3704.013
PingServlet2JNDI 21188.262 13274.154
EJB3 PingServlet2Session 11396.99 2680.211
PingServlet2Entity 5066.833 4434.003
PingServlet2Session2EntityCollection 1306.057 635.269
PingServlet2Session2CMROne2One 5152.952 1271.41
PingServlet2Session2CMROne2Many 1155.966
TradeScenario JDBC Direct 696.466 265.345
Full EJB3 572.112 258.502
Figure 4. Sample average response-time chart
Sample average response-time chart
Sample average response-time chart
Table 4. Sample average response-time data sheet
PrimitiveWebSphere Application Server CE (seconds)JBoss (seconds)
Web PingServlet 0.012 0.013
PingSession1 0.014 0.016
PingJSP 0.034 0.027
PingServletWriter 0.013 0.013
PingServlet2Servlet 0.013 0.014
PingJspEL 0.046 0.048
PingJDBCRead 0.061 0.08
PingServlet2JNDI 0.014 0.023
EJB3 PingServlet2Session 0.026 0.112
PingServlet2Entity 0.058 0.067
PingServlet2Session2EntityCollection 0.229 0.468
PingServlet2Session2CMROne2One 0.058 0.236
PingServlet2Session2CMROne2Many 0.255
TradeScenario JDBC Direct 0.359 1.122
Full EJB3 0.5 1.152

The next logical task would be performance analysis, but that discussion is outside the scope of this article.


In this article, you learned about using the DayTrader sample application to perform a benchmark on Java application servers. The article explored requirements gathering, tools, test environment, test execution, and reporting the results.

Downloadable resources

Related topics

  • Explore the Java client/server benchmark standard SPECjAppServer2004. SPECjAppServer2004 is designed to measure the performance of J2EE V1.3 application servers.
  • Get the source code for the Apache DayTrader performance benchmark sample.
  • Get the load-generation tool Apache JMeter to load-test functional behavior and measure performance.
  • Learn more about lots of performance test tools.
  • Follow developerWorks on Twitter.
  • Visit developerWorks Open and collaborate with some of IBM’s brightest minds in a community setting.
  • Start developing with product trials, free downloads, and IBM Bluemix services.
Zone=Open source
ArticleTitle=A performance benchmark method for comparing open source Java application servers