Automation for the people: Hands-off load testing

Use Apache JMeter with Apache Ant to run load tests frequently

Load testing is often relegated to late-cycle activities, but it doesn't need to be that way. In this installment of Automation for the people, automation expert Paul Duvall describes how you can discover and fix problems throughout the development cycle by creating a scheduled integration build that runs JMeter tests.

Paul Duvall (paul.duvall@stelligent.com), CTO, Stelligent Incorporated

Paul DuvallPaul Duvall is the CTO of Stelligent Incorporated, an Agile consultancy that helps development teams deliver production-ready software. He is the co-author of the Addison-Wesley Signature Series book, Continuous Integration: Improving Software Quality and Reducing Risk (Addison-Wesley Professional, 2007; Jolt Award 2008 winner). He also contributed to the UML 2 Toolkit (Wiley, 2003) and the No Fluff Just Stuff Anthology (Pragmatic Programmers, 2007).



08 April 2008

Also available in Chinese Japanese

How many concurrent users can access your software system? How much data can be loaded without performance degradation? What are your system's throughput requirements? How often are these requirements tested? What if you could specify and validate that these load and performance requirements are being met at least once a day? By running load tests as part of a scheduled and automated build, you can more quickly determine how your system performs under certain load conditions and quickly adapt to changes.

About this series

As developers, we work to automate processes for end-users; yet, many of us overlook opportunities to automate our own development processes. To that end, Automation for the people is a series of articles dedicated to exploring the practical uses of automating software development processes and teaching you when and how to apply automation successfully.

A project I once worked on established a good set of automated tests that could load test the application while it ran multiple transactions. The problem was that the tests required some manual tweaking, so the development team couldn't run them without human intervention. This limited testing to times when the tester was available (usually waking hours only). In practice, testing occurred only every couple of days — not frequently enough for timely problem detection.

In this article, I explore techniques for creating automated tests using JMeter, running the tests as part of an automated build, and scheduling the tests to run automatically every day — typically when machine usage is lower. Running the tests as part of a scheduled build lets you:

  • Execute load tests at any time
  • Detect and resolve load and performance problems early in the development process
  • Monitor the latest load- and performance-test reports from your build server
  • Reduce bottlenecks and errors that can occur when you depend on a single person to configure and run the tests

Letting JMeter do the heavy lifting

Apache JMeter is an open source project you can use to simulate a heavy load on a server (see Resources for more information on JMeter). JMeter's impressive documentation set describes how to use its many features and provides a plethora of examples.

Running JMeter

After downloading and extracting the JMeter ZIP file (see Resources for a link to download JMeter), open a command prompt to the location where you extracted JMeter and type cd bin to change to the bin directory. From the bin directory, type jmeter to open the JMeter Swing application, shown in Figure 1:

Figure 1. The JMeter GUI
Using the JMeter GUI to create a test plan

Creating a test plan

Writing tests, by example

JMeter comes with many example test plans and scripts. Instead of creating a test plan from scratch, use the examples in the docs directory and incrementally configure the test plans as your project evolves. Much of the complexity is in learning to write load tests that effectively exercise the load and performance requirements.

You can use the JMeter GUI to create test plans. The different types of test plans in JMeter include:

  • Web test plan
  • Database test plan
  • FTP test plan
  • LDAP test plan
  • Extended LDAP test plan
  • Web service test plan
  • JMS point-to-point test plan
  • JMS topic test plan
  • Monitor test plan
  • Listeners

Each of the test plans is saved in XML format in a file with a .jmx suffix. This nonbinary format makes plans easier to edit later. Although you can create test plans by following JMeter's XML schema, it's much easier to use the GUI. Later, you'll see an example of parameterizing JMeter configuration values to customize ways the tests are run.


Labor-saving load testing

Using the GUI to run tests, though, requires that a person be present to execute them. This increases the chance of process bottlenecks and knowledge silos. By running the tests through an automated build such as an Ant build, you can configure the JMeter tests to run without opening the JMeter application. What's more, the tests are run the same way every time — and without requiring overtime pay.

Driving JMeter tests with Ant

Once I've learned how to use a GUI software tool, I like to see if it can run certain utilities from the command line, so that I don't need to perform the same actions repeatedly. For example, every time I open the JMeter application, I'd like to select File > Open, open the file, and run one or more tests. I can script this set of actions and run them the same way every time. Fortunately, an Ant task has been written to do this very thing for JMeter: it executes load tests while providing a way to pass in optional parameters and properties.

Example build script

The JMeter distribution's extras directory includes an example build.xml script that demonstrates the use of the JMeter Ant task.

In Listing 1, I use Ant's taskdef task to define the JMeter task. I name it jmeter so that I can use it elsewhere in the Ant script. To use this script you must have the ant-jmeter.jar file (see Resources for a download link) in the Ant classpath.

Listing 1. Defining the JMeter task in Ant
<taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>

The example code in Listing 2 runs a single JMeter load test, called BreweryTestPlan.jmx. To run all of the tests in a certain directory, simply enter *.jmx in place of a specific filename. The required attributes of the jmeter task are jmeterhome, testplan(s), and resultlog or resultlogdir. (Listing 2 doesn't show resultlogdir, because it uses resultlog.)

Listing 2. Running JMeter from Ant
<jmeter
  jmeterhome="${jmeter.home}"
  resultlog="${basedir}/target/JMeterResults.xml">
  <testplans dir="${basedir}/tests/load" includes="BreweryTestPlan.jmx"/>
</jmeter>

The Ant code in Listing 2 creates an output file called JMeterResults.xml, which is used to create HTML reports.

Rendering reports using XSLT

By providing the JMeterResults.xml file as input to the xslt Ant task in Listing 3, I can generate an HTML report of all the JMeter tests I ran in Listing 2. The XSL stylesheet (jmeter-results-detail-report_21.xsl) provided in the JMeter extras directory is used to transform the JMeterResults file into HTML.

Listing 3. Creating JMeter HTML report using XSLT
<xslt in="${basedir}/target/JMeterResults.xml"
  out="${basedir}/target/JMeterResults.html"
  style="${jmeter.home}/extras/jmeter-results-detail-report_21.xsl"/>

JMeter also provides a less detailed XSL stylesheet file that summarizes the load tests' results.

Revealing reports in HTML

Figure 2 is an example of the HTML report that is generated using the xslt task in Listing 3. It displays each of the load tests run, along with test status, time, and aggregate status and time of all of the tests.

Figure 2. Generating the JMeter HTML report
Generating the JMeter HTML report

Later in the article, I'll show you how these reports can be displayed from a CruiseControl Continuous Integration (CI) server (see Resources).

Passing parameters to JMeter

Depending on the types of tests you run, you may want to pass parameters and properties to vary the way a single test or suite of tests executes. For example, Listing 4 demonstrates how to increase the JVM memory and specify the number of threads and loops:

Listing 4. Passing optional parameters and properties to JMeter
<jmeter
  jmeterhome="${jmeter.home}"
  resultlog="${basedir}/target/JMeterResults.xml">
  <jvmarg value="-Xincgc"/>
  <jvmarg value="-Xmx128m"/>
  <jvmarg value="-Dproperty=value"/>
  <property name="request.threads" value="5"/>
  <property name="request.loop" value="50"/>
  <property name="jmeter.save.saveservice.assertion_results" value="all"/>
  <property name="jmeter.save.saveservice.output_format" value="xml"/>
  <testplans dir="${basedir}/tests/load" includes="BreweryTestPlan.jmx"/>
</jmeter>

Many other built-in parameters and properties are available for modifying how JMeter tests are run (see Resources for more information).

Using parameters and properties gives you a certain level of flexibility in how to execute load tests, but it doesn't address how to run load tests in different target environments, such as testing and staging. To add environment-specific information to test plans, you need to put tokens in the .jmx files that can be filtered and modified when the load tests run in an automated build script.


Just-in-time load testing

Once you get a load test running with an automated build, schedule it to run on a periodic basis, such as nightly. You can do this using a CI or build-management server.

Scheduling CruiseControl to run load tests daily

The purpose of using a CI server is to run an automated build whenever a change is applied to a project's version-control repository. You can also configure it to run builds at specific times. Because load testing usually requires an elevated level of computing resources, running tests when these resources are not at capacity — such as late at night or early morning — can be beneficial.

In Listing 5, an automated build is scheduled to run at 11:00 PM (2300) with CruiseControl (see Resources). You can modify the CruiseControl configuration file to run a delegating build with a specific Ant target, such as one named run-load-tests.

Listing 5. Run a scheduled load test with CruiseControl
  ...
  <modificationset>
    <svn RepositoryLocation="${svnrepo.location}"/>
    <timebuild username="admin" time="2300"/>
  </modificationset>
  ...

By scheduling load tests to run nightly, as in Listing 5, you won't hear excuses about workload, vacation days, or forgetting to run the tests — they just run.

Displaying reports in CruiseControl

You've already seen how to display JMeter test reports using Ant. However, by itself, the JMeter report limits the communication to one developer on a single machine. Load testing affects the entire application, so the whole team will want to see the results. The beauty is that you can easily configure your CI server to display these reports. Because the report is already generated with Ant, it's just a matter of making the JMeter HTML report accessible from the CruiseControl project dashboard. You can do this by simply adding a few lines to CruiseControl's config.xml file, as shown in Listing 6:

Listing 6. Configuring CruiseControl to display JMeter reports
<project name="brewery">
...
<log>
  <merge dir="merge dir="projects/${project.name}/reports/jmeter" />
</log>
...
</project>

Now, everyone on the team can be (literally) on the same page. Many of the other CI and build-management servers provide similar report-integration capabilities.


Conclusion

I've demonstrated how to add automated load tests to your development toolbox. By running load tests with the automated build and then scheduling the tests to run periodically, you can learn about system-capacity issues long before they become a problem. This approach makes it easier to assess the impact of architecture and data changes. When coupled with other techniques described in this article series, it enables development teams to deliver higher-quality software frequently.


Download

DescriptionNameSize
Sample Ant scripts for this articlej-ap04088-jmeter-example.zip6KB

Resources

Learn

Get products and technologies

  • Ant: Download Ant and start building software in a predictable and repeatable manner.
  • JMeter: Download JMeter.
  • JMeter Ant task: Ant task used to execute JMeter test plans.
  • CruiseControl: Download the CruiseControl Continuous Integration Server.
  • Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Java technology on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java technology
ArticleID=300198
ArticleTitle=Automation for the people: Hands-off load testing
publish-date=04082008