Test cloud-based applications with Apache JMeter

Learn effective techniques and best practices for RESTful API testing with JMeter

JMeter is a powerful testing tool that you can use to test applications that run in the cloud. But managing and maintaining JMeter scripts can be challenging if you don't know how to use JMeter effectively. This article presents techniques that can help you implement well-designed, automated JMeter tasks in your cloud-based application testing.

Share:

Shalini Gilra (shagilra@in.ibm.com), Automation Lead, IBM

Author photoShalini Gilra is an automation developer for the SmartCloud platform at the IBM India Software Labs, Pune. She has 12 years of experience in test automation with Rational Functional Tester, Selenium, JMeter, JUnit, STAF/STAX, and Perl for functional and REST API testing. Among her publications is the developerWorks article "Incorporating external features in IBM Rational Functional Tester." She is a Sun-certified Java programmer and also holds a certification in WebSphere Portal Server.



Prem Prakash (premprakash@in.ibm.com), Software Test Specialist, IBM

Author photoPrem Prakash is a software test specialist for the SmartCloud platform at IBM India Software Labs, Pune. He has around 12 years of IT experience working and specializing in server-side testing, shell and Perl scripting, and database testing.



Tom Tuohy (tuohy@us.ibm.com), Automation Developer, IBM

Author photoTom Tuohy is an automation developer for the SmartCloud platform at the IBM Littleton, Massachusetts Labs. While at IBM, he has designed, built, and contributed to automation frameworks for Lotus Notes/Domino, Lotus Quickplace/Quickr, Lotus Connections, and Sterling Commerce. He has more than 20 years of experience specializing in back-end testing for APIs, web services, and enterprise applications written in C, C++, and Java.



09 September 2013

Also available in Chinese Russian Japanese

Cloud-based applications typically involve several components that interact through an API that exchanges data in either XML or JavaScript Object Notation (JSON) format. This article presents techniques for using Apache JMeter — a GUI-based open source testing application — to perform functional, performance, and reliability testing of cloud-enabled applications that use RESTful web APIs and JSON.

RESTful web APIs

A RESTful web API uses HTTP and Representational State Transfer (REST) principles that consists of:

  • The base Uniform Resource Identifier (URI) for the API
  • The data supported by the API, which is often in JSON format
  • The set of operations that the API supports by using HTTP GET, PUT, POST, and DELETE methods

Learn about JMeter's capabilities as a testing tool for RESTful APIs and its relevance to testing cloud based-applications. Because multitenancy is an important cloud feature, this article describes techniques for incorporating data separation and data retrieval programmatically to ensure data integrity. It shows how to write effective JMeter test scripts (also known as test plans or JMX files) that facilitate maintenance, re-usability, and modularity. Learn how to use configuration and properties files to ensure that the same scripts can run in multiple environments.

The authors assume you are familiar with the JMeter UI and have significant experience with using JMeter.

Where to set properties for optimal script reuse

Because cloud-enabled applications are generally easy and quick to replicate and deploy, it's possible to test them in multiple environments. If you need to test and run automation scripts in more than one environment, it's beneficial to use a separate properties file in JMeter to define data (including login credentials) for connecting to resources such as application servers and databases.

JMeter properties and variables are defined in three files in the JMETER_HOME/bin directory. When JMeter launches, it loads these files in the following sequence:

  1. jmeter.properties
  2. An optional user-defined properties file
  3. system.properties

If you add any new properties or change existing properties while JMeter is running, you must shut JMeter down and restart it for the changes to take effect.

The jmeter.properties file stores properties that pertain to the JMeter application itself. Keep only JMeter program- or framework-specific properties in this file. Create a separate file (with the file name of your choice) for storing test-environment-specific properties and variables that are global to all scripts that are associated with the application that's under test — for example, an administrator username/password. In the jmeter.properties file, uncomment the user.properties setting and set the user.properties value to the name of the file you created. The example in Listing 1 sets the value to myuser.properties:

Listing 1. Specifying a user properties file in jmeter.properties
# Should JMeter automatically load additional JMeter properties?
# File name to look for (comment to disable)
user.properties=myuser.properties

The sample user properties file in Listing 2 shows the format for defining the variables in the user properties file. (No spaces are allowed anywhere to the left of the equals sign in a definition; property values can include spaces when necessary.)

Listing 2. Sample user properties file
#----------------------------------------------------------------
# FVT API Test Environment parameters
#----------------------------------------------------------------
#
# --- Login Credentials
USER_LOGIN=admin@in.ibm.com
USER_PASSWORD=password
#
# --- Application Server
APP_SERVER=localhost
APP_PORT=80
APP_CONTEXT=ctx
PROTOCOL=http
#
# --- Database Server${DB_NAME}
DB_HOST=localhost
DB_PORT=50000
DB_NAME=dbname
DB_USER=dbadmin
DB_PASSWORD=dbpassword

JMeter's third properties file, system.properties, should be reserved for system-wide properties that you must define for all of your scripts. For example, if all of your scripts use a specific database server, you can specify the relevant properties in the system.propterties file.

The JMeter User Defined Variables control panel, shown in Figure 1, shows how a JMeter script reads the properties that are defined in the user properties file.

Figure 1. How a JMeter script reads configuration data that's defined in the user properties file
Screen capture of the JMeter User Defined Variables control panel, showing how a JMeter script reads the properties that are defined in user properties file

Each item in the Value column of the control panel is in the format:

${__property(VARIABLE_NAME,VARIABLE_NAME)}

For example, the USER_LOGIN variable from the user properties file is read as the ${__property(USER_LOGIN, USER_LOGIN)} function in the script. The first USER_LOGIN in the parentheses is the name of the variable that's defined in the properties file (and listed in the control panel's Name column). The second instance is the default or fallback value if the variable isn't defined in the properties file.

There's no firm rule for when to define a variable in a properties file and when to define it as a variable inside a JMeter script. But two guidelines can help you impose consistency and cut down on unnecessary duplication of variable definitions across multiple JMeter scripts:

  • If you're using the same value consistently in several scripts, define the data in either the user properties file or the system.properties file. Examples include system variables such as database names and server names, and execution-scoped variables such as logging levels.
  • If several of your scripts use a value that might change from script to script, define it as a script variable or define it in an external data file, such as comma-separated value (CSV) file.

Payload separation using JSON template files

Many cloud APIs require a JSON payload as input. JSON defines a structured set of elements that can be nested within other elements. Each element defines one or more name/value pairs. Functional tests involve supplying data in a specified format repeatedly. For example, in a typical REST API invocation, the JSON payload is passed in the body of the REST HTTP request and usually contains hard-coded data. The hard-coded data is usually repeated in more than a single test and dispersed throughout the script.

A common problem with this approach is that if the JSON structure (or data) changes — perhaps because the API parameters change — you must go into the JMeter test, locate the HTTP request body, and change the JSON structure (and data) to match the new requirement. If you have thousands of HTTP requests across multiple JMeter test cases that use this JSON structure, you must do lots of repetitive editing.

A better approach is to create a template of the JSON structure and define substitution strings where the data will go. The JSON templates don't contain any hard-coded data values but instead define reference variables that are loaded with the actual data when the script runs. The template is then read into a variable in the JMeter script and substituted for in the HTTP request body.

Listing 3 shows the traditional way of defining a JSON payload:

Listing 3. Static JSON definition in JMeter test plan
{
   "Customer":{
      "CustomerType":"DIRECT",
      "CustomerNumber":"1234567890",
      "Organization":{
         "OrgName":"IBM",
         "Phone":"999-999-9999",
         "AddressSet":[
            {
               "AddressLine1":"Tech Park One",
               "AddressLine2":"",
               "AddressType":"MAILING",
               "City":"Pune",
               "Country":"India",
               "State":"Maharashtra",
               "StateCode":"MH",
               "PostalCode":"411006"
            }
         ],
         "Contact":{
            "FamilyName":"Gilra",
            "GivenName":"Shalini",
            "EmailAddress":"shagilra@in.ibm.com",
            "NamePrefix":"Miss",
            "LanguagePreference":"EN_US",
            "WorkPhone":"999-9999"
         }
      }
   }
}

Listing 4 shows the dynamic way of defining the JSON in a template:

Listing 4. Dynamic JSON definition in an external JSON template file
{ 
   "Customer":{ 
      "CustomerType":"${__eval(${STR_CUSTOMERTYPE})}", 
      "CustomerNumber":"${__eval(${STR_CUSTOMERNUMBER})}", 
      "Organization":{ 
         "OrgName":"${__eval(${STR_ORGNAME})}", 
         "Phone":"${__eval(${STR_PHONE})}", 
         "AddressSet":[ 
            { 
               "AddressLine1":"${__eval(${STR_ADDRESSLINE1})}", 
               "AddressLine2":"${__eval(${STR_ADDRESSLINE2})}", 
               "AddressType":"${__eval(${STR_ADDRESSTYPE})}", 
               "City":"${__eval(${STR_CITY})}", 
               "Country":"${__eval(${STR_COUNTRY})}", 
               "State":"${__eval(${STR_STATE})}", 
               "StateCode":"${__eval(${STR_STATECODE})}", 
               "PostalCode":"${__eval(${STR_POSTALCODE})}", 
            } 
         ], 
         "Contact":{ 
            "FamilyName":"${__eval(${STR_FAMILYNAME})}", 
            "GivenName":"${__eval(${STR_GIVENNAME})}", 
            "EmailAddress":"${__eval(${STR_EMAILADDRESS})}", 
            "NamePrefix":"${__eval(${STR_NAMEPREFIX})}", 
            "LanguagePreference":"${__eval(${STR_LANGUAGEPREFERENCE})}", 
            "WorkPhone":"${__eval(${STR_WORKPHONE})}", 
         } 
      } 
   } 
}

The JSON entities in Listing 3 contain only hard-coded data. The template in Listing 4, in contrast, contains only reference-variable names, so any JMeter test plan can use the template. The actual data is stored separately in a CSV data file, which is discussed in the next section.

Notice that the JSON in Listing 4 calls the JMeter __eval() function for each defined substitution variable. This addition enables the variable to be evaluated at runtime when the JMeter script executes.

Figure 2 and Figure 3 show how the JSON entity template file is specified in the JMeter test script. Figure 2 shows the HTTP Request control panel:

Figure 2. Using the contents of a template file in your tests
Screen capture of the JMeter HTTP Request control panel, configured to use the contents of a template file in tests

In the example in Figure 2, the CUSTOMER_JSON variable represents the entire JSON Customer element. That variable, enclosed in the _eval() function, appears as the HTTP request body (under the Send Parameters With the Request heading on the Parameters tab. In Figure 2, then, the request body is ${_eval(${CUSTOMER_JSON})}.

The CUSTOMER_JSON variable is defined in the User Parameters control panel, shown in Figure 3:

Figure 3. Reading a JSON template file in a JMeter script
Screen capture of the User Parameters control panel for a JMeter test, which shows how the script reads a JSON template file

In Figure 3, the CUSTOMER_JSON variable is set to FileToString() with a path to the JSON template file as a parameter. The entire contents of the JSON entity template file is read into the CUSTOMER_JSON variable. As a result, the contents of the JSON entity template file are evaluated at runtime, and all of the defined substitution strings are translated into the data that's defined for them. (The next section explains how the substitution variables are associated with the actual data.)

Because the JMeter JSON entity template files are external to the JMeter test scripts, you can store them in a separate directory, such as JMETER_HOME/tests/jsontemplates. When you access a JSON entity template from a JMeter test plan, specify the name relative to the JMETER BIN directory — for example:

../tests/jsontemplates/customer_template.json

You can also store the templates outside the JMETER_HOME directory, in which case you must provide the absolute path.


Data abstraction using CSV Data Set Config elements

Although it might seem like extra work initially to separate the test data from JMeter test plans, separation results in cleaner tests that are also easier to manage. You realize the benefits quickly when you need to make changes to a test. Some data might still exist locally in each test plan, but most of the test data is abstracted out into external files — either in a properties file (as we discussed earlier) or a CSV data configuration file. The result is a test suite that's easier to maintain and that in most cases doesn't require you to edit JMeter test plans to change any data values.

JMeter uses CSV files to store lines of comma-separated data. The JMeter framework's CSV Data Set Config functionality provides a way to read CSV file test data dynamically at runtime. An additional benefit of storing your test data as CSV is that you can store multiple lines of data that represent multiple data objects/variables or data for multiple iterations of loop processing. JMeter 2.3.4 and later also support providing a CSV header as the first line of the file. The header's delimited string names are then used to store the defined data values. Typically, the data on each subsequent line represents an iteration of a loop. In this sense, the script is largely "data-driven," and a tester can change the data in the CSV file without requiring any changes to the JMX script.

The data can also be shared across multiple test threads, and you can store your test data in more than one CSV file. For example, if you have a test that logs in with a set of user credentials and then creates an order for that user, you can create two CSV files — one to hold the user credentials and another to hold the order information. Sometimes, more than a single JMX script might need access to a CSV file. In such cases you must place the CSV file in a location where multiple scripts can access it.

The iteration data defined within a CSV file is usually associated with variable names. You define these variable names in the header, separated by commas, and they match the number of data values one-for-one. When JMeter processes each line of the file, it assigns the corresponding data value to the associated variable name for that position. Listing 5 shows sample CSV file contents:

Listing 5. Sample CSV file (customer.csv)
"STR_TESTCASE","STR_CUSTOMERTYPE","STR_CUSTOMERNUMBER","STR_ORGNAME","STR_PHONE",
"STR_ADDRESSLINE1","STR_ADDRESSLINE2","STR_ADDRESSTYPE","STR_CITY","STR_COUNTRY",
"STR_STATE","STR_STATECODE","STR_POSTALCODE","STR_FAMILYNAME","STR_GIVENNAME",
"STR_EMAILADDRESS","STR_NAMEPREFIX","STR_LANGUAGEPREFERENCE","STR_WORKPHONE", 

"Testcase1","DIRECT","1234567890","IBM","999-999-9999","Tech Park One","","MAILING",
"Pune","India","Maharashtra","MH","411006","Gilra","Shalini","shagilra@in.ibm.com","Miss",
"EN_US","999-9999", 

"Testcase2","DIRECT","1234567891","IBM","999-999-9999","Tech Park One","","MAILING",
"Pune","India","Maharashtra","MH","411006","Prakash","Prem","premprakash@in.ibm.com",
"Mr","EN_US","999-9999", 

"Testcase3","DIRECT","1234567892","IBM","999-999-9999","550, Kings Street","","MAILING",
"Littleton","United States","MA","MA","1460-1250","Tuohy","Tom","tuohy@us.ibm.com","Mr",
"EN_US","999-9999",

Downloadable conversion script

The implementation details of converting a JSON payload into a JSON template file and converting data to a CSV file can be tedious. You need to define a reference variable for each attribute-value pair and map to the CSV file with the corresponding data value. To ease this otherwise time-consuming task, this article provides a sample Windows batch file that performs these tasks for a specified JSON payload. See Download to get the batch script and instructions for using it.

When JMeter processes the CSV file in Listing 5, for the first iteration it assigns Pune to the STR_CITY variable, India to the STR_COUNTRY variable, and so on. It then does the same thing for the next two lines/iterations, but using the values specified on those lines. When the script runs, the specified variables load and contain their values until they are overwritten or the script ends. To avoid overwriting variables unintentionally, name your variables carefully. Also, when you debug a JMeter script, it's important to know where the variables originate, because the location isn't defined in the script itself. Use of consistent naming conventions for your variable names in CSV files will help.

Figure 4 shows how to specify the CSV file in the CSV Data Set Config control panel:

Figure 4. Specifying and reading a CSV file in JMeter
Screen capture of the JMeter CSV Data Set Config control panel, which is used for specifying and reading a CSV data file

In Figure 4, the Filename field is set to the relative CSV file path that is to be read. The Allow quoted data? value is set to true, because the data values in the CSV file are enclosed in quotation marks. The Recycle on EOF? option is set to false, and Stop thread on EOF? is set to true, which causes the thread to stop on reaching the end of file (EOF).


Using While Controllers for looping

After you separate test data into separate CSV Data Set Config files, it's much easier to use the data set to iterate through a series of similar but separate actions. JMeter includes looping constructs such as the While Controller to help with iteration. By organizing your test actions under a looping construct, you can execute all required data-driven tests from within a single JMeter test plan.

This approach is useful for cloud testing because it addresses the need to test access to an API from multiple access roles or different sets of data (positive, negative, boundary values, and so on). With looping, you can put all of the test permutations for a single API test target in a single test plan, which makes test cases at least 20 times faster to write.

Figure 5 shows how to use the While Controller control panel to tell JMeter to loop through the CSV file data:

Figure 5. Looping through all the data in the CSV file
Screen capture of the JMeter While Controller control panel, which tells JMeter to loop through all the data in the CSV file

In Figure 5, the Condition (function or variable) field is set to the jexl() function. This function compares the variable from the CSV data file against EOF and tells JMeter to exit the loop when there are no more data rows in the CSV file. The condition can be any variable or function that eventually evaluates to false.


Scripting with BeanShell

You can include Java programming in your JMeter scripts by using the BeanShell scripting language (see Resources). BeanShell scripting is useful, for example, when your script must manipulate variables that you've captured with JMeter's Regular Expression Extractor. The BeanShell script executes the program that's passed to it and returns the result at runtime.

In some cloud applications the response data (likely to be in JSON format) returned from an API command is used, with some modifications, as request data to another API command. Working with unknown, dynamic data can be tricky and difficult unless you can use programming in your scripts. With BeanShell scripting, you can manipulate the JSON payload or read a JSON payload's attribute values at runtime by making use of the JSONObject class library (see Resources). To use JSONObject in your JMeter script, include java-json.jar (see Resources for a download link) in JMeter's classpath by copying the JAR to the JMETER_HOME/lib folder.

You can define and use BeanShell scripts in many ways for different purposes. With the BeanShell PreProcessor or PostProcessor, you can apply a piece of code in the JMeter BeanShell Sampler either before or after the sampler executes. With BeanShell Assertion, you can test for conditions such as whether your JMeter variable is holding an expected value.

The general steps for defining and using a BeanShell script are:

  1. In JMeter, create a BeanShell Listener, a BeanShell PreProcessor, a BeanShell PostProcessor, or BeanShell Sampler.
  2. Get your variable by using vars.get("variable") in the BeanShell script.
  3. Use the Java programming language to process the BeanShell script.
  4. Put your processed variable(s) back into the specified JMeter variable (which can be an existing variable or new one) by using vars.put("variable")in the BeanShell script.

The example BeanShell script in Listing 6 modifies the GivenName and WorkPhone values that are nested at the third level in the JSON in Listing 3:

Listing 6. Using BeanShell scripting to modify two JSON values
custPayload= vars.get("testResp");
org.json.JSONObject custJSON= new org.json.JSONObject(custPayload);

if (custJSON.has("Customer") && custJSON.get("Customer")!= null) {
   org.json.JSONObject contactJSON = custJSON.getJSONObject("Customer").getJSONObject(
   "Organization").getJSONObject("Contact");
   contactJSON.put("GivenName", "Shalini");
   contactJSON.put("MobilePhone", "9923406159");
}
vars.put("updatedCustPayload", custJSON.toString());

Now the request can use the ${updatedCustPayload}variable in the API UPDATE command.

BeanShell scripting can be used in many other ways to manipulate JMeter variables or JSON data — such as performing arithmetic operations, fetching the value of a variable, or substituting the value of a particular variable for another. Overall, BeanShell can be useful for performing tasks that JMeter doesn't support directly.


Using Module Controller to modularize reusable fragments

It's not unusual for a complex piece of a test plan to include many variables and functions. Often these segments are also used more than once or twice in other test plans. In such a situation, you can reduce maintenance effort by breaking these segments into submodules that can be reused elsewhere. Then if the piece of reusable functionality requires any changes in the future, you only need to modify it in one place.

The JMeter Module Controller is a mechanism for substituting test-plan fragments into the current test plan at runtime. The fragment can be located in any thread group or on the WorkBench. Any fragment that the Module Controller uses must have a unique name, because the name is used to find the target controller when a test plan is reloaded.

Figure 6 shows an example of where to place the Module Controller and the called module in the script:

Figure 6. Placement of a Module Controller and the module that it points to
Screen capture of a JMeter test plan in which a Module Controller is created in between the Login and Logout routines for the Customer Service module, and the Register Customer module that the controller points to is created in the WorkBench

In the test plan shown in Figure 6, Register Customer is defined as a separate module and placed in the WorkBench section. A Module Controller (Module Controller - Register Customer) — placed after the Login User routine and before the Logout routine — points to the Register Customer module. When the script runs, the Module Controller substitutes the Register Customer module in the test plan at the position where Module Controller is located.

Figure 7 shows how to define a Module Controller in the script:

Figure 7. Module Controller pointing to the simple controller that's defined in the WorkBench
Screen capture of a JMeter Module Controller pointing to the simple controller that's defined in the WorkBench

In Figure 7, the Register Customer module is selected from the Module To Run field's drop-down list, which lists all of the available modules.

Use Module Controller when the reusable component is reused within the same script (JMX file). If the reusable fragment will also be used in other scripts, move the fragment to a separate JMX file and call it by using Include Controller, as detailed in the next section.


Including reusable external JMeter files with Include Controller

JMeter's Include Controller provides a placeholder where a JMX file (the parent script) can call another JMeter script (the child script). By breaking a script into small scripts or modules/routines and building a test suite using these subroutines, you achieve modularity in your scripting to enhance comprehensibility and reusability. Use Include Controller to separate reusable code snippets or prerequisites such as Login User and Register User to help better manage and maintain the scripts.

To use Include Controller, first create a child JMX file that contains a reusable routine (such as Login Admin User). Figure 8 shows an example child script that can be included within the parent script via Include Controller:

Figure 8. Child JMX file (LoginUser.jmx)
Screen capture of a child script (LoginUser.jmx)

The child script in Figure 8 defines a test plan that contains a sampler with an HTTP request for Login User. In the HTTP request, the variables used for the protocol, server name, and other settings are defined in the parent script.

The next step is to add an Include Controller in the parent script at the point where you want the child script to be called, and to point the Include Controller to the path of the child JMX file. Figure 9 shows an Include Controller defined in the parent script:

Figure 9. Adding an Include Controller in a parent JMX file
Screen capture of the parent script's Include Controller

In the Include Controller control panel shown in Figure 9, the Filename field stores the relative path to the child JMX file (Login_User.jmx in this example).

The child JMX file can access any variable defined in the parent that is accessible to the Include Controller, and the parent JMX file can use the variables defined in the child JMX file.


Using regular expressions

When JSON is the request payload for REST operations, the response is either in JSON format or is a string representation. After the operation, you must extract the response string, error code, and error-message string to validate whether the feature works as expected. You can extract response data in variables by using JMeter's regular-expressions feature.

Figure 10 shows how to create a simple regular-expression extractor in JMeter's Regular Expression Extractor control panel:

Figure 10. Simple regular-expression extractor
Screen capture of a simple Regular Expression Extractor in JMeter

In Figure 10, the Reference Name (User in this example) stores the values that are extracted using regular-expression syntax. In the Regular Expression field, you provide the syntax for the regular expression (\"LoginName\":\"(.*?)\" in Figure 11) that runs against the response to extract specific data. Examples of frequently used regular-expression syntax for JSON response data include:

  • \s*(.+)\s* to extract the entire response string
  • \"LoginName\":\"(.*?)\" to extract "LoginName": "abc@testmail.com"
  • \"CustomerId\":(\d+) to extract "CustomerId": 2000006

In the Template field in Figure 10, $1$ denotes the grouping of reference variables. The Default Value field is for providing the default value, for debugging purposes, in case the regular expression doesn't match. Follow the practice of stating the default value only at the scripting stage when you add and test regular-expression patterns.


Extending functional JMeter scripts for performance testing

Ideally, performance testing for any application involves two scenarios: a spike in the number of users, and an increase in system load. If you have an existing functional test scenario that covers basic REST functions, it's easy to extend and update it to test the performance of REST services that are hosted on cloud infrastructure. You can load-test the application by modifying the number of requests that should be sent to individual server(s). And you can control the number of requests by configuring ramp-up periods that use appropriate loop iterations.

For example, suppose you have a test plan that executes and verifies a simple set of REST principles based on the following algorithm:

  1. Execute an HTTP POST method to create a customer object.
  2. Execute an HTTP GET method to verify creation of the customer object.
  3. Execute an HTTP PUT method to verify modification of the customer object.
  4. Execute an HTTP DELETE method to verify that the object was deleted.

You can tweak your simple functional test plan — originally designed for a RESTful API functional test scenario — into a performance test script that hits the servers deployed in the cloud environment.

  1. Navigate to the test plan's Thread Group control panel, as shown in Figure 11:
    Figure 11. Figure 11. Performance-simulation settings example
    Screen capture of the JMeter Thread Group control panel, showing performance-simulation settings
  2. Enter ${Thread} in the Number of Threads (users) field. The Thread variable's value (which you'll configure in step 5) sets the number of available threads to reflect how many virtual users are accessing the servers.
  3. Enter ${RampUp} in the Ramp-Up Period (in seconds) field. The RampUp variable's value (which you'll configure in step 5) determines the speed at which the requests (which are based on the number of threads you defined in step 1) are distributed. For example, if you have 10 threads and a ramp-up time of 100 seconds, each thread begins 10 seconds after the previous thread started, for a total time of 100 seconds to get the test up to speed.
  4. Enter ${Loop} in the Loop Count field. The Loop variable's value (which you'll configure in step 5) determines the number of times the test plan will execute.
  5. Navigate to the User Defined Variables control panel and define Thread, RampUp, and Loop as global variables, as shown in Figure 12. For each load that you want to simulate, assign appropriate values to the variables.
    Figure 12. Performance test simulation variables
    Screen capture of the User Defined Variables control panel showing sample performance test simulation variables

    In Figure 12, the Loop value is set to 5, the RampUp value is set to 50, and the Loop value is set to 2.

Figure 13 shows the result of executing JMeter against one of the node servers deployed in a cloud application, based on the variable values that are configured in Figure 12:

Figure 13. Performance-test results
Screenshot showing the result running a JMeter test against one of the node servers deployed in cloud application, based on the configuration variables defined in Figure 12

Figure 13 displays a heavy load of simultaneous requests that go to the server to simulate performance-test conditions.


Extending functional JMeter scripts for reliability testing

Reliability testing is a way to ensure overall system stability by continuously running a set of scripts under specific sets of conditions. The result of reliability testing should be viewed in conjunction with the results of stress testing, functional testing, and testing of networking features.

As with performance testing, you can convert a test plan that executes and verifies a simple set of REST principles into a suitable set of reliability test scripts that determine the stability of any product or features running on the cloud instance.

An important aspect of reliability testing is to determine if failure occurs when you run a script continuously for many days. Check the Thread Group control panel's Forever check box, as shown in Figure 14 to enable the threads to run until the test fails or the script is stopped forcibly:

Figure 14. Reliability settings for JMeter script
Screen capture of a JMeter Thread Group control panel with the Forever box checked for the purpose of reliability testing

After configuring the reliability-test settings, run the test for several days and monitor the result continuously through a JMeter result graph or through the assertion options that JMeter provides.


Conclusion

This article showed you ways in which you can put JMeter to use for testing cloud-based applications effectively. The article is by no means an exhaustive discussion, and we encourage you to investigate other techniques for improving the implementation of automation tasks in JMeter. The JMeter Wiki (see Resources) is a great place to continue your explorations.


Download

DescriptionNameSize
Batch file for converting JSON payloadConversion_Script.zip2KB

Resources

Learn

Get products and technologies

Discuss

  • Get involved in the developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Cloud computing on developerWorks


  • Bluemix Developers Community

    Get samples, articles, product docs, and community resources to help build, deploy, and manage your cloud apps.

  • developerWorks Labs

    Experiment with new directions in software development.

  • DevOps Services

    Software development in the cloud. Register today to create a project.

  • Try SoftLayer Cloud

    Deploy public cloud instances in as few as 5 minutes. Try the SoftLayer public cloud instance for one month.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Cloud computing, Java technology
ArticleID=943499
ArticleTitle=Test cloud-based applications with Apache JMeter
publish-date=09092013