This article is organized into three aspects of preparing your IBM® Rational® Performance Tester environment. In the first section, you will learn about methods for deploying the Rational Performance Tester load generation environment, and information about deployment patterns to avoid. In the second section, you will learn about tuning parameters that facilitate optimal performance test load generation and operating system performance. The last section includes information about what you can do before you launch a performance test run to run a successful performance test.

Rational Performance Tester deployment practices to adopt and to avoid

Consider the following suggestions for deploying your Rational Performance Tester load generation environment:

• Use many agent machines with low-end configurations (in terms of number of CPUs and amount of memory) and have these machines generate load. If you use fewer agent machines with high-end configurations, then the operating system is limited by the total number of available sockets, processing parallelism, and limits on multi-threading. Therefore, use many agent machines with low-end configurations that are horizontally scaled out for performance testing, rather than vertically scaling agents on a few high-end machines.
  
  
  Note:
  Low-end configuration machines are those with a low number of CPUs and memory. However, the CPU speed should not be considered as low configuration. Recommended CPU speed for Rational Performance Tester agent machines is 2.33 GHz or greater.

Larger view of Figure 1.

• If the test agents are in a virtual environment, ensure that the logical partitions on the machines where the performance test agent software is installed have dedicated physical Ethernet controllers to support high volume network traffic.
  
  Important:
  Do not use logical host Ethernet adapters or virtualized Ethernet devices as networking interfaces for logical partitions that run performance test agent services on Rational Performance Tester agent logical partitions.
  
  
• Ensure that all agents are in the same network as the controller and the performance test setup. Doing so ensures that there is minimal network latency and enables faster transfer of periodic statistics, test assets, and data between the host controller and agent machines.

Good topology decisions

The first two topologies are illustrated in figures 2 and 3. They are best practices that you can adopt for deploying Rational Performance Test environment

Topology 1: Best mechanism for performance testing the solution

During performance testing, you must host the load generation environment in the same network as that of the solution that is being tested so that there is minimal network latency between the load generation environment and the solution. After you minimize network latencies, it is not only easier to diagnose performance bottlenecks, but there is also a decrease in network errors and packet drops that can be difficult to troubleshoot, and might lead to transaction errors, despite the server operating correctly.

Larger view of Figure 2.

Topology 2: Rational Performance Tester host controller is outside of the solution network, but the performance test agents are inside

This topology is almost the same as the previous one. Here, there can be communication problems between the host controller and its agents due to varying networks issues, such as the bandwidth of the network pipe and the network cards of the servers that host Rational Performance Tester. With that, the shortcoming of this mechanism is primarily seen in issues that prevent smooth execution of performance tests in the following situations:

1. General execution of performance test runs or execution of performance test runs for large durations / Endurance performance tests which run for days together and with quick sampling rates
   
   
2. There are many test agents involved for load generation purposes, approximately 25 test agents or more.

If the network connectivity and bandwidth between the host controller and its agents are not uniform, then this might cause communication errors (network packet losses) and further result in abnormal termination of performance tests. The following errors might occur:

• "Workbench did not receive any message from Agent for more than xx milliseconds"
  
  
• "Could not connect to Agent. Connection timed out"

Larger view of Figure 3.

Back to top

Deployment patterns to avoid

Avoid the topology patterns illustrated in Figures 4 and 5 when you deploy your Rational Performance Tester environment.

Pattern 1: Rational Performance Tester agents are deployed on the same hardware as the solution components

Although this topology of performance testing guarantees minimal network latency between performance test agents and the solution, the operating system resources are shared between the solution and performance test environment. In this case, resource utilization analysis becomes very difficult due to extreme complexity involved in differentiating resource utilization of the same operating system between the solution environment and the performance test environment. Inaccurate analysis leads to inaccurate data for performance issues identification, analysis, and troubleshooting. Therefore, this topology is not suggested for performance test environment deployment.

Consider the following operating system resources:

• Processor use
  
  
• Memory use
  
  
• TCP/IP sockets for both the solution component and agent processes
  
  
• Network bandwidth, including the number of bytes sent and received per second for both the solution component and agent process

Note:
Deploying performance test agents on solution components also has an impact on the use of available TCP/IP sockets at the operating system layer in addition to process, memory and network bandwidth utilization.

Larger view of Figure 4.

Pattern 2: The entire load generation environment (host controller and test agents) is out of the solution network

Avoid this topology for testing the solution because it results in network errors, packet drops, and so forth. Thus this topology not only leads to disruptions and errors during performance testing, but also becomes difficult to drive the desired throughput and load on solution servers.

Larger view of Figure 5.

Back to top

Performance tuning for your test environment

The following tuning parameters for Rational Performance Tester have been tried and tested successfully for generating a large user volume load of 35,000 virtual users with a user concurrency of 3,000 virtual user against a 3 tier web-based application that uses HTTP protocol.

You can apply these parameters when you test a web application against a user load between 1000 – 10,000 concurrent virtual users.

Disclaimer: These tuning parameters have been noted to provide 25 – 40% improvement in the network throughput behavior of the operating system and are subjected to the application workload characteristics as well. The performance improvement that was achieved varied by 5 – 10% on different platforms. The AIX platform had the most significant improvement.
These parameters are a reference point. You can map them against the application workloads and volumes that you are testing so that you get an appropriate set of values for your tests.

Back to top

Practices to consider before you launch a performance test

• Ensure that the Rational Performance Test agent service has been started on the agent machines.
  
  
• Ensure that there are no connections in CLOSE_WAIT or TIME_WAIT that use the netstat command before you start the tests from the agent.
  
  Note:
  TIME_WAIT connections happen when the server closes its connection, but the client waits for an acknowledgement signal to close the connection. Connections go to CLOSE_WAIT after exceeding the time duration under the TIME_WAIT state and the operating system must close this connection. Having a connection in any of these two states before you launch a performance test prevents this connection from being used by Rational Performance Tester during the test run, which suggests that no occurrences of these types of connections are there before the test. This ensures a clean network state before launch.
  
  
• For agent machines on UNIX operating systems, look for a process named rpt_runner and stop any occurrence of it before you run a performance test. In Windows, look for Java in the Task Manager, and stop any occurrences of the same before you start your test.
  
  Note:
  Stopping the rpt_runner process in Linux, AIX, or Java in Windows does not stop the Rational Performance Tester agent service. The agent process is started at the beginning of every test launch. Hence, stopping these processes before launch of next test (if any exist after a test run completes) ensures that processes of the agent that is associated with previous tests are cleaned up from the operating system. These processes might continue to exist after test completion due to abrupt termination of performance test or errors during performance test.
  
  
• On Microsoft Windows machines, enable the show heap status option on the host controller: Windows > Preferences > General > Show heap status. You can use this to keep track of memory utilization for Rational Performance Tester during tests. If the memory utilization increases, force the garbage collection manually by clicking on the garbage collection icon in the heap status bar. Knowing this helps you size the memory for the host controller machine.
  
  
• Close the host controller workbench and launch it again to achieve minimal use of the JVM heap memory before you start your test. You must force garbage collection once to clean up any unreferenced objects in the Rational Performance Tester JVM heap memory before starting your test.
  
  
• Delete all the contents in the following directories and restart the agent server before you run your test:
  • Temp directory of the operating system. (/tmp for Unix based operating systems, and C:\Documents and Settings\<User>\temp directory for Windows)
    
    
  • Deployment_root directories of agents
    
    
  • Shared class resources that are present in C:\Documents and Settings\user name\Local Settings\Application Data\javasharedresources
    
    
  • In certain cases, you must restart the operating system of the performance test agent machines.
    
    
• Ensure that adequate disk space is available in temp directory and the file system that contains the deployment_root folder. Recommended minimum value for the temp directory is 10 GB, for the deployment directory is 5 GB.
  
  
• When generating high loads and considering sync points, use many small configuration agent machines rather than fewer large configuration machines to get better results
  
  
• If the test duration is long or involves high throughput generation of requests, then avoid statistics sampling at a fast rate. A high statistics sampling rate is overhead for the performance test tool and must be kept at an optimal rate. For example, in a 2-hour test run, a 15-second sampling interval is an optimal statistics sampling rate.

• Set the error logs to minimum logging. Only errors and warning can be logged for primary test actions when running large volume tests to achieve optimal performance in load generation.

Resources

Learn

• Find out more on the Rational Performance Tester product overview page. Then explore the Rational Performance Tester page on IBM® developerWorks® for links to technical articles and browse the user assistance in the Rational Performance Tester Information Center.
  
  
• Visit the Rational software area on developerWorks for technical resources and best practices for Rational Software Delivery Platform products.
  
  
• Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
  • Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends.
    
    
  • Watch developerWorks on-demand demos, ranging from product installation and setup demos for beginners to advanced functionality for experienced developers.
    
    
  
  
• Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, any time, and many of the "Getting Started" ones are free.
  
  

Get products and technologies

• Download the trial version of IBM Rational Performance Tester.
  
  
• Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.
  
  

Discuss

• Join the Performance Testing forum, where you can share you questions and knowledge about IBM performance testing products.
  
  
• Share your knowledge and help others who use Rational software by writing a developerWorks article. You'll get worldwide exposure, RSS syndication, a byline and a bio, and the benefit of professional editing and production on the developerWorks Rational website.
  
  
• Follow Rational software on Facebook and Twitter (@ibmrational), and add your comments and requests.
  
  
• Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
  
  
• Connect with others who share your interests by joining the developerWorks community and responding to the developer-driven blogs.
  
  

About the author

Bharath Raj works at IBM India Software Labs with the High Performance OnDemand Solutions (HiPODS) team as an Enterprise Solutions Performance Analyst. He started his career with IBM as a J2EE application developer using Web 2.0 technologies. He has worked on the performance of middleware, Java, and databases that are used for enterprise solutions in telecommunications, retail, and other industries. With an intensive background and experience in performance tuning, capacity planning, testing and engineering methodologies, Bharath Raj applies a holistic approach to the performance of an enterprise solution. He continues to use cutting edge technologies and imbibe them in enterprise web applications to achieve an optimized, faster and much more reliable solution.

Rate this article

Comments

Back to top

Table of contents

• Rational Performance Tester deployment practices to adopt and to avoid
• Deployment patterns to avoid
• Performance tuning for your test environment
• Practices to consider before you launch a performance test
• Resources
• About the author
• Comments

Next steps from IBM

• Try: Rational Performance Tester
• Community: Forum: Rational Performance Testing
• Buy: Rational Performance Tester

Dig deeper into Rational on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers