Optimize load handling for high-volume tests with IBM Rational Performance Tester

Best practices for your load testing environment

When using IBM® Rational® Performance Tester for testing diversified protocols and large volume load simulations, it is essential to optimize the performance of your testing machines and tools, as well as your network and infrastructure. In this article, you will discover best practices that you can adopt to enhance the load generation capability of Rational Performance Tester per machine by configuring both the testing tool and the operating system. You will also learn about techniques that you can use to alleviate trivial errors that occur during large volume load simulations.

Share:

Bharath Raj Keshavamurthy (bharathrajbk@in.ibm.com), Enterprise Solutions Performance Analyst, IBM

author photoBharath Raj works at IBM India Software Labs with the High Performance OnDemand Solutions (HiPODS) team as an Enterprise Solutions Performance Analyst. He started his career with IBM as a J2EE application developer using Web 2.0 technologies. He has worked on the performance of middleware, Java, and databases that are used for enterprise solutions in telecommunications, retail, and other industries. With an intensive background and experience in performance tuning, capacity planning, testing and engineering methodologies, Bharath Raj applies a holistic approach to the performance of an enterprise solution. He continues to use cutting edge technologies and imbibe them in enterprise web applications to achieve an optimized, faster and much more reliable solution.



24 May 2011

Also available in Chinese

This article is organized into three aspects of preparing your IBM® Rational® Performance Tester environment. In the first section, you will learn about methods for deploying the Rational Performance Tester load generation environment, and information about deployment patterns to avoid. In the second section, you will learn about tuning parameters that facilitate optimal performance test load generation and operating system performance. The last section includes information about what you can do before you launch a performance test run to run a successful performance test.

Rational Performance Tester deployment practices to adopt and to avoid

Consider the following suggestions for deploying your Rational Performance Tester load generation environment:

  • Use many agent machines with low-end configurations (in terms of number of CPUs and amount of memory) and have these machines generate load. If you use fewer agent machines with high-end configurations, then the operating system is limited by the total number of available sockets, processing parallelism, and limits on multi-threading. Therefore, use many agent machines with low-end configurations that are horizontally scaled out for performance testing, rather than vertically scaling agents on a few high-end machines.


    Note:
    Low-end configuration machines are those with a low number of CPUs and memory. However, the CPU speed should not be considered as low configuration. Recommended CPU speed for Rational Performance Tester agent machines is 2.33 GHz or greater.
Figure 1. Comparison of different topologies in designing the load generation environment
Topologies of load generation environment and their impacts

Larger view of Figure 1.

  • If the test agents are in a virtual environment, ensure that the logical partitions on the machines where the performance test agent software is installed have dedicated physical Ethernet controllers to support high volume network traffic.

    Important:
    Do not use logical host Ethernet adapters or virtualized Ethernet devices as networking interfaces for logical partitions that run performance test agent services on Rational Performance Tester agent logical partitions.
  • Ensure that all agents are in the same network as the controller and the performance test setup. Doing so ensures that there is minimal network latency and enables faster transfer of periodic statistics, test assets, and data between the host controller and agent machines.

Good topology decisions

The first two topologies are illustrated in figures 2 and 3. They are best practices that you can adopt for deploying Rational Performance Test environment

Topology 1: Best mechanism for performance testing the solution

During performance testing, you must host the load generation environment in the same network as that of the solution that is being tested so that there is minimal network latency between the load generation environment and the solution. After you minimize network latencies, it is not only easier to diagnose performance bottlenecks, but there is also a decrease in network errors and packet drops that can be difficult to troubleshoot, and might lead to transaction errors, despite the server operating correctly.

Figure 2. First topology for the performance test environment
Topology 1 of performance testing

Larger view of Figure 2.

Topology 2: Rational Performance Tester host controller is outside of the solution network, but the performance test agents are inside

This topology is almost the same as the previous one. Here, there can be communication problems between the host controller and its agents due to varying networks issues, such as the bandwidth of the network pipe and the network cards of the servers that host Rational Performance Tester. With that, the shortcoming of this mechanism is primarily seen in issues that prevent smooth execution of performance tests in the following situations:

  1. General execution of performance test runs or execution of performance test runs for large durations / Endurance performance tests which run for days together and with quick sampling rates
  2. There are many test agents involved for load generation purposes, approximately 25 test agents or more.

If the network connectivity and bandwidth between the host controller and its agents are not uniform, then this might cause communication errors (network packet losses) and further result in abnormal termination of performance tests. The following errors might occur:

  • "Workbench did not receive any message from Agent for more than xx milliseconds"
  • "Could not connect to Agent. Connection timed out"
Figure 3. Second topology for performance testing
Topology 2 of performance testing

Larger view of Figure 3.


Deployment patterns to avoid

Avoid the topology patterns illustrated in Figures 4 and 5 when you deploy your Rational Performance Tester environment.

Pattern 1: Rational Performance Tester agents are deployed on the same hardware as the solution components

Although this topology of performance testing guarantees minimal network latency between performance test agents and the solution, the operating system resources are shared between the solution and performance test environment. In this case, resource utilization analysis becomes very difficult due to extreme complexity involved in differentiating resource utilization of the same operating system between the solution environment and the performance test environment. Inaccurate analysis leads to inaccurate data for performance issues identification, analysis, and troubleshooting. Therefore, this topology is not suggested for performance test environment deployment.

Consider the following operating system resources:

  • Processor use
  • Memory use
  • TCP/IP sockets for both the solution component and agent processes
  • Network bandwidth, including the number of bytes sent and received per second for both the solution component and agent process

Note:
Deploying performance test agents on solution components also has an impact on the use of available TCP/IP sockets at the operating system layer in addition to process, memory and network bandwidth utilization.

Rational Performance Tester creates connections for the requests that it makes during performance test load generation. Every connection uses TCP/IP sockets. This number is limited to only 65,535 (as the TCP/IP protocol allows only 65,535 TCP/IP ports to be available at any operating system platform).

If the performance test agents use a large number of TCP/IP sockets, then the solution components have a shortage of socket connections, which creates a performance bottleneck and leads to poor performance.

Figure 4. First pattern to avoid for performance testing
First pattern to avoid for testing

Larger view of Figure 4.

Pattern 2: The entire load generation environment (host controller and test agents) is out of the solution network

Avoid this topology for testing the solution because it results in network errors, packet drops, and so forth. Thus this topology not only leads to disruptions and errors during performance testing, but also becomes difficult to drive the desired throughput and load on solution servers.

Figure 5. Second pattern to avoid for performance testing
Second pattern to avoid for testing

Larger view of Figure 5.


Performance tuning for your test environment

The following tuning parameters for Rational Performance Tester have been tried and tested successfully for generating a large user volume load of 35,000 virtual users with a user concurrency of 3,000 virtual user against a 3 tier web-based application that uses HTTP protocol.

You can apply these parameters when you test a web application against a user load between 1000 – 10,000 concurrent virtual users.

Disclaimer: These tuning parameters have been noted to provide 25 – 40% improvement in the network throughput behavior of the operating system and are subjected to the application workload characteristics as well. The performance improvement that was achieved varied by 5 – 10% on different platforms. The AIX platform had the most significant improvement.
These parameters are a reference point. You can map them against the application workloads and volumes that you are testing so that you get an appropriate set of values for your tests.
Table 1: Operating system tuning for optimal performance in load generation
OSParameterValueImplementation details
Windows MaxUserPort 65534 Edit the registry under HKEY_LOCAL_MACHINE > System > CurrentControlSet > Services > TCPIP > Parameters. All the parameters have to added as a DWORD and the values set as Decimal
TcpTimedWaitDelay 30
MaxFreeTcbs 72000
MaxHashTableSize 65535
TcpWindowSize 65535
EnableDynamicBacklog 1
MinimumDynamicBacklog 20
MaximumDynamicBacklog 1000
DynamicBacklogGrowthDelta 10
TcpAckFrequency 1
Linux net.ipv4.ip_local_port_range 2048 65000 Edit /etc/sysctl.conf
with the following parameters and save the file. Run the following command – sysctl –p
net.core.rmem_default 262144
net.core.wmem_default 262144
net.core.wmem_max 33554432
net.core.rmem_max 33554432
net.core.netdev_max_backlog 5000
net.ipv4.tcp_no_metrics_save 1
net.ipv4.tcp_rmem 4096 16777216 33554432
net.ipv4.tcp_wmem 4096 16777216 33554432
net.core.optmem_max 40960
AIX sb_max 6192000 no –p –o <parameter> = <value>
chdev -l sys0 -a maxuproc=<value>

clean_partial_conns 1
tcp_timewait 1
tcp_keepidle 600
tcp_keepinit 40
tcp_nodelayack 1
tcp_keepintvl 10
Maxuproc 8192
tcp_sendspace 4096000
tcp_recvspace 4096000
tcp_ephemeral_low 1024
rfc1323 1

Practices to consider before you launch a performance test

  • Ensure that the Rational Performance Test agent service has been started on the agent machines.
  • Ensure that there are no connections in CLOSE_WAIT or TIME_WAIT that use the netstat command before you start the tests from the agent.

    Note:
    TIME_WAIT connections happen when the server closes its connection, but the client waits for an acknowledgement signal to close the connection. Connections go to CLOSE_WAIT after exceeding the time duration under the TIME_WAIT state and the operating system must close this connection. Having a connection in any of these two states before you launch a performance test prevents this connection from being used by Rational Performance Tester during the test run, which suggests that no occurrences of these types of connections are there before the test. This ensures a clean network state before launch.
  • For agent machines on UNIX operating systems, look for a process named rpt_runner and stop any occurrence of it before you run a performance test. In Windows, look for Java in the Task Manager, and stop any occurrences of the same before you start your test.

    Note:
    Stopping the rpt_runner process in Linux, AIX, or Java in Windows does not stop the Rational Performance Tester agent service. The agent process is started at the beginning of every test launch. Hence, stopping these processes before launch of next test (if any exist after a test run completes) ensures that processes of the agent that is associated with previous tests are cleaned up from the operating system. These processes might continue to exist after test completion due to abrupt termination of performance test or errors during performance test.
  • On Microsoft Windows machines, enable the show heap status option on the host controller: Windows > Preferences > General > Show heap status. You can use this to keep track of memory utilization for Rational Performance Tester during tests. If the memory utilization increases, force the garbage collection manually by clicking on the garbage collection icon in the heap status bar. Knowing this helps you size the memory for the host controller machine.
  • Close the host controller workbench and launch it again to achieve minimal use of the JVM heap memory before you start your test. You must force garbage collection once to clean up any unreferenced objects in the Rational Performance Tester JVM heap memory before starting your test.
  • Delete all the contents in the following directories and restart the agent server before you run your test:
    • Temp directory of the operating system. (/tmp for Unix based operating systems, and C:\Documents and Settings\<User>\temp directory for Windows)
    • Deployment_root directories of agents
    • Shared class resources that are present in C:\Documents and Settings\user name\Local Settings\Application Data\javasharedresources
    • In certain cases, you must restart the operating system of the performance test agent machines.
  • Ensure that adequate disk space is available in temp directory and the file system that contains the deployment_root folder. Recommended minimum value for the temp directory is 10 GB, for the deployment directory is 5 GB.
  • When generating high loads and considering sync points, use many small configuration agent machines rather than fewer large configuration machines to get better results
  • If the test duration is long or involves high throughput generation of requests, then avoid statistics sampling at a fast rate. A high statistics sampling rate is overhead for the performance test tool and must be kept at an optimal rate. For example, in a 2-hour test run, a 15-second sampling interval is an optimal statistics sampling rate.
Figure 6. Statistics sampling rate in Rational Performance Tester
Sampling rate set to large values for large test durations
  • Set the error logs to minimum logging. Only errors and warning can be logged for primary test actions when running large volume tests to achieve optimal performance in load generation.
Figure 7. Logging level options in Rational Performance Tester
Logging levels can be set to low during main performance tests

Resources

Learn

Get products and technologies

Discuss

  • Join the Performance Testing forum, where you can share you questions and knowledge about IBM performance testing products.
  • Share your knowledge and help others who use Rational software by writing a developerWorks article. You'll get worldwide exposure, RSS syndication, a byline and a bio, and the benefit of professional editing and production on the developerWorks Rational website.
  • Follow Rational software on Facebook and Twitter (@ibmrational), and add your comments and requests.
  • Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
  • Connect with others who share your interests by joining the developerWorks community and responding to the developer-driven blogs.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=659931
ArticleTitle=Optimize load handling for high-volume tests with IBM Rational Performance Tester
publish-date=05242011