Optimize load handling for high-volume tests with IBM Rational Performance Tester
Best practices for your load testing environment
This article is organized into three aspects of preparing your IBM® Rational® Performance Tester environment. In the first section, you will learn about methods for deploying the Rational Performance Tester load generation environment, and information about deployment patterns to avoid. In the second section, you will learn about tuning parameters that facilitate optimal performance test load generation and operating system performance. The last section includes information about what you can do before you launch a performance test run to run a successful performance test.
Rational Performance Tester deployment practices to adopt and to avoid
Consider the following suggestions for deploying your Rational Performance Tester load generation environment:
- Use many agent machines with low-end configurations (in terms of
number of CPUs and amount of memory) and have these machines generate
load. If you use fewer agent machines with high-end configurations,
then the operating system is limited by the total number of available
sockets, processing parallelism, and limits on multi-threading.
Therefore, use many agent machines with low-end configurations that
are horizontally scaled out for performance testing, rather than
vertically scaling agents on a few high-end machines.
Low-end configuration machines are those with a low number of CPUs and memory. However, the CPU speed should not be considered as low configuration. Recommended CPU speed for Rational Performance Tester agent machines is 2.33 GHz or greater.
Figure 1. Comparison of different topologies in designing the load generation environment
- If the test agents are in a virtual environment, ensure that the
logical partitions on the machines where the performance test agent
software is installed have dedicated physical Ethernet controllers to
support high volume network traffic.
Do not use logical host Ethernet adapters or virtualized Ethernet devices as networking interfaces for logical partitions that run performance test agent services on Rational Performance Tester agent logical partitions.
- Ensure that all agents are in the same network as the controller and the performance test setup. Doing so ensures that there is minimal network latency and enables faster transfer of periodic statistics, test assets, and data between the host controller and agent machines.
Good topology decisions
The first two topologies are illustrated in figures 2 and 3. They are best practices that you can adopt for deploying Rational Performance Test environment
Topology 1: Best mechanism for performance testing the solution
During performance testing, you must host the load generation environment in the same network as that of the solution that is being tested so that there is minimal network latency between the load generation environment and the solution. After you minimize network latencies, it is not only easier to diagnose performance bottlenecks, but there is also a decrease in network errors and packet drops that can be difficult to troubleshoot, and might lead to transaction errors, despite the server operating correctly.
Figure 2. First topology for the performance test environment
Topology 2: Rational Performance Tester host controller is outside of the solution network, but the performance test agents are inside
This topology is almost the same as the previous one. Here, there can be communication problems between the host controller and its agents due to varying networks issues, such as the bandwidth of the network pipe and the network cards of the servers that host Rational Performance Tester. With that, the shortcoming of this mechanism is primarily seen in issues that prevent smooth execution of performance tests in the following situations:
- General execution of performance test runs or execution of performance test runs for large durations / Endurance performance tests which run for days together and with quick sampling rates
- There are many test agents involved for load generation purposes, approximately 25 test agents or more.
If the network connectivity and bandwidth between the host controller and its agents are not uniform, then this might cause communication errors (network packet losses) and further result in abnormal termination of performance tests. The following errors might occur:
- "Workbench did not receive any message from Agent for more than
- "Could not connect to Agent. Connection timed out"
Figure 3. Second topology for performance testing
Deployment patterns to avoid
Avoid the topology patterns illustrated in Figures 4 and 5 when you deploy your Rational Performance Tester environment.
Pattern 1: Rational Performance Tester agents are deployed on the same hardware as the solution components
Although this topology of performance testing guarantees minimal network latency between performance test agents and the solution, the operating system resources are shared between the solution and performance test environment. In this case, resource utilization analysis becomes very difficult due to extreme complexity involved in differentiating resource utilization of the same operating system between the solution environment and the performance test environment. Inaccurate analysis leads to inaccurate data for performance issues identification, analysis, and troubleshooting. Therefore, this topology is not suggested for performance test environment deployment.
Consider the following operating system resources:
- Processor use
- Memory use
- TCP/IP sockets for both the solution component and agent processes
- Network bandwidth, including the number of bytes sent and received per second for both the solution component and agent process
Deploying performance test agents on solution components also has an impact on the use of available TCP/IP sockets at the operating system layer in addition to process, memory and network bandwidth utilization.
Rational Performance Tester creates connections for the requests that it makes during performance test load generation. Every connection uses TCP/IP sockets. This number is limited to only 65,535 (as the TCP/IP protocol allows only 65,535 TCP/IP ports to be available at any operating system platform).
If the performance test agents use a large number of TCP/IP sockets, then the solution components have a shortage of socket connections, which creates a performance bottleneck and leads to poor performance.
Figure 4. First pattern to avoid for performance testing
Pattern 2: The entire load generation environment (host controller and test agents) is out of the solution network
Avoid this topology for testing the solution because it results in network errors, packet drops, and so forth. Thus this topology not only leads to disruptions and errors during performance testing, but also becomes difficult to drive the desired throughput and load on solution servers.
Figure 5. Second pattern to avoid for performance testing
Performance tuning for your test environment
The following tuning parameters for Rational Performance Tester have been tried and tested successfully for generating a large user volume load of 35,000 virtual users with a user concurrency of 3,000 virtual user against a 3 tier web-based application that uses HTTP protocol.
You can apply these parameters when you test a web application against a user load between 1000 – 10,000 concurrent virtual users.
Disclaimer: These tuning parameters have been noted to provide 25 – 40% improvement in the network throughput behavior of the operating system and are subjected to the application workload characteristics as well. The performance improvement that was achieved varied by 5 – 10% on different platforms. The AIX platform had the most significant improvement.
These parameters are a reference point. You can map them against the application workloads and volumes that you are testing so that you get an appropriate set of values for your tests.
Table 1: Operating system tuning for optimal performance in load generation
|Windows||MaxUserPort||65534||Edit the registry under HKEY_LOCAL_MACHINE > System > CurrentControlSet > Services > TCPIP > Parameters. All the parameters have to added as a DWORD and the values set as Decimal|
|Linux||net.ipv4.ip_local_port_range||2048 65000|| Edit /etc/sysctl.conf|
with the following parameters and save the file. Run the following command
|net.ipv4.tcp_rmem||4096 16777216 33554432|
|net.ipv4.tcp_wmem||4096 16777216 33554432|
Practices to consider before you launch a performance test
- Ensure that the Rational Performance Test agent service has been started on the agent machines.
- Ensure that there are no connections in CLOSE_WAIT or TIME_WAIT that
netstatcommand before you start the tests from the agent.
TIME_WAIT connections happen when the server closes its connection, but the client waits for an acknowledgement signal to close the connection. Connections go to CLOSE_WAIT after exceeding the time duration under the TIME_WAIT state and the operating system must close this connection. Having a connection in any of these two states before you launch a performance test prevents this connection from being used by Rational Performance Tester during the test run, which suggests that no occurrences of these types of connections are there before the test. This ensures a clean network state before launch.
- For agent machines on UNIX operating systems, look for a process
named rpt_runner and stop any occurrence of it before you run a
performance test. In Windows, look for Java in the Task Manager, and
stop any occurrences of the same before you start your test.
rpt_runnerprocess in Linux, AIX, or Java in Windows does not stop the Rational Performance Tester agent service. The agent process is started at the beginning of every test launch. Hence, stopping these processes before launch of next test (if any exist after a test run completes) ensures that processes of the agent that is associated with previous tests are cleaned up from the operating system. These processes might continue to exist after test completion due to abrupt termination of performance test or errors during performance test.
- On Microsoft Windows machines, enable the show heap status option on the host controller: Windows > Preferences > General > Show heap status. You can use this to keep track of memory utilization for Rational Performance Tester during tests. If the memory utilization increases, force the garbage collection manually by clicking on the garbage collection icon in the heap status bar. Knowing this helps you size the memory for the host controller machine.
- Close the host controller workbench and launch it again to achieve minimal use of the JVM heap memory before you start your test. You must force garbage collection once to clean up any unreferenced objects in the Rational Performance Tester JVM heap memory before starting your test.
- Delete all the contents in the following directories and restart the
agent server before you run your test:
- Temp directory of the operating system. (/tmp for Unix based operating systems, and C:\Documents and Settings\<User>\temp directory for Windows)
- Deployment_root directories of agents
- Shared class resources that are present in C:\Documents and Settings\user name\Local Settings\Application Data\javasharedresources
- In certain cases, you must restart the operating system of the performance test agent machines.
- Ensure that adequate disk space is available in temp directory and the file system that contains the deployment_root folder. Recommended minimum value for the temp directory is 10 GB, for the deployment directory is 5 GB.
- When generating high loads and considering sync points, use many small configuration agent machines rather than fewer large configuration machines to get better results
- If the test duration is long or involves high throughput generation of requests, then avoid statistics sampling at a fast rate. A high statistics sampling rate is overhead for the performance test tool and must be kept at an optimal rate. For example, in a 2-hour test run, a 15-second sampling interval is an optimal statistics sampling rate.
Figure 6. Statistics sampling rate in Rational Performance Tester
- Set the error logs to minimum logging. Only errors and warning can be logged for primary test actions when running large volume tests to achieve optimal performance in load generation.