Tips for using the Load Generation Agent in performance testing
A guide to the agent introduced in Rational Performance Tester Version 8.3
Highlights of Load Generation Agent features and benefits
A new Load Generation Agent is a vital part of IBM® Rational® Performance Tester Rational Performance Tester Version 8.3.0. As a replacement for the Agent Controller, the Load Generation Agent offers the following features and benefits:
- Very strong execution process launch reliability for agents. The Load Generation Agent is designed specifically to work with the Rational Performance Tester workbench in the launch and control of load tests.
- Agents poll the supported workbenches, looking for schedule execution work, rather than the workbench having to establish connections with agents. This scheme provides instant assessment of agent readiness and rapid delivery of launch commands.
- With the Load Generation Agent, the workbench can launch agents in parallel, thus greatly reducing overall launch time for large load testing operations involving many agents.
- The Load Generation Agent supports intelligent test asset deployment. It pulls only assets from the workbench that it does not have or that have changed. This feature results in faster launch times for subsequent executions.
- The Load Generation Agent uses an HTTP server and the HTTP protocol between the workbench and agents. Loss of a connection simply means that another is established with no harm to the overall execution, thereby providing robustness during multiple-day executions.
- The Load Generation Agent is more firewall friendly. It reverses the contact initiation so that agents connect to the workbench rather than vice versa. The use of an HTTP server within the workbench means that you have only one port to configure with the firewall rather than the many ports on both ends that the Agent Controller required.
- The ability to secure communications between the workbench and agents by using TLS/SSL is much easier with the Load Generation Agent. You can indicate your preference for secure communication from a single location on the workbench.
Status of the Agent Controller
The Agent Controller is still provided with Version 8.3.0. As an optional part of the installation, you can install the Agent Controller to support these features:
- Response time breakdown
- Service-oriented architecture (SOA) stubs
The Agent Controller cannot be used for load generation with the Rational Performance Tester 8.3.0 workbench.
Although not a typical customer configuration, installing both the Load Generation Agent and the Agent Controller is a supported configuration.
Load Generation Agent architecture
Unlike the Agent Controller architecture, the workbench does not make connections to the agents when a schedule is to be executed. The ports that had to be opened by firewalls for those connections and the connections that the agents made back to the workbench on ephemeral ports are no longer necessary.
With the Load Generation Agent, the agent machines poll for work by sending HTTP requests to the Rational Performance Tester server that is running with the workbench. When you press the Run button to start executing the schedule, the next time that a participating agent contacts the workbench, it responds to the agent by indicating that its participation is needed.
The agent then launches the execution engine as the Agent Controller would have. The execution engine contacts the Rational Performance Tester server, and the usual exchange of commands begins execution of the schedule.
In Figure 1, the Workbench box represents the Eclipse user interface of Rational Performance Tester, which is a single process and one Java virtual machine (JVM). Version 8.3.0 also includes the Jetty web server to support communication between the workbench and agents. Within the workbench, each agent has a load test executor for command, control, and state management of each agent.
Figure 1. Load Generation Agent architecture
On a single agent, there is the Majordomo service, which is one Java virtual machine that polls supported workbenches for schedule execution work. Each Domo box in the diagram in Figure 1 is a single Java virtual machine that includes the execution engine. Typically, there is only one Java process running on the agent during schedule execution. That single Java process is the Domo and engine process. The diagram shows that it is possible for one agent machine to support multiple execution engines. For garbage collection considerations, if you have a powerful 64-bit agent with a lot of memory, you might find it more efficient to run with multiple engine processes of modest heap size rather than one single process with an enormous heap size.
The Load Generation Agent is simple and compact. The Domo component downloads from the workbench the assets needed for schedule execution, including the execution engine. On subsequent executions, Domo downloads only assets that have changed since the previous execution.
The installation of the Load Generation Agent is similar to the installation of the Agent Controller. In fact, both features are available when you install the Rational Performance Tester Agent for Version 8.3.0.
Workbench local execution
Although installing the Rational Performance Tester Agent product with the workbench is a supported configuration, it is not required. As with previous releases, it is possible to execute schedules locally on the same machine that the workbench is running on without installing the Agent. The workbench will automatically spawn an instantiation of the Load Generation Agent to support local execution during the life of the execution of the schedule.
By default, an Agent installation installs the Load Generation Agent but not the Agent Controller.
Figure 2. Installation options
Configure the Load Generation Agent
There are two Load Generation Agent configuration parameters that you must enter during installation:
- Workbench host name
- Enter one and only one host name that this agent will poll for work.
- Workbench port
- Enter the port number supported by the workbench. The default port number is 7080.
Figure 3. Configuration options
The Load Generation Agent has several options that you can configure from the workbench. To configure the options, select Window > Preferences > Test > Server.
Figure 4. Server configuration options
Select the unsecure and secure ports that the server will use for communication with agents. By default, communication is unsecure, so the unsecure port is the only one that might need to be changed. The port does not have to be changed unless it is already in use by another product running on the workbench. If you change the workbench port, you must also change the agent configuration to use the same port.
The secure port is used only when running with the option to encrypt messages between the workbench and an agent. The secure port number is configured only on the workbench. There is no configuration change required on the agent.
Secure communication using TLS/SSL
Check Workbench and agent communication is encrypted using TLS/SSL to have all messages and execution data encrypted for exchanges between the workbench and agents. All schedules executed from this workbench will use SSL. The workbench will tell the agent what port to use for secure communication via the unsecure port before starting schedule execution.
Delete deployment directory
Check Delete deployment directory on the agent after execution to have the contents of the deployment directory removed from the agent after the schedule is executed. The deployment directory contains all of the test assets that the agent requires for schedule execution. Selecting this option defeats the cached test assets launch time improvement feature, which makes subsequent schedule launch times no shorter than the first schedule launch. The feature is of interest if you are concerned about disk space accumulating on the agent. If you are executing many unique schedules, the test artifacts of each schedule are left on the agent machine after execution is finished.
Modification of agent configurations
After a workbench and many agents are installed, it might become necessary to make configuration changes. This section describes how to modify an agent configuration after installation.
Add support for another workstation to an agent
If you recall from the Installation section, it is possible to specify the host name of only one agent when you configure the Load Generation Agent. So what happens if you add another workstation to the lab and want an agent to support it? The best approach is to use the Agent Status button on an already supported workbench to add the new workbench to the agent's configuration.
Figure 5. Agent status button
When you click the Agent Status button, a pop-up window shows the status of agents in contact with this workbench. To have an agent support another workbench in the lab, check the box beside the agent and click Share Agent with New Workbench.
As Figure 6 shows, you will then be prompted to enter the host name and port number for the workbench to add to the agent's configuration.
Figure 6. Share an agent with a new workbench
After clicking OK, the request to add the new workbench will be delivered to the agent. Within a short time, the agent should begin polling the new workbench for schedule execution work.
Load Generation Agent configuration file
After installation of the Rational Performance Tester Agent with the Load Generation Agent feature, there will be an agent configuration file, majordomo.config, stored on the agent. You can accomplish these tasks by editing, copying, or replacing the configuration file:
- Change the host name of a workbench that the agent polls for work.
- Change the port that the agent will use when contacting the workbench.
- Add one or more workbenches for the agent to support.
- Replace the entire configuration by using a copy command or gesture to replace the existing majordomo.config file with a new one.
- Push a new configuration to many agents by using a remote copy command, such as this one:
scp /opt/IBM/SDP/Majordomo/majordomo.config root@chickmagnet:/opt/IBM/SDP/Majordomo
A few seconds after you make the change, the Load Generation Agent will refresh its configuration based on the contents of the file. There is no need to stop and restart the agent service.
The contents of the configuration file are in XML, and the format is intuitive. Listing 1 is an example of a configuration file for an agent that is supporting two workbenches.
Listing 1. Configuration file contents
<MajordomoConfig xmlns="http://www.example.org/MajordomoConfiguration"> <debug>false</debug> <canonicalHostName></canonicalHostName> <ipAddress></ipAddress> <pollInterval>10</pollInterval> <workbenches> <hostName>kmdesk.raleigh.ibm.com</hostName> <port>7081</port> </workbenches> </MajordomoConfig>
The Load Generation Agent configuration file is typically stored in one of these locations:
- Windows 64-bit
- C:\Program Files\IBM\SDP\Majordomo\majordomo.config
- Windows 32-bit
- C:\Program Files (x86)\IBM\SDP\Majordomo\majordomo.config
Agent Status update feature
As mentioned previously, the workbench contains an Agent Status button. When selected, it is possible to see what agents are polling the workbench for work and whether they are ready to participate in schedule execution.
Figure 7. Agent Status view
The agent can display these states:
- The agent is ready to execute a schedule.
- The agent is currently executing a schedule. While it is busy, it is possible for an agent to initiate and begin executing multiple additional schedules at the same time.
- Lost Contact
- The agent is no longer polling this workbench.
Share Agent with New Workbench option
If you select one or more agents and click Share Agent with New Workbench, you can specify the host name and port of a new workbench that you want the agent to poll for work in addition to the this workbench.
Disconnect Agent from this Workbench option
If you select one or more agents and click Disconnect Agent from this Workbench, the host name for this workbench will be removed from the agents' configuration files, and they will no longer poll this workbench for work.
This section describes some common problem scenarios and provides suggestions for finding a solution.
Port In Use
When you try to run a schedule, you get the error message shown in Figure 8.
Figure 8. Check agents fails due to port in use
Some software that is running on the workbench is already using port 7080, the default unsecure port that Rational Performance Tester uses for communication with agents.
Select Window > Preferences > Test > Server, and then choose a different port, such as 7081. All agent configuration files must also specify port 7081. There is no need to restart the workbench or the agent for these changes to take effect.
It is possible that you no longer need the software that is using port 7080 on this machine. If so, stop or uninstall that software.
Check Agents Failed
When you try to run a schedule, you get the error message in Figure 9.
Figure 9. Lost contact with agent
One or more of the agents listed as participants in executing this schedule are not in active contact with this workbench. You can use the Agent Status button to view status before executing schedules to determine the likelihood of encountering the Check Agents Failed message.
The Rational Performance Tester Agent is not installed. Use the IBM Installation Manager View Installed Packages feature to ensure that the Agent product is installed.
The Load Generation Agent is installed, but the agent service, Majordomo, is not running.
Look for the MajordomoService process.
Use the Task Manager to verify that the agent is running. If it is not running, use this command:
cd "Program Files\SDP\IBM\Majordomo"
Look for the Majordomo process. Use the
pscommand to verify that the agent is running. If it is not running, use this command:
Enter these commands by using a command prompt or a shell. Substitute paths based on your installation.
Majordomo is running on the agent, but it is not polling the workbench for work. Verify that the agent can resolve the host name through Domain Name Services as specified in the configuration file. Use the
ping command, and specify the workbench host name exactly as it is entered in the majordomo.config file. If the ping command fails using the host name but you can successfully ping the workbench by using its IP address, a workaround could be to specify the workbench IP address in the majordomo.config file rather than the host name.
If Majordomo is running and the workbench can be pinged from the agent, verify that there is not a firewall blocking access. From a command prompt or shell, use a Telnet client to connect to the server, using a command similar to this example:
telnet WORKBENCH_HOST_NAME 7080
Type a few characters, and then press Enter. You should see the screen output shown in Figure 10: Connection to host lost.
Figure 10. Telnet session with server
If the client is unable to connect or you do not receive a response like the one above, try temporarily disabling firewalls to confirm whether that is the source of the problem.
Windows 7 does not enable the Telnet client by default, but you can enable it by selecting Start > Control Panel > Programs > Programs and Features > Turn Windows feature on or off.
It is also possible to receive the Check Agents Failed dialog if the agent configuration contains workbenches that are not running, workbenches where Rational Performance Tester has been uninstalled, or workbenches that are shut down. An agent that encounters communication problems might get delayed in its polling work long enough that a supported workbench might conclude that agent is out of contact. If you suspect that has happened, remove any stale workbench information from each agent's configuration file.
Process on driver terminated
When you try to run a schedule, you get the error message shown in Figure 11.
Figure 11. Execution process terminated
Shortly after Majordomo launched the engine execution process on the agent, that engine process terminated unexpectedly. There might be messages explaining the reason for the failure in the workbench Error Log. You can open that log by selecting Window > Show View > Error Log. At the default log level of WARNING, all messages written to the stderr output stream by the native Java program on the agent will be sent to the workbench and stored in the Error Log.
The execution process will terminate at startup if incorrect arguments are passed to the Java virtual machine (JVM). Figure 12 is an example of the contents of the Error Log if that happens.
Figure 12. Error log with unrecognized virtual machine option
Notice the JVM error notice about an unrecognized command line option. In this case, examine any arguments specified in the RPT_VMARGS general property for the location, and ensure that there are no typos or bogus command line options.
Another example of a problem that produces the process terminated message is when the wrong JVM was used somehow when launching the engine execution process. Figure 13 shows an example of what happens if the agent engine execution is launched with Java 1.6 rather than the required Version 1.7 of IBM Java Virtual Machine (JVM).
Figure 13. Error log showing that the JVM version is incorrect
The appearance of UnsupportedClassVersionError is a clue that the JVM on the agent is not the required Version 1.7. This problem could occur if the Rational Performance Tester Agent was installed in a non-default manner or if the installation path contained unexpected characters. Here are suggested workarounds:
- Uninstall and reinstall Rational Performance Tester. Use default values for where to install the agent.
- Use the RPT_JAVA environment variable to explicitly specify the Java executable file that the agent should launch for the execution engine. For example:
RPT_JAVA=C:\Program Files\IBM\SDP\jdk\bin\java.exeThe java.exe file to use should be the IBM JVM 1.7 binary installed with the Rational Performance Tester Agent. After you set this environment variable, stop and restart the Majordomo service. From a command prompt or shell, enter these commands:
cd "Program Files/IBM/SDP/Majordomo"
The Rational Performance Tester Agent must be installed by a user with Administrator privileges on Windows and as root on UNIX.
Multiple engines on one agent machine
In previous versions of Rational Performance Tester, you could use host aliases to run multiple execution engines on one agent machine. The only restriction is that the location host names and deployment directories must be unique. This trick also works with the new Load Generation Agent. Rational Performance Tester does not limit the number of engines executing on one agent.
There are no changes related to locations and the Load Generation Agent. If you are running the software using a remote agent, a location must be created and added to the schedule as in previous releases of Rational Performance Tester.
Ports with multiple workbenches
If one agent is supporting multiple workbenches, it is fine for the port number to be the same for every workbench listed in the configuration file.
Previous releases of Rational Performance Tester required one successful execution of a schedule in order to set the RPT_DEFAULT_MEMORY_SIZE property that specifies the maximum Java heap for a location. This requirement remains for the new Load Generation Agent. On the first execution, the max heap will be 256 megabytes (256 MB), which is an insufficient heap amount that will ensure failure for a large load test.
The heart of the Load Generation Agent is called Majordomo.
- On Windows, it runs as a service called MajordomoService.exe. Because it is a service, MajordomoService.exe should be running automatically after reboot on Windows. MajordomoService will log informational messages at startup in the system Temp directory, which is usually C:\Windows\Temp\majordomoservice.log. If there is a problem, perhaps related to finding the IBM JVM, there might be messages in that log that could help determine the cause of the problem.
- On UNIX, it runs as Majordomo. The UNIX systems require your intervention if you want Majordomo running after a reboot.
Majordomo debug flag
You might have noticed that there is a debug flag in the majordomo.config file. If set to true, the Majordomo process and any engine processes launched create log files that can be useful for debugging Load Generation Agent launch and problems that occur early in schedule execution. Examples of these files:
Similar files can be found on UNIX systems stored under %TMP%/USERNAME.
- Check the Rational Performance Tester page on IBM developerWorks for links to technical articles and browse the user assistance in the Rational Performance Tester 8.3 Information Center. Also see the Rational Performance Tester product overview page.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, anytime, and many of the Getting Started ones are free.
- Download and evaluate IBM Rational Performance Tester Version 8.3.
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.