Tips for using the Load Generation Agent in performance testing

A guide to the agent introduced in Rational Performance Tester Version 8.3

Kevin Mooney and Richard Ptasnik provide information about the benefits of the IBM® Rational® Performance Tester Load Generation Agent and insight into its architecture. Because correct configuration of the workbench and Load Generation Agent is critical, they also cover common configuration and troubleshooting scenarios to help you ensure a successful load test.

Share:

Kevin Mooney (kmooney@us.ibm.com), Advisory Software Engineer, IBM

author photo>Kevin Mooney is a technical lead and member of the IBM Rational Performance Tester development group in Research Triangle Park, North Carolina. He regularly supports customers engaged in high volume load testing. He has written several white papers on Rational load test software and has been a speaker at Rational user conferences. Kevin has more than 25 years of industry experience since earning a degree in mathematics from the University of North Carolina at Chapel Hill.



Richard Ptasnik (ptasnik@us.ibm.com), Software Developer, IBM

author photoRichard Ptasnik joined IBM in 2000. He has been developing software in Rational tools since 2003, including working on Rational Performance Tester since its inception.



16 April 2013

Highlights of Load Generation Agent features and benefits

A new Load Generation Agent is a vital part of IBM® Rational® Performance Tester Rational Performance Tester Version 8.3.0. As a replacement for the Agent Controller, the Load Generation Agent offers the following features and benefits:

  • Very strong execution process launch reliability for agents. The Load Generation Agent is designed specifically to work with the Rational Performance Tester workbench in the launch and control of load tests.
  • Agents poll the supported workbenches, looking for schedule execution work, rather than the workbench having to establish connections with agents. This scheme provides instant assessment of agent readiness and rapid delivery of launch commands.
  • With the Load Generation Agent, the workbench can launch agents in parallel, thus greatly reducing overall launch time for large load testing operations involving many agents.
  • The Load Generation Agent supports intelligent test asset deployment. It pulls only assets from the workbench that it does not have or that have changed. This feature results in faster launch times for subsequent executions.
  • The Load Generation Agent uses an HTTP server and the HTTP protocol between the workbench and agents. Loss of a connection simply means that another is established with no harm to the overall execution, thereby providing robustness during multiple-day executions.
  • The Load Generation Agent is more firewall friendly. It reverses the contact initiation so that agents connect to the workbench rather than vice versa. The use of an HTTP server within the workbench means that you have only one port to configure with the firewall rather than the many ports on both ends that the Agent Controller required.
  • The ability to secure communications between the workbench and agents by using TLS/SSL is much easier with the Load Generation Agent. You can indicate your preference for secure communication from a single location on the workbench.

Status of the Agent Controller

The Agent Controller is still provided with Version 8.3.0. As an optional part of the installation, you can install the Agent Controller to support these features:

  • Response time breakdown
  • Service-oriented architecture (SOA) stubs

Note:
The Agent Controller cannot be used for load generation with the Rational Performance Tester 8.3.0 workbench.

Although not a typical customer configuration, installing both the Load Generation Agent and the Agent Controller is a supported configuration.


Load Generation Agent architecture

Unlike the Agent Controller architecture, the workbench does not make connections to the agents when a schedule is to be executed. The ports that had to be opened by firewalls for those connections and the connections that the agents made back to the workbench on ephemeral ports are no longer necessary.

With the Load Generation Agent, the agent machines poll for work by sending HTTP requests to the Rational Performance Tester server that is running with the workbench. When you press the Run button to start executing the schedule, the next time that a participating agent contacts the workbench, it responds to the agent by indicating that its participation is needed.

The agent then launches the execution engine as the Agent Controller would have. The execution engine contacts the Rational Performance Tester server, and the usual exchange of commands begins execution of the schedule.

In Figure 1, the Workbench box represents the Eclipse user interface of Rational Performance Tester, which is a single process and one Java virtual machine (JVM). Version 8.3.0 also includes the Jetty web server to support communication between the workbench and agents. Within the workbench, each agent has a load test executor for command, control, and state management of each agent.

Figure 1. Load Generation Agent architecture
Workbench and agent communicate via HTTP

On a single agent, there is the Majordomo service, which is one Java virtual machine that polls supported workbenches for schedule execution work. Each Domo box in the diagram in Figure 1 is a single Java virtual machine that includes the execution engine. Typically, there is only one Java process running on the agent during schedule execution. That single Java process is the Domo and engine process. The diagram shows that it is possible for one agent machine to support multiple execution engines. For garbage collection considerations, if you have a powerful 64-bit agent with a lot of memory, you might find it more efficient to run with multiple engine processes of modest heap size rather than one single process with an enormous heap size.

The Load Generation Agent is simple and compact. The Domo component downloads from the workbench the assets needed for schedule execution, including the execution engine. On subsequent executions, Domo downloads only assets that have changed since the previous execution.


Installation

The installation of the Load Generation Agent is similar to the installation of the Agent Controller. In fact, both features are available when you install the Rational Performance Tester Agent for Version 8.3.0.

Workbench local execution

Although installing the Rational Performance Tester Agent product with the workbench is a supported configuration, it is not required. As with previous releases, it is possible to execute schedules locally on the same machine that the workbench is running on without installing the Agent. The workbench will automatically spawn an instantiation of the Load Generation Agent to support local execution during the life of the execution of the schedule.

Default installation

By default, an Agent installation installs the Load Generation Agent but not the Agent Controller.

Figure 2. Installation options
Load Generation Agent is installed by default

Configure the Load Generation Agent

There are two Load Generation Agent configuration parameters that you must enter during installation:

Workbench host name
Enter one and only one host name that this agent will poll for work.
 
Workbench port
Enter the port number supported by the workbench. The default port number is 7080.
 
Figure 3. Configuration options
Workbench host name and port configuration options

Workbench configuration

The Load Generation Agent has several options that you can configure from the workbench. To configure the options, select Window > Preferences > Test > Server.

Figure 4. Server configuration options
Specify port number, security, deployment deletion

Server port

Select the unsecure and secure ports that the server will use for communication with agents. By default, communication is unsecure, so the unsecure port is the only one that might need to be changed. The port does not have to be changed unless it is already in use by another product running on the workbench. If you change the workbench port, you must also change the agent configuration to use the same port.

The secure port is used only when running with the option to encrypt messages between the workbench and an agent. The secure port number is configured only on the workbench. There is no configuration change required on the agent.

Secure communication using TLS/SSL

Check Workbench and agent communication is encrypted using TLS/SSL to have all messages and execution data encrypted for exchanges between the workbench and agents. All schedules executed from this workbench will use SSL. The workbench will tell the agent what port to use for secure communication via the unsecure port before starting schedule execution.

Delete deployment directory

Check Delete deployment directory on the agent after execution to have the contents of the deployment directory removed from the agent after the schedule is executed. The deployment directory contains all of the test assets that the agent requires for schedule execution. Selecting this option defeats the cached test assets launch time improvement feature, which makes subsequent schedule launch times no shorter than the first schedule launch. The feature is of interest if you are concerned about disk space accumulating on the agent. If you are executing many unique schedules, the test artifacts of each schedule are left on the agent machine after execution is finished.


Modification of agent configurations

After a workbench and many agents are installed, it might become necessary to make configuration changes. This section describes how to modify an agent configuration after installation.

Add support for another workstation to an agent

If you recall from the Installation section, it is possible to specify the host name of only one agent when you configure the Load Generation Agent. So what happens if you add another workstation to the lab and want an agent to support it? The best approach is to use the Agent Status button on an already supported workbench to add the new workbench to the agent's configuration.

Figure 5. Agent status button
Agent status button circled on the task bar

When you click the Agent Status button, a pop-up window shows the status of agents in contact with this workbench. To have an agent support another workbench in the lab, check the box beside the agent and click Share Agent with New Workbench.

As Figure 6 shows, you will then be prompted to enter the host name and port number for the workbench to add to the agent's configuration.

Figure 6. Share an agent with a new workbench
Workbench host name and port fields, required

After clicking OK, the request to add the new workbench will be delivered to the agent. Within a short time, the agent should begin polling the new workbench for schedule execution work.

Load Generation Agent configuration file

After installation of the Rational Performance Tester Agent with the Load Generation Agent feature, there will be an agent configuration file, majordomo.config, stored on the agent. You can accomplish these tasks by editing, copying, or replacing the configuration file:

  1. Change the host name of a workbench that the agent polls for work.
  2. Change the port that the agent will use when contacting the workbench.
  3. Add one or more workbenches for the agent to support.
  4. Replace the entire configuration by using a copy command or gesture to replace the existing majordomo.config file with a new one.
  5. Push a new configuration to many agents by using a remote copy command, such as this one:
    scp /opt/IBM/SDP/Majordomo/majordomo.config root@chickmagnet:/opt/IBM/SDP/Majordomo

A few seconds after you make the change, the Load Generation Agent will refresh its configuration based on the contents of the file. There is no need to stop and restart the agent service.

The contents of the configuration file are in XML, and the format is intuitive. Listing 1 is an example of a configuration file for an agent that is supporting two workbenches.

Listing 1. Configuration file contents
<MajordomoConfig xmlns="http://www.example.org/MajordomoConfiguration">
    <debug>false</debug>
    <canonicalHostName></canonicalHostName>
    <ipAddress></ipAddress>
    <pollInterval>10</pollInterval>
    <workbenches>
        <hostName>kmdesk.raleigh.ibm.com</hostName>
        <port>7081</port>
    </workbenches>
</MajordomoConfig>

The Load Generation Agent configuration file is typically stored in one of these locations:

Windows 64-bit
C:\Program Files\IBM\SDP\Majordomo\majordomo.config
 
Windows 32-bit
C:\Program Files (x86)\IBM\SDP\Majordomo\majordomo.config
 
UNIX
/var/opt/IBM/SDP/Majordomo/majordomo.config
 

Agent Status update feature

As mentioned previously, the workbench contains an Agent Status button. When selected, it is possible to see what agents are polling the workbench for work and whether they are ready to participate in schedule execution.

Figure 7. Agent Status view
Agent status indicates agents are ready

The agent can display these states:

Ready
The agent is ready to execute a schedule.
 
Busy
The agent is currently executing a schedule. While it is busy, it is possible for an agent to initiate and begin executing multiple additional schedules at the same time.
 
Lost Contact
The agent is no longer polling this workbench.
 

Share Agent with New Workbench option

If you select one or more agents and click Share Agent with New Workbench, you can specify the host name and port of a new workbench that you want the agent to poll for work in addition to the this workbench.

Disconnect Agent from this Workbench option

If you select one or more agents and click Disconnect Agent from this Workbench, the host name for this workbench will be removed from the agents' configuration files, and they will no longer poll this workbench for work.


Troubleshooting

This section describes some common problem scenarios and provides suggestions for finding a solution.

Port In Use

Symptom
When you try to run a schedule, you get the error message shown in Figure 8.

Figure 8. Check agents fails due to port in use
Server port is in use and cannot be accessed

Meaning
Some software that is running on the workbench is already using port 7080, the default unsecure port that Rational Performance Tester uses for communication with agents.

Solution 1
Select Window > Preferences > Test > Server, and then choose a different port, such as 7081. All agent configuration files must also specify port 7081. There is no need to restart the workbench or the agent for these changes to take effect.

Solution 2
It is possible that you no longer need the software that is using port 7080 on this machine. If so, stop or uninstall that software.

Check Agents Failed

Symptom
When you try to run a schedule, you get the error message in Figure 9.

Figure 9. Lost contact with agent
Workbench lost contact with an agent

Meaning
One or more of the agents listed as participants in executing this schedule are not in active contact with this workbench. You can use the Agent Status button to view status before executing schedules to determine the likelihood of encountering the Check Agents Failed message.

Solution 1
The Rational Performance Tester Agent is not installed. Use the IBM Installation Manager View Installed Packages feature to ensure that the Agent product is installed.

Solution 2
The Load Generation Agent is installed, but the agent service, Majordomo, is not running.

Microsoft Windows

Look for the MajordomoService process.
Use the Task Manager to verify that the agent is running. If it is not running, use this command: cd "Program Files\SDP\IBM\Majordomo"
NGAStart

UNIX

Look for the Majordomo process. Use the ps command to verify that the agent is running. If it is not running, use this command:
cd /opt/IBM/SDP/Majordomo
./MDStart.sh

Enter these commands by using a command prompt or a shell. Substitute paths based on your installation.

Solution 3
Majordomo is running on the agent, but it is not polling the workbench for work. Verify that the agent can resolve the host name through Domain Name Services as specified in the configuration file. Use the ping command, and specify the workbench host name exactly as it is entered in the majordomo.config file. If the ping command fails using the host name but you can successfully ping the workbench by using its IP address, a workaround could be to specify the workbench IP address in the majordomo.config file rather than the host name.

Solution 4
If Majordomo is running and the workbench can be pinged from the agent, verify that there is not a firewall blocking access. From a command prompt or shell, use a Telnet client to connect to the server, using a command similar to this example:

telnet WORKBENCH_HOST_NAME 7080

Type a few characters, and then press Enter. You should see the screen output shown in Figure 10: Connection to host lost.

Figure 10. Telnet session with server
Telnet client can get a message from the server

If the client is unable to connect or you do not receive a response like the one above, try temporarily disabling firewalls to confirm whether that is the source of the problem.

Windows 7 does not enable the Telnet client by default, but you can enable it by selecting Start > Control Panel > Programs > Programs and Features > Turn Windows feature on or off.

Solution 5
It is also possible to receive the Check Agents Failed dialog if the agent configuration contains workbenches that are not running, workbenches where Rational Performance Tester has been uninstalled, or workbenches that are shut down. An agent that encounters communication problems might get delayed in its polling work long enough that a supported workbench might conclude that agent is out of contact. If you suspect that has happened, remove any stale workbench information from each agent's configuration file.

Process on driver terminated

Symptom
When you try to run a schedule, you get the error message shown in Figure 11.

Figure 11. Execution process terminated
Driver process terminated unexpectedly

Meaning
Shortly after Majordomo launched the engine execution process on the agent, that engine process terminated unexpectedly. There might be messages explaining the reason for the failure in the workbench Error Log. You can open that log by selecting Window > Show View > Error Log. At the default log level of WARNING, all messages written to the stderr output stream by the native Java program on the agent will be sent to the workbench and stored in the Error Log.

Solution 1
The execution process will terminate at startup if incorrect arguments are passed to the Java virtual machine (JVM). Figure 12 is an example of the contents of the Error Log if that happens.

Figure 12. Error log with unrecognized virtual machine option
3 entries show yellow caution icons plus errors

Notice the JVM error notice about an unrecognized command line option. In this case, examine any arguments specified in the RPT_VMARGS general property for the location, and ensure that there are no typos or bogus command line options.

Solution 2
Another example of a problem that produces the process terminated message is when the wrong JVM was used somehow when launching the engine execution process. Figure 13 shows an example of what happens if the agent engine execution is launched with Java 1.6 rather than the required Version 1.7 of IBM Java Virtual Machine (JVM).

Figure 13. Error log showing that the JVM version is incorrect
Error log screen capture

Larger view of Figure 13.

The appearance of UnsupportedClassVersionError is a clue that the JVM on the agent is not the required Version 1.7. This problem could occur if the Rational Performance Tester Agent was installed in a non-default manner or if the installation path contained unexpected characters. Here are suggested workarounds:

  1. Uninstall and reinstall Rational Performance Tester. Use default values for where to install the agent.
  2. Use the RPT_JAVA environment variable to explicitly specify the Java executable file that the agent should launch for the execution engine. For example:
    RPT_JAVA=C:\Program Files\IBM\SDP\jdk\bin\java.exeThe java.exe file to use should be the IBM JVM 1.7 binary installed with the Rational Performance Tester Agent. After you set this environment variable, stop and restart the Majordomo service. From a command prompt or shell, enter these commands:
    Windows
    cd "Program Files/IBM/SDP/Majordomo"
    NGAStop
    NGAStart

    UNIX:
    cd /opt/IBM/SDP/Majordomo
    ./MDStop.sh
    ./MDStart.sh

More tips

Privileges

The Rational Performance Tester Agent must be installed by a user with Administrator privileges on Windows and as root on UNIX.

Multiple engines on one agent machine

In previous versions of Rational Performance Tester, you could use host aliases to run multiple execution engines on one agent machine. The only restriction is that the location host names and deployment directories must be unique. This trick also works with the new Load Generation Agent. Rational Performance Tester does not limit the number of engines executing on one agent.

Locations

There are no changes related to locations and the Load Generation Agent. If you are running the software using a remote agent, a location must be created and added to the schedule as in previous releases of Rational Performance Tester.

Ports with multiple workbenches

If one agent is supporting multiple workbenches, it is fine for the port number to be the same for every workbench listed in the configuration file.

Max heap

Previous releases of Rational Performance Tester required one successful execution of a schedule in order to set the RPT_DEFAULT_MEMORY_SIZE property that specifies the maximum Java heap for a location. This requirement remains for the new Load Generation Agent. On the first execution, the max heap will be 256 megabytes (256 MB), which is an insufficient heap amount that will ensure failure for a large load test.

Majordomo

The heart of the Load Generation Agent is called Majordomo.

  • On Windows, it runs as a service called MajordomoService.exe. Because it is a service, MajordomoService.exe should be running automatically after reboot on Windows. MajordomoService will log informational messages at startup in the system Temp directory, which is usually C:\Windows\Temp\majordomoservice.log. If there is a problem, perhaps related to finding the IBM JVM, there might be messages in that log that could help determine the cause of the problem.
  • On UNIX, it runs as Majordomo. The UNIX systems require your intervention if you want Majordomo running after a reboot.

Majordomo debug flag

You might have noticed that there is a debug flag in the majordomo.config file. If set to true, the Majordomo process and any engine processes launched create log files that can be useful for debugging Load Generation Agent launch and problems that occur early in schedule execution. Examples of these files:

C:\Windows\Temp\SERVICE\majordomo.log

C:\Windows\Temp\SERVICE\engine_HOSTNAME.log

Similar files can be found on UNIX systems stored under %TMP%/USERNAME.

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=870501
ArticleTitle=Tips for using the Load Generation Agent in performance testing
publish-date=04162013