IBM Rational performance Tester offers testers to execute the standalone scripts or wrap these scripts into a schedule and run it for multiple virtual user run executions. For any such executions triggered from RPT, you first need to make the RPT application understand and read Java parameters which should be configured within the 'eclipse.ini' file residing in the RPT install directory.
Lets discuss heap memory allocation from RPT perspective in details here....
Generally, The Java Virtual Memory (JVM) heap size for Performance Tester is determined based on the physical memory available on the machine. A percentage of available memory is allocated to the workbench heap (Javaw.exe). If this is not sufficient, you can manually reset the JVM heap size by editing the configuration file ('eclipse.ini'). This file, eclipse.ini, is located in the product install directory (for example, C:\Program Files\IBM\SDP)
NOTE: The heap memory is the memory allocated to the java process by the -xMx parameter whereas the native memory is the memory available to the OS
Now, the question would be ..."How much memory is allocated to the JVM when executing a Test or Schedule in RPT?"
From RPT v8.x and later releases, The playback JVM's maximum heap size when running a Test is set to 1/2 of installed memory (i.e. on a computer with 2 GB of RAM, the heap would be set to 1 GB). For schedule playback, the maximum heap size is set to 3/4 of installed memory. But if you run Schedule using a remote agent controller (for example a new location) you can set a specific memory value using the VMARGS property for the remote agent.
You can refer to the section "Increasing Memory Allocation" under RPT knowledge base.
Also note that we do not recommend playing back the schedule scripts on the RPT workbench machine itself, rather suggest to distribute the user load across explicitly deployed RPT Agent machines.
So, when you trigger the scripts, each of the configured RPT Agents gets its share (as configured in the schedule design in RPT) for execution. During this playback course, the RPT Agent machine is solely responsible until it finishes the execution and then transfer the executed data back to RPT (Test Log Data Transfer Stage) for computing the results. Now, the RPT workbench machine should provide enough memory resources required to complete this result computation process