RFT throws "InvalidSpyHeapReference" while starting "Player" or "Recoder" or while starting the IDE itself .
After trying out a few times we end up restarting the machine
When an RFT process (recorder/player etc) is running, most of times a dll named "rtxivsys.dll" will be loaded into different processes in the system apart from the actual AUT(Application Under Test) process. Once the RFT process exits/ends, this DLL would get unloaded (except from the AUT process or RFT process itself).
In case of AUT process, once the AUT is closed we need not bother about the rtxivsys.dll anymore.
Now sometimes due some error conditions rtxivsys.dll that was loaded into the other processes remains loaded there indefinitely(the error conditions could be possible recording failure or some abnormal halt to the playback process etc).
Now the next time the RFT process is started , user may (not necessarily) see the infamous "InvalidSpyHeapReference" errors (or some other spyheap error), and even though we restart RFT and the AUT we still see the same error.
How to handle it ?
When we run into this situation we try closing RFT and the AUT and sometimes end up restarting the machine which ideally should be the last resort.
What does restarting the machine do? Well it terminates ALL the processes and the system and starts afresh. Which effectively means that no process has the DLL rtxivsys.dll loaded into it when the system restarts and therefore there would be no invalid memory reference errors after that.
Now the question is, if all that a system restart is doing to fix the issue, is terminate the all the running processes out of which some/many processes were the ones where rtxivys.dll was loaded, can't we identify those processes and kill them ourselves ?
And the answer is " Yes, certainly we can and we should, especially if restarting the machine is a pain "
Well here are a few things I do when I run into the problem to avoid restarting the machine.
Close RFT and any open browsers or the AUT.
Launch command prompt ( ie. start>Run> cmd) and Type the followin
tasklist /m rtxiv*
The above command will list all the processes where rtxivsys.dll is loaded
Go to the taskmanger and terminate those processes** and again check to see if the problem is fixed.
If that does not solve the issue, then possibly there are processes which still have rtxivsys.dll loaded but are not shown by the tasklist /m command we just ran (may be because the command prompt window is 32 bit and the processes are 64 bit) and in that case :
I use a utility called "Process Explorer" ( from Microsoft ), and use that utility to search for "rtxivysys.dll" and once found such processes I terminate them**
Now try to perform the action that was giving issue and now we should not see the problem anymore .
The steps listed above may seem like too many but once you do it you would find that its much easier than restarting the entire system.
Hope that helps.
RFT - Rational Functional Tester
AUT - Application Under Test or Software Under Test (SUT)
**We don't have to terminate the process from the taskmanger if we know which Application that process belongs to , we can simply close that application
**Before terminating a process from taskmanger or ProcessExplorer , make sure to save your data if the process being terminated has some unsaved data.