IBM Support

API recording stops with "Access violation" when McAfee VirusScan Enterprise 8.0.0 is installed

Troubleshooting


Problem

Attempts to run an IBM Rational Robot API recording result in "fatal application error" when McAfee VirusScan Enterprise 8.0.0 is installed.

Symptom

IBM Rational Robot API recording no longer works after the installation of McAfee VirusScan Enterprise 8.0. Prior to installing McAfee "VirusScan v8.0i for Win NT/2000/XP" on a system where Robot is installed, Robot API records successfully.

After installing this version of McAfee, recording in Internet Explorer soon fail. After completing the Start Application dialog, you see the following message in a "Session Recorder" dialog window:



    The application (C:\PROGRAM FILES\INTERNET EXPLORER\IEXPLORE.EXE)
    experienced a fatal application error
    Code: C0000005 (ACCESS_VIOLATION)
    Addr: 77E42DFC
    Flags: 00000000

Cause

It was determined that API recording will not work with McAfee VirusScan Enterprise 8.0.0 if McAfee's "Buffer Overflow Protection" is enabled.

Also McAfee's Host Intrusion Prevention (HIPS) is a possible cause.

Resolving The Problem

Disable Buffer Overflow Protection when you do API recording.


WORKAROUND

Open the McAfee settings.

  1. Disable the "Buffer Overflow Protection" from the VirusScan Console.


You need to disable this setting only during the API recording .After you have finished the recording, you can then enable "Buffer Overflow Protection" again.

NOTE: According to the readme.txt of McAfee, other products (they list four different firewall products) also have trouble with Buffer Overflow Protection. The readme.txt recommends disabling the Buffer Overflow feature if you need to run these products.

The second issue has a relation with the Host Intrusion Prevention (HIPS) software that McAfee pushes. Consider the following.

  1. Uninstall McAfee,

  2. Uninstall then HIPS, along with the KB955759 patch.

  3. Reinstall McAfee without HIPS.



    Follow the procedure below before you create Robot 7.0.1.1 scripts.


  4. Disable access protection

  5. Disable buffer overflow

  6. Disable on-access

  7. Go to Services and stop McAfee Framework Service (the agent) and set to Disabled startup

  8. Switch the McAfee Framework Service back to Automatic startup and start the service after you complete testing.

[{"Product":{"code":"SSSHDX","label":"Rational Robot"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Installation","Platform":[{"code":"PF033","label":"Windows"}],"Version":"7.0;7.0.0.1;7.0.0.10;7.0.0.2;7.0.0.3;7.0.0.4;7.0.0.5;7.0.0.6;7.0.0.7;7.0.0.8;7.0.0.9;7.0.1;7.0.1.1;7.0.1.10;7.0.1.11;7.0.1.12;7.0.1.2;7.0.1.3;7.0.1.4;7.0.1.5;7.0.1.6;7.0.1.7;7.0.1.8;7.0.1.9;7.0.2;7.0.2.1;7.0.2.2","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
16 June 2018

UID

swg21197843

Page Feedback