Troubleshooting
Problem
When running IBM® Rational® Application Developer (RAD) for WebSphere® Software in Microsoft® Windows® Enterprise Edition 2003 Service Pack (SP) 1 or Windows XP Professional SP2 with DEP compatible CPUs, an error message appears, titled: rationalsdp.exe - application error The instruction at "0x00c1ff74" referenced memory at "0x00c1ff74". The memory could not be written. Rational Application Developer cannot be started. The installation process finishes leaving out the final step of license registration.
Cause
The exact sequence of events is described in the following screenshots:
At this point, you can click on "Change Settings" to indicate to DEP to allow rationalsdp.exe to proceed the next time it is started.
But otherwise, DEP has already killed the current execution:
Then the Installation continues trying to enroll the license, which also is caught by DEP:
and also can be added to the list of programs to exclude from DEP control by clicking on "Change Settings":
but this run of enroll.exe is already terminated:
and therefore the installation ends with the following warning:
and finally you will see the end of the installation:
The problem is caused by the Data Execution Protection (DEP) feature that comes as a part of the Athlon 64 processor (or CPUs with Hardware-Enforced capability) under Microsoft Windows XP SP2 or Windows 2003 SP1. Both Advanced Micro Devices™ (AMD) and Intel® Corporation have defined and shipped Windows-compatible architectures that are compatible with DEP. These CPUs enable data execution protection (also known as no-execute protection) so that Memory Manager can mark pages containing data as no-execute, to prevent execution of the data as code.
This feature can be useful for preventing malicious code from exploiting the computer’s system files and Registry; however, it also stops any valid executables from updating them. The result is that RAD users are no longer able to run RAD with DEP enabled, since running it requires executing the file rationalsdp.exe. It also prevents them from manually enrolling licenses after the installation because it requires running the file enroll.exe
Resolving The Problem
This problem would affect any IBM Rational Software Development Platform (SDP) version 6 based products that specifically use rationalsdp.exe. Some specific Rational product examples are: RAD version 6 and Rational Software Architect (RSA) version 6. Examples of non-Rational products based on RSDP are : WebSphere® Message Broker Toolkit (WMBT) Version 6 and WebSphere Integration Developer version 6.
From a user point of view, it might appear that the the installation has possibly failed or been incomplete towards the end. The problem shows towards the end of the installation process as the installer starts RAD version 6 for the very first initialization and again when it tries to enroll the RAD license. It is not necessary to re-install RAD. The installation should have been successful and it should suffice to do the following steps:
- Disable DEP using the Specific DEP Solution options described in the next section (after these steps).
- Since the command
rationalsdp.exedid not complete, the installation is not quite done. Despite RAD being functional at this point, it is not quite equivalent to a proper install, since the performance is impacted. The RAD unique JXEs, which will not be discussed further in this technote, were not generated. These are required to make "the product start faster and features to load faster at run time" as per a normal RAD install.
After excludingrationalsdp.exefrom DEP, you should run:
./rationalsdp.exe -application com.ibm.jxesupport.initialize -clean
and then everything will be fine with no need to uninstall.
- Next, manually run enroll.exe to enroll the RAD license:
enroll.exe -dir <full path to directory where the .lic files are>
By default the directory location is:
C:\Program Files\IBM\Rational\SDP\setup\lum\<product>\is rad.
,where for example <product>
Since the enroll.exe adds a registry key to the Windows Registry during installation, it would be blocked by DEP if it is on. Setting enroll.exe as an exception to DEP, then later on executing enroll.exe should complete the RAD installation process with DEP. Refer to Related Information Item: Getting a "License cannot be found" or "License cannot be found or has expired" error , for the instructions for manually registering licenses in general. The command option shown above makes it easier by referring directly to the directory containing the product license files.
- rationalsdp.exe with the options in step 2 above does not start RAD. RAD should now be restarted.
Specific DEP Solution options are as follows:
The DEP solutions are based on the following Microsoft (MS) references in the Related Information section:
A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003 (MS kb 875353)
How to determine that hardware DEP is available and configured on your computer (MS kb 912923)
Changes to Functionality in Microsoft Windows XP Service Pack 2 Part 3: Memory Protection Technologies
In particular, Microsoft Knowledge Base (kb) article 875353 explains system-wide DEP configuration and pre-program DEP configuration for Windows XP SP2 and 2003 SP1. These are official Microsoft references in resolving DEP-related issues and you should consult Microsoft for further DEP configuration assistance or information, unless further assistance is needed with RAD in this context.
Option 1
The quickest way to run RAD is to disable DEP altogether:
- Change the setting /NOEXECUTE to ALWAYSOFF (or replace /NOEXECUTE=OPTOUT with /EXECUTE=OPTOUT) in the hidden C:\boot.ini system file.
- Save the file and restart the computer.
- Run RAD after the computer restarts.
However, this method will lower the security of the computer by opening it to attacks from other malicious executables. It is important to turn on the firewall and anti-virus software while DEP is disabled.
Here is the list of /NOEXECUTE options that can be specified in the hidden system file C:\boot.ini to enable and disable DEP:
/NOEXECUTE=OPTIN Enables DEP for core system images and those specified in the DEP configuration dialog.
/NOEXECUTE=OPTOUT Enables DEP for all images except those specified in the DEP configuration dialog.
/NOEXECUTE=ALWAYSON Enables DEP on all images.
/NOEXECUTE=ALWAYSOFF Disables DEP.
See option 2 next...
Option 2
- Keep DEP enabled by setting /NOEXECUTE to OPTOUT in the C:\boot.ini file. It enables DEP for all images except those specified in the DEP configuration dialog.
- From the desktop, right mouse click on My Computer and choose Properties > Advanced, under Performance click Settings and under the Performance Options dialogue box choose the Data Execution Prevention tab.
- Select the radio button that says "Turn on DEP for all programs and services except those I select". Add both <rad home>\rationsdp.exe and <rad home>\setup\lum\rad\enrol.exe to the exception list.
By following the steps in Method 2, the computer can still be protected from malicious executables while enrolling and running RAD.
Re-Install
If RAD does not start and function correctly at this point it might be necessary to re-install because of other reasons unrelated to DEP. If using Add/Remove Programs in Windows is not sufficient, consult Related Information item: How to manually un-install Rational Application Developer on Windows operating system.
Note:
The following RAD version 6 installation and runtime defect records have been created to track this problem in follow-on releases of products such as RAD, for possible future solutions:
- APAR PK24942 : MICROSOFT DATA EXECUTION PROTECTION (DEP) STOPS RAD INSTALLATION
APAR PK21350 : CANNOT RUN RAD WHEN WINDOWS DEP IS ENABLED (DATA EXECUTION PREVENTION)
Related Information
Product Synonym
Rational Method Composer
Was this topic helpful?
Document Information
Modified date:
10 September 2020
UID
swg21225545