Exempting Db2 Data Management Console by using SED on AIX

This topic provides step-by-step instructions for exempting Db2 Data Management Console using Stack/heap Execution Disable (SED) on AIX Java.

Symptoms

While installing the data management console on an AIX server, the installation process reports some errors of Illegal instruction(coredump)

You get the following message:
./ibm-datamgmtconsole/bin/check.sh[23]: 31850890 Illegal instruction(coredump)
./ibm-datamgmtconsole/bin/check.sh[23]: 12517816 Illegal instruction(coredump)
./ibm-datamgmtconsole/bin/check.sh[23]: 40632666 Illegal instruction(coredump)

Causes

The Stack Execution Disable Protection (SED) is enabled on the AIX server. It prevents buffer overflow attacks by not executing code in data areas of memory.

Resolving the problem

To resolve this issue, exempt the Db2 Data Management Console from SED's buffer overflow protection.

  1. Stop the Db2® Data Management Console.
  2. Go to the Db2 Data Management Console installation directory
    cd <your DMC installation directory>
  3. Run the following command to exempt all the executable applications under the JRE directory (./java_tmp/jre/bin) of the console:
    find java_tmp/jre/bin -type f -perm 0755 -exec sedmgr -c exempt {} \;
  4. Verify that all the executable applications are exempted.
    find java_tmp/jre/bin -type f -perm 0755 -exec sedmgr -d {} \;
    Expected output:
    java_tmp/jre/bin/java : exempt
    java_tmp/jre/bin/jextract : exempt
    java_tmp/jre/bin/jjs : exempt
    java_tmp/jre/bin/jpackcore : exempt
    java_tmp/jre/bin/jrunscript : exempt
    java_tmp/jre/bin/keytool : exempt
    java_tmp/jre/bin/pack200 : exempt
    java_tmp/jre/bin/rmid : exempt
    java_tmp/jre/bin/rmiregistry : exempt
    java_tmp/jre/bin/traceformat : exempt
    java_tmp/jre/bin/unpack200 : exempt
  5. Run the setup.sh script to install the Db2 Data Management Console.