Technical Blog Post
Abstract
MDM JVM coredumps during startup/shutdown on Red Hat Linux
Body
Problem(Abstract)
When using InfoSphere Master Data Management (MDM) customers will see coredumps being created when an application server is being started or stopped in an environment with multiple application servers.
Symptom
There are several use cases that can lead to this behavior, all involve a WebSphere Application Server topology of more than one JVM in a clustered or non-clustered environment.
Some examples below:
- JVMs recycle by themselves no apparent cause
- JVM1 is up and running, JVM2 is started and causes JVM1 to crash
- JVM1 and JVM2 are up and running, runbatch.sh process is started and causes JVM1 to crash
- JVM1 and JVM2 are up and running, if a user attempts to stop JVM1 or JVM2, the JVM that is being stopped crashes
Cause
This issue has been identified by Red Hat Linux Support as a defect Bug 1327623 - replacing .so which was opened and closed, leads to segfault on next dlopen/dlsym.
Diagnosing the problem
The javacore and heapdumps will have two things in common:
1. The crash occurring in the ld-linux-x86-64.so.2 library (Red Hat Linux) which is called from dlsym() call in the JVM
2. A process tries to access the libMAD.so library (MDM) from an existing JVM causing it to crash
Contact IBM Support if you suspect you are experiencing this issue. Ensure the MustGather: Crash on Linux is followed to collect all required information and that the core has been processed through jextract.
Resolving the problem
In order to have immediate relief to this issue a workaround is available.
Follow steps below:
1. Stop any JVM that has the core MDM application installed
2. Locate the libMAD.so library and set attribute to IMMUTABLE
Example: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/expandedBundles/com.ibm.mdm.mds.jni.app_11.4.0.FP00IF000_20140924-0832/com.ibm.mdm.mds.jni_11.4.0.FP00IF000_20140924-0832.jar/linux64
Ensure to locate the library in expandedBundles directory.
3. As root, run the command below:
chattr +i /some/path/libMAD.so
Note:
chattr (Change Attribute) is a command line Linux utility that is used to set/unset certain attributes to a file in Linux system to secure accidental deletion or modification of important files and folders.
The defect seems to be opening/closing the libMAD.so, setting this property will prevent such activity, which will prevent the crash.
4. Restart JVMs and test
Related information
Bug 1327623 - replacing .so which was opened and closed
MustGather: Crash on Linux
UID
ibm11142158