Comments (2)
  • Add a Comment
  • Edit
  • More Actions v
  • Quarantine this Entry

1 DrStephenH commented Permalink

Fork-n-kill the child process is a cunning and quick way - cost of process creation/fork - to clone the content of the process and offload the more time consuming matter of writing the core file to disk (provided sufficient physical memory resources - equal to the size of the parent java process - of virtual memory exists to be able to conduct the process fork ; these pages will be reserved, rather can committed usually, to enable parent and child processes to take unique divergent execution paths with the data they first share immediately following the fork). (This will probably influence the chances of success for this technique with larger 64-bit processes!...With so many physical pages of memory reserved, this could be a further source for some subtle transient native OutOfMemory problems, with potential knock-on consequences elsewhere in the running workload (for the time it takes to write the corefile i.e. the lifetime of our child 'java' process).) <div>&nbsp;</div> There is (at least) one other subtle outcome for the UNIX/POSIX fork though : you are not cloning the many threads of the multi-threaded (java) process. The child process created by the fork comes with only a distinct new single primordial thread upon which the execution stream of instructions (shared with the parent!) continues... its this instruction stream that in typical UNIX fork-n-exec process creation would lead to subsequent execution of the exec() in the child process. But it would seem that the loss of the many original threads of the parent process could represent quite a lot of lost current information about the subject (parent) process? For example, the typical classic example of roots for the Java heap is the references maintained within the stack frames (and other local data) of the executing Java threads, themselves based on the native threads. Likewise, thread stacks - especially that of the last executing thread - are often one of the first places reviewed within a core file... but the child process will only exist with the one thread? (And, that is executing only the instruction stream from the fork to the kill!) Similarly, the state of (semaphore) locks cannot be replicated into the child process - perhaps affecting the Java monitor/lock information preserved in the child? <div>&nbsp;</div> So, the usefulness of the fork-n-kill mechanism would seem to depend on exactly what analysis you want to perform on the resulting corefile. A number of enquiries would seem to be prevented, unless some additional deeper magic/cunning has been enacted to somehow replicate these information structures that were previously the purview of the data of the threads of the parent process? Perhaps gcore, as a mechanism, still has its place? <br /> Or, has some crafty developer (on/for Linux?) implemented the deeper magic that would seem to be needed? Perhaps why we are discussing a fork-n-kill (with a user-defined 'diagnostic' JVM signal handler) rather than a fork-n-abort?

2 kgibm commented Permalink

Hey Stephen, good points. <div>&nbsp;</div> I wonder if the default Linux memory overcommit settings will help avoid issues in reserving so many extra pages? <div>&nbsp;</div> Also, on the point of not cloning threads, the InfoCenter does not this that gdb will only see one thread, but DTFJ is able to constitute the rest of the threads. I know that DTFJ is even able to figure out what was on each Java stack (e.g. in the registers), but I don't know if this is only for Java threads or native also. I will follow-up with the team to see what limitations there are for this fork-n-kill approach. How is fork-n-abort different than a SIGSEGV? <div>&nbsp;</div> The broader point you bring up is a good one though: for advanced analysis such as for the original purpose of this series - to understand total virtual usage - maybe we can get the best of both worlds using gcore. We can grab a fully original core dump using gcore and grab /proc/PID/smaps at the same time. In fact, I updated the ibmgcore.sh script (linked) to do just that. The only problem then is that DTFJ doesn't see everything, but at least from a gdb POV we do.