Native memory exhaustion in Windows

The operating system process running the Java virtual machine (VM) does not have enough memory to produce the dump file.

Sometimes the VM can fail to produce a dump file on Windows because there is not enough native memory available to the operating system process that is running the VM. For example, the following output might be written to STDERR:
Unhandled exception
Type=Segmentation error vmState=0x00000001
J9Generic_Signal_Number=00000004 ExceptionCode=c0000005 ExceptionAddress=707D1A95 ContextFlags=0001003f
Handler1=7064F300 Handler2=7134EF90 InaccessibleAddress=00000000
EDI=01F1FB78 ESI=00000000 EAX=00001388 EBX=003BBA18
ECX=00000000 EDX=01DB0048
EIP=707D1A95 ESP=01F1FB5C EBP=01F1FB68 EFLAGS=00010246
GS=002B FS=0053 ES=002B DS=002B
Module_base_address=707D0000 Offset_in_DLL=00001a95
Target=2_60_20110521_082820 (Windows 7 6.1 build 7601 Service Pack 1)
CPU=x86 (2 logical CPUs) (0x12e54d000 RAM)
----------- Stack Backtrace -----------
JVMDUMP006I Processing dump event "gpf", detail "" - please wait.
JVMDUMP032I JVM requested System dump using 'C:\test\native\OOMCrash\core.20110525.134230.4148.0001.dmp' 
in response to an event
JVMDUMP012E Error in System dump: Dump failed - could not load library DBGHELP.DLL
You might also see this error:
JVMDUMP012E Error in System dump: Dump failed - could not begin dump thread

The library DBGHELP.DLL is needed for the VM to generate a system dump file on Windows, and is provided in the SDK. A failure to load this library indicates that the process has exhausted the available native memory.

You can use a Windows debugger such as WinDbg, or Visual Studio, or the Windows Task Manager to generate a system dump file in these circumstances. These debuggers load the required libraries in a separate process. You might need to force the VM to stop at the point of failure in order to obtain a dump file using these debuggers. If the failure is a crash, as in the previous example, you can set the -Xrs command-line option. This option causes the operating system to stop the VM and display a Windows dialog box when the crash occurs. If the failure is not a crash, for example, if the VM was producing a dump file for a Java™ exception, you can use the sleep trigger action on a tracepoint to halt the VM while you obtain the dump file. Use this command:
java "-Xtrace:print=j9dmp,trigger=tpnid{j9dmp.10,sleep}"