Core dump debugging for the IBM SDK for Node.js
Use the IDDE Eclipse add-on to diagnose program crashes and memory leaks
In Node.js development, core dump analysis can help with debugging program crashes and memory leaks. The IBM SDK for Node.js provides a new method of core dump analysis and debugging for Node.js applications, via the IBM Monitoring and Diagnostic Tools — Interactive Diagnostic Data Explorer (IDDE). IDDE is an Eclipse add-on, available under a free license, that supports core dumps from all platforms on which the IBM SDK for Node.js is distributed, except for Mac OS X. And the dumps are portable; you can take a core dump from one computer and open it in IDDE on another — even one that runs a different (supported) OS.
You can install IDDE from the Eclipse Marketplace or use the update site. Read on to learn how to use IDDE for program crashes and memory leaks as you develop your Node.js apps.
Generating a core dump
The method of generating a core dump differs for different systems. Joyent
(corporate steward of Node.js) recommends running all production Node.js
systems with the --abort-on-uncaught-exception flag. On UNIX
systems, you also need to set ulimit -c unlimited to enable
core files to be generated without a limit on their size.
If an exception hasn't been thrown, you need to use a system tool such as
gcore on Linux or gencore on AIX to generate a
core file, or — if you're happy to kill the process
—kill
-11. On Windows 7 and later, it's possible to
generate a core dump by using the Task Manager: Press
Ctrl+Alt+Delete and select Start Task
Manager. From the Processes tab, right-click the Node.js
process and select Create Dump File. The free ProcDump utility for Windows can also be used.
Debugging a crash
To generate a program crash, we'll use the Test.js script. This simple script loops five times, then throws an error.
Listing 1. Test.js
function main() {
var inputObject = {
input: ["one", "two", "three", "fifteen", "one hundred"],
counter:0,
};
for(; inputObject.counter< inputObject.input.length; inputObject.counter++) {
if (inputObject.input[inputObject.counter].length > 8) {
throw "Input String Too Big";
}
}
}
main();You can run test.js on Linux like this:
$ cd node-v0.12.4-linux-x64/bin/ $ ulimit -c unlimited $ ./node --abort-on-uncaught-exception ../../test.js Uncaught Input String Too Big FROM main (/home/sian/test.js:11:7) Object.<anonymous> (/home/sian/test.js:16:1) Module._compile (module.js:460:26) Object.Module._extensions..js (module.js:478:10) Module.load (module.js:355:32) Function.Module._load (module.js:310:12) Function.Module.runMain (module.js:501:10) startup (node.js:129:16) node.js:814:3 Illegal instruction (core dumped)
Opening a core dump in IDDE
If you have IDDE installed and running on the same computer where your core dump is, you can open the core dump directly from its disk location. Right-click the PD Navigator View (PD stands for problem determination) and select New PD Artifact.
Figure 1. Selecting New PD Artifact
Browse to the location of your core dump and click Finish.
To open the dump on another computer, copy the dump file using your usual tool, then open it in IDDE on the second system in the way we just described. For better native stack traces when you copy the dump to a different location or different computer, also copy the Node executable into the same directory to enable symbols to be resolved.
You now need to work in the IDDE Editor. Select Start Investigation from below your new core file to open the editor.
Figure 2. Opening the IDDE Editor
Large dump files can take extra time to load.
IDDE commands
It can help to think of the IDDE Editor as a hybrid between an editor and a console. You can enter and run commands in the IDDE Editor as if it were a console, but your progress is saved as in an editor.
To run a command, you must precede it with ! and press
Ctrl+Enter. For example, !help, followed
by Ctrl+Enter, outputs the help message, which lists the
other available commands for working with the dump. The Node command set
is different from the Java set and can even be different for different
versions of Node (or Java) as new commands are added in later versions of
IDDE.
As with other Eclipse editors, Ctrl+Space shows completion suggestions, which in the case of the IDDE editor is the list of commands.
Figure 3. Command list opened with Ctrl+Space
A good command to start with is nodeoverview, which gives a
basic summary of the Node version you were running, dependencies, and so
on.
!nodeoverview {
Node Property Value
---------------------- -------------------------------------------------------------
Node version 0.12.4
Path to executable /home/sian/node-v0.12.4-linux-x64/bin/node
Architecture x64
Platform linux
Command Line Arguments /home/sian/node-v0.12.4-linux-x64/bin/node /home/sian/test.js
Execution Arguments --abort-on-uncaught-exception
Process ID 5643
Dependency Version
----------- ----------
http_parser 2.3
node 0.12.4
v8 3.28.71.19 2.2
[...]Investigating the problem
You can see from the preceding console output that the program crashed with
an uncaught exception, so the first thing to do here is to look at the
stack trace. Run the threads command to find thread IDs.
!threads {
Thread ID: 0x74c9 (29897) IP: 0x0000000000eb112d
!stack 29897
}Only one thread was running, so it must be the JavaScript thread.
Try the stack command against the thread. A shortcut to the
command is included in the output of the threads command, so
you can put your cursor on the end of the line containing
!stack 29897 and press Ctrl+Space. Here's
the output (for better readability; we've removed the arguments column and
truncated some lines):
!stack 29897 {
Instruction Pointer Frame Address Location / Frame Type
------------------- ------------------ ------------------------------------------------------------------------------------
0x0000000000eb112d 0x00007FFF0A81F960 node::_ZN2v84base2OS11GetUserTimeEPjS2_+0x9d
0x0000000000a890c1 0x00007FFF0A81FAC0 node::_ZN2v88internal7Isolate29CaptureAndSetSimpleStackTraceENS0_6HandleINS0_8JSO...
0x0000000000b63c9d 0x00007FFF0A81FAF0 node::_ZN2v88internal25Runtime_ThrowNotDateErrorEiPPNS0_6ObjectEPNS0_7IsolateE+0xdd
0x00000E7CAA2A8E32 0x00007FFF0A81FB40 main [/home/sian/test.js]
!jsobject 0x00001519ED09B451
0x00000E7CAA2A8B34 0x00007FFF0A81FB78 <anonymous> [/home/sian/test.js]
!jsobject 0x00001519ED09B341
0x00000E7CAA224AC6 0x00007FFF0A81FBE0 INTERNAL FRAME
0x00000E7CAA2A7D26 0x00007FFF0A81FC58 Module._compile [module.js]
!jsobject 0x00000DA39E6185F9
0x00000E7CAA2A220C 0x00007FFF0A81FCA0 Module._extensions..js [module.js]
!jsobject 0x00000DA39E618691
0x00000E7CAA29E940 0x00007FFF0A81FCE8 Module.load [module.js]
!jsobject 0x00000DA39E618569
0x00000E7CAA295565 0x00007FFF0A81FD70 Module._load [module.js]
!jsobject 0x00000DA39E6184D9
0x00000E7CAA294F24 0x00007FFF0A81FDB8 Module.runMain [module.js]
!jsobject 0x00000DA39E618721
0x00000E7CAA26B31F 0x00007FFF0A81FE28 startup [node.js]
!jsobject 0x0000079578AC8A61
0x00000E7CAA269D10 0x00007FFF0A81FE58 <anonymous> [node.js]
!jsobject 0x0000079578A6EF49
0x00000E7CAA21EF40 0x00007FFF0A81FE98 INTERNAL FRAME
0x00000E7CAA21DE90 0x00007FFF0A81FF20 ENTRY FRAME
0x00000E7CAA21DE90 0x00007FFF0A81FF20 ENTRY FRAME
0x0000000000914E28 0x00007FFF0A81FFF0 NONE FRAME
0x0000000000813bda 0x00007FFF0A820060 node::_ZN2v88Function4CallENS_6HandleINS_5ValueEEEiPS3_+0xba
0x0000000000c9d40f 0x00007FFF0A820190 node::_ZN4node15LoadEnvironmentEPNS_11EnvironmentE+0x1df
0x0000000000c9d67f 0x00007FFF0A8202E0 node::_ZN4node15LoadEnvironmentEPNS_11EnvironmentE+0x44f
0x0000003d0fa1ed5d 0x0000000000000000 node::_fini+0x3d0eb6c8c5
}You can see that the crash occurred in the main method. To go
back to the code to see what might have happened, you don't need to leave
IDDE; if you run the command shown in bold in the preceding output, IDDE
shows you the source.
!jsobject 0x00001519ED09B451 {
Object has fast properties
Number of descriptors : 5
Name Value More Information
--------- ------------------ -------------------------------
length 0x0000079578A0FE19 <EXECUTABLE_ACCESSOR_INFO_TYPE>
name 0x0000079578A0FE51 <EXECUTABLE_ACCESSOR_INFO_TYPE>
arguments 0x0000079578A0FE89 <EXECUTABLE_ACCESSOR_INFO_TYPE>
caller 0x0000079578A0FEC1 <EXECUTABLE_ACCESSOR_INFO_TYPE>
prototype 0x0000079578A0FEF9 <EXECUTABLE_ACCESSOR_INFO_TYPE>
Object is a function
Name: main
Source:
() {
var inputObject = {
input: ["one", "two", "three", "fifteen", "one hundred"],
counter:0,
};
for(; inputObject.counter< inputObject.input.length; inputObject.counter++) {
if (inputObject.input[inputObject.counter].length > 8) {
throw "Input String Too Big";
}
}
}
}Note: This feature is also useful for confirming that the code you thought you were running is the code you were actually running.
It seems obvious from the code that mainObject.counter must
have reached 4, which points to the string "one hundred",
whose length is longer than 8. You can confirm that this is the case by
using two other IDDE commands: jsfindbyproperty and
jsobject. jsfindbyproperty searches the heap for
all objects that have a property with the name supplied
—counter in this example. jsobject shows
the properties of that object.
!jsfindbyproperty counter {
!jsobject 0x00001519ED09B539
!jsobject 0x00001519ED09B689
}
!jsobject 0x00001519ED09B689 {
Object has fast properties
Number of descriptors : 2
Name Value More Information
------- ------------------ ---------------------------------------------
input 0x00001519ED09B6C1 <JS Array[5]> :- !jsobject 0x00001519ed09b6c1
counter 0x0000000400000000 SMI = 4
}The hex numbers produced by jsfindbyproperty (for example,
0x00003CF51F09B441) show you the memory addresses of the
objects found by that command. You can run the jsobject
command against one of these hex addresses, and if a JavaScript object is
located at that address, the command prints out information about that
object's properties. In this case, we can see that counter
had reached 4 on this object.
Now look at the input array, using the shortcut command from
the preceding output.
!jsobject 0x00001519ed09b6c1 {
Array at !hexdump 0x00001519ED09B4E1
Array len = 5
0 : 0x00000CEAAC1A3679, one
1 : 0x00000CEAAC1A3699, two
2 : 0x00000CEAAC1A36B9, three
3 : 0x00000CEAAC1A36D9, fifteen
4 : 0x00000CEAAC1A36F9, one hundred
}You can see the array element that failed the test in the test.js code and caused an exception to be thrown.
Finding memory leaks
Memory leaks can be a common issue in any program. IDDE has several commands to help track down which objects are taking up memory. In this example, we're starting with a core dump taken from a Node.js application that we think is leaking memory.
One way to track down a leak is to take two or more dumps with a significant time gap in between them and to compare the output of some of the commands between the two dumps.
The jsmeminfo command can help to show straight away if one
very large object is using up a lot of space, as in this example.
!jsmeminfo {
Memory allocator, used: 1423 MB, available: 0 MB
Total Heap Objects: 29497
Largest 5 heap objects Type Size (bytes) More information
---------------------- ----------------- ------------ --------------------------
0x0000000088a6d319 JS_OBJECT_TYPE 1311125 !jsobject 0x0000000088a6d319
0x0000000088aac6d9 FIXED_ARRAY_TYPE 98360 !array 0x000003ff88aac6d9
0x000003ff8abe31b9 ASCII_STRING_TYPE 48176 !string 0x000003ff8abe31b9
0x000003ff8ab34f09 ASCII_STRING_TYPE 48104 !string 0x000003ff8ab34f09
0x000003ff8ab04101 ASCII_STRING_TYPE 40936 !string 0x000003ff8ab04101As in the Debugging a crash section, running the
jsobject command against this object will enable you to
relate it to an object in your program and fix the problem.
For another type of memory issue in which a program is creating many
objects and not disposing of them, jsgroupobjects groups
objects of the same type and shows you how many are in the program.
jsgroupobjects also help to identify where Node.js buffers
have been used by displaying the constructor of the object. (Buffers in
Node are a way of allocating memory outside of the heap.) In this example,
Buffer is the most frequently occurring object:
!jsgroupobjects {
Representative Object Address Object Type Num Objects Constructor Num Properties Properties
----------------------------- ------------- ---------- ----------- -------------- ---------------
!jsobject 0x000003ffec004101 JS_OBJECT_TYPE 2572 Buffer 2 length, parent
...You can print out the contents of the buffer in IDDE by identifying the location of the external array allocated by the buffer (in bold in the following output).
print 0x000003ffec004101 {
Object at 0x000003FFEC004101 is JSObject
Class hierarchy :-
|-JSObject
| |- kElementsOffset 0x10 (EXTERNAL_UINT8_ARRAY_TYPE, !print 0x000003FFEC004159)
| |- kPropertiesOffset 0x8 (FIXED_ARRAY_TYPE, !print 0x000003FF92A04111)
| |-JSReceiver
| | |-HeapObject
| | | |- kMapOffset 0x0 (MAP_TYPE, !print 0x000003FF8BE1F6E9)
| | | |-Object
...Take the address of kElementsOffset and feed into the
array command.
!array 0x000003FFEC004159 {
Array type : EXTERNAL_UINT8_ARRAY_TYPE
Len : 10
0 0x48 H
1 0x65 e
2 0x6c l
3 0x6c l
4 0x6f o
5 0x20
6 0x6e n
7 0x6f o
8 0x64 d
9 0x65 e
..The objtypes command is also useful for memory issues. It
shows the count and size in memory of V8 heap object types.
Full command reference
Here's the complete command reference for IDDE. Commands shown here in italics are available for any core dump; all others are specific to Node.js dumps:
array- Display the elements of a fixed array at the specified address
find,findall,findnext- Find a string in memory
frame- Display detailed information about a single JavaScript stack frame
help- Display a list of commands
hexdump- Outputs a section of memory in hexadecimal and ASCII format
jsfindbyproperty- Find JavaScript objects with the specified property
jsgroupobjects- List groups of JavaScript objects that share the same
Map jslistobjects- List heap objects of the specified V8 object type
jsmeminfo- Display information about JavaScript memory usage, including the five largest objects
jsobject- Display JavaScript object details
jsobjectsmatching- Print JavaScript objects that share the same
Mapas the object supplied jsstringsearch- Search for a given string on the heap
locate- Search for a given string in memory
nodeoverview- Display an overview of Node information including version
objtype- List all V8 object types
objtypes- Show a breakdown of heap usage by V8 instance type
print- Display the C++ hierarchy for the specified heap object
ranges- Print list of available memory ranges
stack- Display a JavaScript stack trace for a given thread
string- Display the string at the specified address
threads- List all threads
Conclusion
In this tutorial, you learned how to generate a Node.js core dump, open the dump in IDDE, enter and run commands in the IDDE editor, and get a list of all the commands available for your dump. And you've learned which IDDE commands can best help you track down the root cause of program crashes and memory leaks.
If you have any questions about using IDDE or would like to report a bug or problem you have had with the tool, send email to javatool@uk.ibm.com.
Downloadable resources
Related topics
- IDDE Knowledge Center: Visit the IDDE Knowledge Center for online help with IDDE.
- IBM developer kits: Get information about other IBM diagnostic tools for Java and Node.js, including Health Center and GCMV.
- MDB and Linux: Learn how Joyent recommends getting a core dump from Node.js.
- Buffers: Read about buffers in the Node.js documentation.