Contents


Core dump debugging for the IBM SDK for Node.js

Use the IDDE Eclipse add-on to diagnose program crashes and memory leaks

Comments

In Node.js development, core dump analysis can help with debugging program crashes and memory leaks. The IBM SDK for Node.js provides a new method of core dump analysis and debugging for Node.js applications, via the IBM Monitoring and Diagnostic Tools — Interactive Diagnostic Data Explorer (IDDE). IDDE is an Eclipse add-on, available under a free license, that supports core dumps from all platforms on which the IBM SDK for Node.js is distributed, except for Mac OS X. And the dumps are portable; you can take a core dump from one computer and open it in IDDE on another — even one that runs a different (supported) OS.

You can install IDDE from the Eclipse Marketplace or use the update site. Read on to learn how to use IDDE for program crashes and memory leaks as you develop your Node.js apps.

Generating a core dump

The method of generating a core dump differs for different systems. Joyent (corporate steward of Node.js) recommends running all production Node.js systems with the --abort-on-uncaught-exception flag. On UNIX systems, you also need to set ulimit -c unlimited to enable core files to be generated without a limit on their size.

If an exception hasn't been thrown, you need to use a system tool such as gcore on Linux or gencore on AIX to generate a core file, or — if you're happy to kill the process —kill -11. On Windows 7 and later, it's possible to generate a core dump by using the Task Manager: Press Ctrl+Alt+Delete and select Start Task Manager. From the Processes tab, right-click the Node.js process and select Create Dump File. The free ProcDump utility for Windows can also be used.

Debugging a crash

To generate a program crash, we'll use the Test.js script. This simple script loops five times, then throws an error.

Listing 1. Test.js
function main() {
  var inputObject = {
    input: ["one", "two", "three", "fifteen", "one hundred"],
    counter:0,
  };

  for(; inputObject.counter< inputObject.input.length; inputObject.counter++) {
    if (inputObject.input[inputObject.counter].length > 8) {
      throw "Input String Too Big";
    }
  }
}

main();

You can run test.js on Linux like this:

$ cd node-v0.12.4-linux-x64/bin/
$ ulimit -c unlimited
$ ./node --abort-on-uncaught-exception ../../test.js
Uncaught Input String Too Big

FROM
main (/home/sian/test.js:11:7)
Object.<anonymous> (/home/sian/test.js:16:1)
Module._compile (module.js:460:26)
Object.Module._extensions..js (module.js:478:10)
Module.load (module.js:355:32)
Function.Module._load (module.js:310:12)
Function.Module.runMain (module.js:501:10)
startup (node.js:129:16)
node.js:814:3
Illegal instruction (core dumped)

Opening a core dump in IDDE

If you have IDDE installed and running on the same computer where your core dump is, you can open the core dump directly from its disk location. Right-click the PD Navigator View (PD stands for problem determination) and select New PD Artifact.

Figure 1. Selecting New PD Artifact
Image shows selecting New PD Artifact
Image shows selecting New PD Artifact

Browse to the location of your core dump and click Finish.

To open the dump on another computer, copy the dump file using your usual tool, then open it in IDDE on the second system in the way we just described. For better native stack traces when you copy the dump to a different location or different computer, also copy the Node executable into the same directory to enable symbols to be resolved.

You now need to work in the IDDE Editor. Select Start Investigation from below your new core file to open the editor.

Figure 2. Opening the IDDE Editor
Image shows opening the IDDE Editor
Image shows opening the IDDE Editor

Large dump files can take extra time to load.

IDDE commands

It can help to think of the IDDE Editor as a hybrid between an editor and a console. You can enter and run commands in the IDDE Editor as if it were a console, but your progress is saved as in an editor.

To run a command, you must precede it with ! and press Ctrl+Enter. For example, !help, followed by Ctrl+Enter, outputs the help message, which lists the other available commands for working with the dump. The Node command set is different from the Java set and can even be different for different versions of Node (or Java) as new commands are added in later versions of IDDE.

As with other Eclipse editors, Ctrl+Space shows completion suggestions, which in the case of the IDDE editor is the list of commands.

Figure 3. Command list opened with Ctrl+Space
Image shows command list opened with                     Ctrl+Space
Image shows command list opened with Ctrl+Space

A good command to start with is nodeoverview, which gives a basic summary of the Node version you were running, dependencies, and so on.

!nodeoverview {

Node Property           Value                                                        
----------------------  -------------------------------------------------------------
Node version            0.12.4                                                       
Path to executable      /home/sian/node-v0.12.4-linux-x64/bin/node                   
Architecture            x64                                                          
Platform                linux                                                        
Command Line Arguments  /home/sian/node-v0.12.4-linux-x64/bin/node /home/sian/test.js
Execution Arguments     --abort-on-uncaught-exception                                
Process ID              5643                                                        

Dependency   Version   
-----------  ----------
http_parser  2.3       
node         0.12.4    
v8           3.28.71.19          2.2    

[...]

Investigating the problem

You can see from the preceding console output that the program crashed with an uncaught exception, so the first thing to do here is to look at the stack trace. Run the threads command to find thread IDs.

!threads {

Thread ID: 0x74c9 (29897) IP: 0x0000000000eb112d
    !stack 29897

}

Only one thread was running, so it must be the JavaScript thread.

Try the stack command against the thread. A shortcut to the command is included in the output of the threads command, so you can put your cursor on the end of the line containing !stack 29897 and press Ctrl+Space. Here's the output (for better readability; we've removed the arguments column and truncated some lines):

!stack 29897 {

Instruction Pointer  Frame Address       Location / Frame Type                                                                                                                                     
-------------------  ------------------  ------------------------------------------------------------------------------------
 0x0000000000eb112d  0x00007FFF0A81F960  node::_ZN2v84base2OS11GetUserTimeEPjS2_+0x9d                                              
 0x0000000000a890c1  0x00007FFF0A81FAC0  node::_ZN2v88internal7Isolate29CaptureAndSetSimpleStackTraceENS0_6HandleINS0_8JSO...                                                 
 0x0000000000b63c9d  0x00007FFF0A81FAF0  node::_ZN2v88internal25Runtime_ThrowNotDateErrorEiPPNS0_6ObjectEPNS0_7IsolateE+0xdd
 0x00000E7CAA2A8E32  0x00007FFF0A81FB40  main [/home/sian/test.js]                                                                                                                                    
                                         !jsobject 0x00001519ED09B451                                                                                                                                       
 0x00000E7CAA2A8B34  0x00007FFF0A81FB78  <anonymous> [/home/sian/test.js]                                                                                    
                                         !jsobject 0x00001519ED09B341                                                                                                    
 0x00000E7CAA224AC6  0x00007FFF0A81FBE0  INTERNAL FRAME                                                                                                                                                     
 0x00000E7CAA2A7D26  0x00007FFF0A81FC58  Module._compile [module.js]                                                                                                                 
                                         !jsobject 0x00000DA39E6185F9                                                                                            
 0x00000E7CAA2A220C  0x00007FFF0A81FCA0  Module._extensions..js [module.js]                                                                                  
                                         !jsobject 0x00000DA39E618691                                                                                            
 0x00000E7CAA29E940  0x00007FFF0A81FCE8  Module.load [module.js]                                                                                                  
                                         !jsobject 0x00000DA39E618569                                                                                                                                      
 0x00000E7CAA295565  0x00007FFF0A81FD70  Module._load [module.js]           
                                         !jsobject 0x00000DA39E6184D9           
 0x00000E7CAA294F24  0x00007FFF0A81FDB8  Module.runMain [module.js]                                                                          
                                         !jsobject 0x00000DA39E618721                                                                                                       
 0x00000E7CAA26B31F  0x00007FFF0A81FE28  startup [node.js]                                                    
                                         !jsobject 0x0000079578AC8A61
 0x00000E7CAA269D10  0x00007FFF0A81FE58  <anonymous> [node.js]       
                                         !jsobject 0x0000079578A6EF49                                                                                                                                      
 0x00000E7CAA21EF40  0x00007FFF0A81FE98  INTERNAL FRAME                                                                                                                                                     
 0x00000E7CAA21DE90  0x00007FFF0A81FF20  ENTRY FRAME                                                                                                                                                        
 0x00000E7CAA21DE90  0x00007FFF0A81FF20  ENTRY FRAME                                                                                                                                                        
 0x0000000000914E28  0x00007FFF0A81FFF0  NONE FRAME                                                                                                                                                         
 0x0000000000813bda  0x00007FFF0A820060  node::_ZN2v88Function4CallENS_6HandleINS_5ValueEEEiPS3_+0xba                                                                                                      
 0x0000000000c9d40f  0x00007FFF0A820190  node::_ZN4node15LoadEnvironmentEPNS_11EnvironmentE+0x1df                                                                                                          
 0x0000000000c9d67f  0x00007FFF0A8202E0  node::_ZN4node15LoadEnvironmentEPNS_11EnvironmentE+0x44f                                                                                                          
 0x0000003d0fa1ed5d  0x0000000000000000  node::_fini+0x3d0eb6c8c5                                                                                                                                          

}

You can see that the crash occurred in the main method. To go back to the code to see what might have happened, you don't need to leave IDDE; if you run the command shown in bold in the preceding output, IDDE shows you the source.

!jsobject 0x00001519ED09B451 {

Object has fast properties
Number of descriptors : 5

Name       Value               More Information               
---------  ------------------  -------------------------------
length     0x0000079578A0FE19  <EXECUTABLE_ACCESSOR_INFO_TYPE>
name       0x0000079578A0FE51  <EXECUTABLE_ACCESSOR_INFO_TYPE>
arguments  0x0000079578A0FE89  <EXECUTABLE_ACCESSOR_INFO_TYPE>
caller     0x0000079578A0FEC1  <EXECUTABLE_ACCESSOR_INFO_TYPE>
prototype  0x0000079578A0FEF9  <EXECUTABLE_ACCESSOR_INFO_TYPE>

Object is a function

Name: main

Source:

() {
  var inputObject = {
    input: ["one", "two", "three", "fifteen", "one hundred"],
    counter:0,
  };

  for(; inputObject.counter< inputObject.input.length; inputObject.counter++) {
    if (inputObject.input[inputObject.counter].length > 8) {
      throw "Input String Too Big";
    }
  }
}
}

Note: This feature is also useful for confirming that the code you thought you were running is the code you were actually running.

It seems obvious from the code that mainObject.counter must have reached 4, which points to the string "one hundred", whose length is longer than 8. You can confirm that this is the case by using two other IDDE commands: jsfindbyproperty and jsobject. jsfindbyproperty searches the heap for all objects that have a property with the name supplied —counter in this example. jsobject shows the properties of that object.

!jsfindbyproperty counter {

!jsobject 0x00001519ED09B539
!jsobject 0x00001519ED09B689

}


!jsobject 0x00001519ED09B689 {

Object has fast properties
Number of descriptors : 2

Name     Value               More Information                             
-------  ------------------  ---------------------------------------------
input    0x00001519ED09B6C1  <JS Array[5]> :- !jsobject 0x00001519ed09b6c1
counter  0x0000000400000000  SMI = 4                                      

}

The hex numbers produced by jsfindbyproperty (for example, 0x00003CF51F09B441) show you the memory addresses of the objects found by that command. You can run the jsobject command against one of these hex addresses, and if a JavaScript object is located at that address, the command prints out information about that object's properties. In this case, we can see that counter had reached 4 on this object.

Now look at the input array, using the shortcut command from the preceding output.

!jsobject 0x00001519ed09b6c1 {

Array at !hexdump 0x00001519ED09B4E1
Array len = 5

0 : 0x00000CEAAC1A3679, one
1 : 0x00000CEAAC1A3699, two
2 : 0x00000CEAAC1A36B9, three
3 : 0x00000CEAAC1A36D9, fifteen
4 : 0x00000CEAAC1A36F9, one hundred

}

You can see the array element that failed the test in the test.js code and caused an exception to be thrown.

Finding memory leaks

Memory leaks can be a common issue in any program. IDDE has several commands to help track down which objects are taking up memory. In this example, we're starting with a core dump taken from a Node.js application that we think is leaking memory.

One way to track down a leak is to take two or more dumps with a significant time gap in between them and to compare the output of some of the commands between the two dumps.

The jsmeminfo command can help to show straight away if one very large object is using up a lot of space, as in this example.

!jsmeminfo {

Memory allocator, used: 1423 MB, available: 0 MB
Total Heap Objects: 29497

Largest 5 heap objects  Type               Size (bytes)  More information          
----------------------  -----------------  ------------  --------------------------
0x0000000088a6d319      JS_OBJECT_TYPE          1311125  !jsobject 0x0000000088a6d319
0x0000000088aac6d9      FIXED_ARRAY_TYPE          98360  !array 0x000003ff88aac6d9
0x000003ff8abe31b9      ASCII_STRING_TYPE         48176  !string 0x000003ff8abe31b9
0x000003ff8ab34f09      ASCII_STRING_TYPE         48104  !string 0x000003ff8ab34f09
0x000003ff8ab04101      ASCII_STRING_TYPE         40936  !string 0x000003ff8ab04101

As in the Debugging a crash section, running the jsobject command against this object will enable you to relate it to an object in your program and fix the problem.

For another type of memory issue in which a program is creating many objects and not disposing of them, jsgroupobjects groups objects of the same type and shows you how many are in the program. jsgroupobjects also help to identify where Node.js buffers have been used by displaying the constructor of the object. (Buffers in Node are a way of allocating memory outside of the heap.) In this example, Buffer is the most frequently occurring object:

!jsgroupobjects {

Representative Object Address  Object Type    Num Objects Constructor  Num Properties  Properties                                                                      
-----------------------------  -------------  ----------  -----------  --------------  ---------------
!jsobject 0x000003ffec004101   JS_OBJECT_TYPE       2572      Buffer         2        length, parent
...

You can print out the contents of the buffer in IDDE by identifying the location of the external array allocated by the buffer (in bold in the following output).

print 0x000003ffec004101 {

Object at 0x000003FFEC004101 is JSObject

Class hierarchy :-

|-JSObject
|  |- kElementsOffset 0x10 (EXTERNAL_UINT8_ARRAY_TYPE, !print 0x000003FFEC004159)
|  |- kPropertiesOffset 0x8 (FIXED_ARRAY_TYPE, !print 0x000003FF92A04111)
| |-JSReceiver
| | |-HeapObject
| | |  |- kMapOffset 0x0 (MAP_TYPE, !print 0x000003FF8BE1F6E9)
| | | |-Object
...

Take the address of kElementsOffset and feed into the array command.

!array 0x000003FFEC004159 {

Array type : EXTERNAL_UINT8_ARRAY_TYPE
Len : 10
0 0x48 H
1 0x65 e
2 0x6c l
3 0x6c l
4 0x6f o
5 0x20 
6 0x6e n
7 0x6f o
8 0x64 d
9 0x65 e
..

The objtypes command is also useful for memory issues. It shows the count and size in memory of V8 heap object types.

Full command reference

Here's the complete command reference for IDDE. Commands shown here in italics are available for any core dump; all others are specific to Node.js dumps:

array
Display the elements of a fixed array at the specified address
find, findall, findnext
Find a string in memory
frame
Display detailed information about a single JavaScript stack frame
help
Display a list of commands
hexdump
Outputs a section of memory in hexadecimal and ASCII format
jsfindbyproperty
Find JavaScript objects with the specified property
jsgroupobjects
List groups of JavaScript objects that share the same Map
jslistobjects
List heap objects of the specified V8 object type
jsmeminfo
Display information about JavaScript memory usage, including the five largest objects
jsobject
Display JavaScript object details
jsobjectsmatching
Print JavaScript objects that share the same Map as the object supplied
jsstringsearch
Search for a given string on the heap
locate
Search for a given string in memory
nodeoverview
Display an overview of Node information including version
objtype
List all V8 object types
objtypes
Show a breakdown of heap usage by V8 instance type
print
Display the C++ hierarchy for the specified heap object
ranges
Print list of available memory ranges
stack
Display a JavaScript stack trace for a given thread
string
Display the string at the specified address
threads
List all threads

Conclusion

In this tutorial, you learned how to generate a Node.js core dump, open the dump in IDDE, enter and run commands in the IDDE editor, and get a list of all the commands available for your dump. And you've learned which IDDE commands can best help you track down the root cause of program crashes and memory leaks.

If you have any questions about using IDDE or would like to report a bug or problem you have had with the tool, send email to javatool@uk.ibm.com.


Downloadable resources


Related topics

  • IDDE Knowledge Center: Visit the IDDE Knowledge Center for online help with IDDE.
  • IBM developer kits: Get information about other IBM diagnostic tools for Java and Node.js, including Health Center and GCMV.
  • MDB and Linux: Learn how Joyent recommends getting a core dump from Node.js.
  • Buffers: Read about buffers in the Node.js documentation.

Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Web development, Linux
ArticleID=1016094
ArticleTitle=Core dump debugging for the IBM SDK for Node.js
publish-date=09302015