Topic
  • 5 replies
  • Latest Post - ‏2014-04-01T08:25:12Z by bungle_77
GothSloth
GothSloth
20 Posts

Pinned topic Opening and Closing Modules Memory Leak

‏2012-08-03T19:44:37Z |
Hi All,

I have a script I am using to look for corrupted modules on a server with about 22,000 modules on it. Once the script gets rolling it is basically just making a read call followed by a close call and reporting on errors for each module. The script slowly consumes memory until the dreaded memory exhausted pop-up appears.

I have run the script launching the DOORS client with the "-notriggers" switch. The behavior does not change.

On most of the "read" calls I do not see objects getting allocated with the memory leak black magic detection code described in this thread:

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14629945&#14629945

On some modules additional objects will be allocated with the "read" call. The corresponding close call does not release the objects. I added some debug code to print all the open modules in the project before and after the close call. From that I can see all the modules that opened with the "target" module are also getting closed.

If anyone has any insights into what might be going on or suggestions on how to track this down further that would be appreciated.
/*
* Continue test, if OK so far
*/

if( moduleStatus == "" )
{
/*
* Prevent module open errors from popping up the DXL dialog and
* stopping script execution.
*/

noError()

/*
* Open module for reading, do not display, load standard view
*/

thisMod = read(mn, false, true)

/*
* If an open error occurred, lastError returns the error message
*/

lastErrorString = lastError() // <---- THIS IS LINE 685

if( null( thisMod ) //-
|| !null( lastErrorString ) //-
)
{
moduleStatus = "Failed to open module" ( null(lastErrorString) ? "." : ": " lastErrorString )

errorString = "Failed to open module: \"" mn "\"\n"

if( writeError( errorFile, errorString ) == false )
{
progressStop()
close( outputStream )
return
}
}
}

/*
* Continue test, if OK so far
*/

if( moduleStatus == "" )
{
/*
* Close module
*/

progressMessage( count "/" numMods " Closing module '" mn "' ..." )
testModuleCount = 0;
for testModule in current Project do
{
print testModule."Name" "\n"
testModuleCount++;
}
if( testModuleCount == 0 )
{
print "no open modules\n"
}
status = close( thisMod )
print "close call complete\n"

testModuleCount = 0;
for testModule in current Project do
{
print testModule."Name" "\n"
testModuleCount++;
}
if( testModuleCount == 0 )
{
print "no open modules\n"
}

if( status == false )
{

moduleStatus = "Failed to close module."

errorString = "Failed to close module: \"" mn "\"\n"

if( writeError( errorFile, errorString ) == false )
{
progressStop()
close( outputStream )
return
}
}

}
I have wrapped the script in the memory leak black magic detection code described in this thread:

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14629945&#14629945

The output is:

Line:365 --> 0
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5016 --> 1
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5096 --> 2
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5100 --> 3
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5102 --> 4
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5106 --> 5
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5114 --> 7
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5115 --> 8
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5116 --> 9
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5117 --> 10
ScriptingAids/RC_ProcessItemsDialogBox.dxl:5123 --> 11
ScriptingAids/RC_ProcessItemsDialogBox.dxl:3964 --> 12
Line:475 --> 13 // <--- here and above is setting up the dialog box and the output stream
A - FDOR Section 1 through 3.2.7 // <--- below this point I expect everything to ballance
close call complete
no open modules
Line:685 --> 19 // <--- opening the "B - FDOR..." module has allocated 6 objects
B - FDOR Section 4.0 through 5.7.9
Derived From // <--- all of these modules opened with the "B - FDOR..."
Derived From
Derived From
Derived From
Derived From
Derived From
Derived From
Derived From
Derived From
Verifies
DOORS Links
Derived from SKR MOD SPEC
Derived From
close call complete
no open modules // <--- All the "secondary" modules closed with the close
Line:685 --> 22 // call for the "B - FDOR..."
D - FDOR Section 6.2.3
Derived From
Derived From
Derived From
Verifies
Derived From
close call complete
no open modules
Line:685 --> 56
C1 - FDOR Section 6.2.1 - 6.2.1.5.1.2
DOORS Links
Derived From
Derived From
Derived From
Derived From
Derived From
Derived From
DOORS Links
DOORS Links
Derived From
Verifies
DOORS Links
Verifies
Derived From
Derived From
Derived from SKR MOD SPEC
Derived From
close call complete
no open modules
Line:685 --> 87
C2 - FDOR Section 6.2.1.5.1.3
Derived From
Derived From
Derived From
Derived From
DOORS Links
Derived From
DOORS Links
Verifies
Derived From
DOORS Links
Derived from SKR MOD SPEC
Derived From
close call complete
no open modules
Line:685 --> 93
C3 - FDOR Section 6.2.1.5.2 - end
DOORS Links
Derived From
Derived From
Derived From
Derived From
Derived From
Derived From
DOORS Links
DOORS Links
Derived From
Derived From
DOORS Links
Verifies
Derived from SKR MOD SPEC
Derived From
close call complete
no open modules
Line:685 --> 97
G - FDOR Section 6.2.6
Derived From
Derived From
Derived From
Derived From
Verifies
Derived from SKR MOD SPEC
Derived From
Derived From
close call complete
no open modules
Line:685 --> 101
E - FDOR Section 6.2.4
Derived From
Verifies
Derived from SKR MOD SPEC
Derived From
close call complete
no open modules
Line:685 --> 104
F1 - FDOR Section 6.2.5.1
Derived From
Verifies
Derived From
close call complete
no open modules

Line 685 is the "lastErrorString = lastError()" just after the "thisMod = read(mn, false, true)" line.

Thanks,
GothSloth
Updated on 2012-08-04T20:15:21Z at 2012-08-04T20:15:21Z by Mathias Mamsch
  • llandale
    llandale
    3010 Posts

    Re: Opening and Closing Modules Memory Leak

    ‏2012-08-04T18:41:33Z  
    I think Memory issues include:
    • Allocated units not de-allocated
    • string manipulation wasting string table space
    • Too many residually open modules

    Having a hard time reading this. But...
    • 104 allocated units is peanuts. I think the "unit" itself is 4 bytes, not counting the information it holds.
    • I think you want "for mod in database do{CountCurrentlyOpenMods++}"; don't just look in the "Current" project.
    • I'm pretty sure the "Progress" commands eat up some memory.
    • your writeError and logging may be eating up memory.
    • It would be interesting to see how you get your module name "mn".

    22,000 modules is a lot, especially since they seem to be linked to lots of other modules; which all get opened and then all get closed. Those other modules, I note, are NOT being opened with their "Standard view", and I don't know what to do about that. I suppose many may have offending Layouts or Attr-DXL in their default view.

    I would be tempted to stop opening and closing these other modules so often; Perhaps modify your "close all" code that it leaves modules open unless there are X of them residually open; X perhaps is 100 (I have no idea what would be 'good'); or when you switch Projects.

    I'm not sure that opening and closing modules is reasonable mechanism for "finding corruption". Add accessing an attr-Value of all the objects and all its links may result in something useful. However, the Integrity checker does a good job finding and fixing corruption.

    Not sure this is too useful, sorry.

    -Louie
  • Mathias Mamsch
    Mathias Mamsch
    2003 Posts

    Re: Opening and Closing Modules Memory Leak

    ‏2012-08-04T20:15:21Z  

    Ok, first of all you should use the black magic code from here: https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14832712&#14832712

    It is much more convenient, since that debug callback stuff is not only slowing your program down like hell, it is also misleading, since the debug callback is executed BEFORE the reported line is actually executed, so you always need to look on the previously executed line.

    Then if you were successful in indentifying a module where the number of allocated objects will rise whenever you open it, then do this:
     

    // ... use allocated objects code from the referenced post here
     
    { int cnt = 0; Module m; for m in database do cnt++; print "Open Modules Before Opening: " cnt "\n" }
    print "Allocated Objects Before Opening the module: " allocatedObjects() "\n"
    Module mod = read ("...", false, true) 
    print "Allocated Objects After Opening the module: " allocatedObjects() "\n"
    close mod
    print "Allocated Objects After Closing the module: " allocatedObjects() "\n"
    { int cnt = 0; Module m; for m in database do cnt++; print "Open Modules After Closing: " cnt "\n" }
    

     


    Then you will see, if you really have allocated objects due to that call. If so you two options:

     

     

    • send that code and a module archive to IBM and ask them what this is.
    • we can try analysing the kind of leak together. The question would be if you are able to send out a module archive? Otherwise I would make a copy of the module, and start stripping everything from it: Links, Baselines, Views, Attributes, Objects ... until the memory leak goes away. This way you might be able to get a hint on what could be responsible.


    Maybe this helps, regards, Mathias

     

     


    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

     

    Updated on 2014-01-09T14:16:57Z at 2014-01-09T14:16:57Z by iron-man
  • bungle_77
    bungle_77
    58 Posts

    Re: Opening and Closing Modules Memory Leak

    ‏2014-03-31T13:49:25Z  

    Ok, first of all you should use the black magic code from here: https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14832712&#14832712

    It is much more convenient, since that debug callback stuff is not only slowing your program down like hell, it is also misleading, since the debug callback is executed BEFORE the reported line is actually executed, so you always need to look on the previously executed line.

    Then if you were successful in indentifying a module where the number of allocated objects will rise whenever you open it, then do this:
     

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">// ... use allocated objects code from the referenced post here { int cnt = 0; Module m; for m in database do cnt++; print "Open Modules Before Opening: " cnt "\n" } print "Allocated Objects Before Opening the module: " allocatedObjects() "\n" Module mod = read ("...", false, true) print "Allocated Objects After Opening the module: " allocatedObjects() "\n" close mod print "Allocated Objects After Closing the module: " allocatedObjects() "\n" { int cnt = 0; Module m; for m in database do cnt++; print "Open Modules After Closing: " cnt "\n" } </pre>

     


    Then you will see, if you really have allocated objects due to that call. If so you two options:

     

     

    • send that code and a module archive to IBM and ask them what this is.
    • we can try analysing the kind of leak together. The question would be if you are able to send out a module archive? Otherwise I would make a copy of the module, and start stripping everything from it: Links, Baselines, Views, Attributes, Objects ... until the memory leak goes away. This way you might be able to get a hint on what could be responsible.


    Maybe this helps, regards, Mathias

     

     


    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

     

    I got the same error.

    I am writing a recursive script that for each object of a module open all the modules linked to the object and comput the status of the object according to the status of the children objects.

    I don't use buffer or big strings, i just open many modules. I usually close all the modules i open at the end of the script in order to make the script faster. I tried also to close the modules each time i open them, but i still get the memory leak.

    Can someone please help me?

    Attachments

    Updated on 2014-03-31T13:53:26Z at 2014-03-31T13:53:26Z by bungle_77
  • llandale
    llandale
    3010 Posts

    Re: Opening and Closing Modules Memory Leak

    ‏2014-03-31T21:08:47Z  
    • bungle_77
    • ‏2014-03-31T13:49:25Z

    I got the same error.

    I am writing a recursive script that for each object of a module open all the modules linked to the object and comput the status of the object according to the status of the children objects.

    I don't use buffer or big strings, i just open many modules. I usually close all the modules i open at the end of the script in order to make the script faster. I tried also to close the modules each time i open them, but i still get the memory leak.

    Can someone please help me?

    [1] Your print statements inside your loops are a disaster causing exponential loss of string table space.  I think that every time you print, the print command retrieves the output in the DXL output window (all your previous prints), concatenates the new statement, then displays the entire thing.  I'm saying the following code does not print 8 characters, it actually prints 20, all wasted in the string table:

    • print "1\n"   // these two chars are printed 4 times each
    • print "2\n"   // ... printed 3 times each
    • print "3\n"   // .. printed 2 times each
    • print "4\n"   // .. printed once

    Instead, declare a top-level global buffer and append messages instead of printing them, then print the buffer at the end:

    • Bufffer g_bufResults = create()
    • ...g_bufResults += o_obj."Requirement ID" " - "
    • print tempStringOf(g_bufResults)

    [2] Updating your Progress bar so frequently causes huge linear waste in string table space; and I think other waste as well.

    [3] Your code will loop indefinately if you have ANY accidental cirular links: oA <- oB <- oC <- oA.  I would have a global Skip erased at about line 87, where function Recurse near the top checks to see if o_obj is in the Skip; and returns doing nothing if it is; otherwise puts o_obj in the Skip.

    -Louie

    Change your "read" statement to presume to load the Standard View.  This will help make "gotoObject" find objects better:

    • otherMod = read(s_srcModName, false, true)

    Your algorithm doesn't look right; but that could be just me.  I see that you are "rolling up" status to the top-level Objects ("System Requirements" perhaps?)  Seems to me that your bottommost objects would support some mindfully setting the Status, and all intermediate and top level objects would deduce based on your algorithm.  If it were me, I'd have your bottommost with one attribute, "Status", and all the others with another, perhaps "Status Derived". 

    In fact, I'd make the Derived status an Attr-DXL; which only accumulates status from one tier lower level objects.  If you write the Attr-DXL such that it sets all objects in the module all at once (and therefor it runs once; not once for every displayed object), that DXL can cleverly keep track of all open modules, do it's work, then close all modules that were not originally open.

    I may be tempted also to only count objects that have no children.  So if a System requirement has two Sub-System and each has 3 bottommost "Unit" requirements, then the System requirement would be based on the 6 Unit requirements; not 8 requirements if you include the Sub-System.

  • bungle_77
    bungle_77
    58 Posts

    Re: Opening and Closing Modules Memory Leak

    ‏2014-04-01T08:25:12Z  
    • llandale
    • ‏2014-03-31T21:08:47Z

    [1] Your print statements inside your loops are a disaster causing exponential loss of string table space.  I think that every time you print, the print command retrieves the output in the DXL output window (all your previous prints), concatenates the new statement, then displays the entire thing.  I'm saying the following code does not print 8 characters, it actually prints 20, all wasted in the string table:

    • print "1\n"   // these two chars are printed 4 times each
    • print "2\n"   // ... printed 3 times each
    • print "3\n"   // .. printed 2 times each
    • print "4\n"   // .. printed once

    Instead, declare a top-level global buffer and append messages instead of printing them, then print the buffer at the end:

    • Bufffer g_bufResults = create()
    • ...g_bufResults += o_obj."Requirement ID" " - "
    • print tempStringOf(g_bufResults)

    [2] Updating your Progress bar so frequently causes huge linear waste in string table space; and I think other waste as well.

    [3] Your code will loop indefinately if you have ANY accidental cirular links: oA <- oB <- oC <- oA.  I would have a global Skip erased at about line 87, where function Recurse near the top checks to see if o_obj is in the Skip; and returns doing nothing if it is; otherwise puts o_obj in the Skip.

    -Louie

    Change your "read" statement to presume to load the Standard View.  This will help make "gotoObject" find objects better:

    • otherMod = read(s_srcModName, false, true)

    Your algorithm doesn't look right; but that could be just me.  I see that you are "rolling up" status to the top-level Objects ("System Requirements" perhaps?)  Seems to me that your bottommost objects would support some mindfully setting the Status, and all intermediate and top level objects would deduce based on your algorithm.  If it were me, I'd have your bottommost with one attribute, "Status", and all the others with another, perhaps "Status Derived". 

    In fact, I'd make the Derived status an Attr-DXL; which only accumulates status from one tier lower level objects.  If you write the Attr-DXL such that it sets all objects in the module all at once (and therefor it runs once; not once for every displayed object), that DXL can cleverly keep track of all open modules, do it's work, then close all modules that were not originally open.

    I may be tempted also to only count objects that have no children.  So if a System requirement has two Sub-System and each has 3 bottommost "Unit" requirements, then the System requirement would be based on the 6 Unit requirements; not 8 requirements if you include the Sub-System.

    Thanks a lot for your tips.

    Before reading your message i already modified the script and i solved the memory leak problem as you can see in the attached file.

    Anyway i will implement your tip 1 and 3 and i will open the module using the "Standard View".

    I didn't use the dxl attribute because they slow down the scrolling of a module. The attached code is just an extract of the whole code, there is a GUI that allow me to compute the "status" for the current object, for the selected object or for all the objects in the module.

    Attachments