Topic
  • 9 replies
  • Latest Post - ‏2013-02-19T18:53:22Z by Mathias Mamsch
SystemAdmin
SystemAdmin
3180 Posts

Pinned topic Stream memory leak

‏2013-02-15T21:08:28Z |
Is there a way to close a file stream so it does not leak memory when using 'write' or 'append'? Even the example in DXL help "Files and streams example program" does not clean up after itself. I used Mathias Mamsch's allocatedObjects() code to test and found that each 'write' and 'append' adds 1 to the the count. The attached sample code ends up with a total of 4.
Updated on 2013-02-19T18:53:22Z at 2013-02-19T18:53:22Z by Mathias Mamsch
  • Mathias Mamsch
    Mathias Mamsch
    1970 Posts

    Re: Stream memory leak

    ‏2013-02-16T21:11:03Z  

    Normally I use an eval_ context for cleaning up the streams, if I need to create a lot of them. In most cases you can get along, by keeping the stream open and just use flush (Stream) to make sure, that the content is indeed written. Anyway, if you need to resort to reading and writing a lot of files, you must use the eval context, for example as follows:
     

    Buffer gBufWriteTextContentBuffer = create()
    Buffer gBufWriteTextFilenameBuffer = create()
     
    string gStrWriteBufferCode = "
        Buffer bufWriteTextContentBuffer  = (addr_ " ((addr_ gBufWriteTextContentBuffer) int) ") Buffer
        Buffer bufWriteTextFilenameBuffer = (addr_ " ((addr_ gBufWriteTextFilenameBuffer) int) ") Buffer
     
        // Create file and write contents; and catch errors if any
        noError
        
        // Create file for write access
        Stream os = write( bufWriteTextFilenameBuffer \"\")
        
        // Write needed contents to the batch file stream:
        os << bufWriteTextContentBuffer
        
        // Save stream contents to file and close it
        flush( os )
        close( os )
        
        return_ lastError
    "
     
    string writeWithoutLeak (string file_Name, Buffer fileContent) {
        gBufWriteTextFilenameBuffer = file_Name
        gBufWriteTextContentBuffer  = tempStringOf fileContent
        noError()    
        string msgEval = eval_ gStrWriteBufferCode
        string strErr = lastError()
        if (msgEval != "") return msgEval 
        return strErr 
    }
    

     


    Note that the code string is a constant here, due to the use of Buffers to marshal both fileName and content into the eval_ context. Note that this way you need to copy the fileContent buffer, so if you write a lot of large files, you will be quicker by using a buffer reference / pointer instead.

    You can do the same for reading a file, which is left as an excercise ;-) Note that for marshalling an integer from an to an eval_ context you do:

     

     

     

    int gFunctionResult = 0 
     
    string strCode = " 
          int &retVal = addr_ " ((addr_ (&gReadFileRetVal)) int) "
          // ... your code ... 
          retVal = 4 // set your return value
    "
    



    again the string code stays static, so you do not need to litter the string table instead of leaking objects. Hope that helps, regards, Mathias



     

     

     


    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

     

    Updated on 2014-01-09T00:21:20Z at 2014-01-09T00:21:20Z by iron-man
  • SystemAdmin
    SystemAdmin
    3180 Posts

    Re: Stream memory leak

    ‏2013-02-18T07:08:28Z  

    Normally I use an eval_ context for cleaning up the streams, if I need to create a lot of them. In most cases you can get along, by keeping the stream open and just use flush (Stream) to make sure, that the content is indeed written. Anyway, if you need to resort to reading and writing a lot of files, you must use the eval context, for example as follows:
     

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">Buffer gBufWriteTextContentBuffer = create() Buffer gBufWriteTextFilenameBuffer = create() string gStrWriteBufferCode = " Buffer bufWriteTextContentBuffer = (addr_ " ((addr_ gBufWriteTextContentBuffer) int) ") Buffer Buffer bufWriteTextFilenameBuffer = (addr_ " ((addr_ gBufWriteTextFilenameBuffer) int) ") Buffer // Create file and write contents; and catch errors if any noError // Create file for write access Stream os = write( bufWriteTextFilenameBuffer \"\") // Write needed contents to the batch file stream: os << bufWriteTextContentBuffer // Save stream contents to file and close it flush( os ) close( os ) return_ lastError " string writeWithoutLeak (string file_Name, Buffer fileContent) { gBufWriteTextFilenameBuffer = file_Name gBufWriteTextContentBuffer = tempStringOf fileContent noError() string msgEval = eval_ gStrWriteBufferCode string strErr = lastError() if (msgEval != "") return msgEval return strErr } </pre>

     


    Note that the code string is a constant here, due to the use of Buffers to marshal both fileName and content into the eval_ context. Note that this way you need to copy the fileContent buffer, so if you write a lot of large files, you will be quicker by using a buffer reference / pointer instead.

    You can do the same for reading a file, which is left as an excercise ;-) Note that for marshalling an integer from an to an eval_ context you do:

     

     

     

    <pre class="javascript dw" data-editor-lang="js" data-pbcklang="javascript" dir="ltr">int gFunctionResult = 0 string strCode = " int &retVal = addr_ " ((addr_ (&gReadFileRetVal)) int) " // ... your code ... retVal = 4 // set your return value " </pre>



    again the string code stays static, so you do not need to litter the string table instead of leaking objects. Hope that helps, regards, Mathias



     

     

     


    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

     

    Hi Mathias

    What is "a lot of files"?

    1.000, 10.000 or 100.000?

    Best regards
    Wolfgang
  • Mathias Mamsch
    Mathias Mamsch
    1970 Posts

    Re: Stream memory leak

    ‏2013-02-18T10:30:03Z  
    Hi Mathias

    What is "a lot of files"?

    1.000, 10.000 or 100.000?

    Best regards
    Wolfgang
    Well, every time you open a Stream you will add an object to the list of allocated objects, which you cannot get rid of anymore. These objects will degrade the performance of your program more and more, from my experience starting with about 1000 objects.

    I remember something, that the time for allocation increased linearly from 10000 objects to 50000 objects by factor 1000, but the real slow down of the code depends on the relative speed of code that contains the allocation.

    There are some posts on the forum, that contain discussions about the slowdown and code to demonstrate the slowdown:

    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14672594&#14672594

    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14775320&#14775320

    If 10000 objects are much or few depends on the code. A script that allocates a buffer for each object will slow down for large modules. A script that will append to a log file for each object of a module too.

    Regards, Mathias

    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS
  • SystemAdmin
    SystemAdmin
    3180 Posts

    Re: Stream memory leak

    ‏2013-02-18T11:44:20Z  
    Well, every time you open a Stream you will add an object to the list of allocated objects, which you cannot get rid of anymore. These objects will degrade the performance of your program more and more, from my experience starting with about 1000 objects.

    I remember something, that the time for allocation increased linearly from 10000 objects to 50000 objects by factor 1000, but the real slow down of the code depends on the relative speed of code that contains the allocation.

    There are some posts on the forum, that contain discussions about the slowdown and code to demonstrate the slowdown:

    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14672594&#14672594

    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14775320&#14775320

    If 10000 objects are much or few depends on the code. A script that allocates a buffer for each object will slow down for large modules. A script that will append to a log file for each object of a module too.

    Regards, Mathias

    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS
    Then it is important even to me. Thank you for the hint and thank you for countAllocatedObjects()
  • SystemAdmin
    SystemAdmin
    3180 Posts

    Re: Stream memory leak

    ‏2013-02-19T00:32:33Z  
    Mathias,
    Your code is most appreciated!!! It works great with one tweak. I changed "Stream os = write(...)" to Stream os = append(..) so the text file was not overwritten each time the writeWithoutLeak procedure was called.
    I admit I need a better understanding of eval_ and addr_ any suggestions where I can get more information?

    Thanks for your help!
  • llandale
    llandale
    2972 Posts

    Re: Stream memory leak

    ‏2013-02-19T14:32:28Z  
    Mathias,
    Your code is most appreciated!!! It works great with one tweak. I changed "Stream os = write(...)" to Stream os = append(..) so the text file was not overwritten each time the writeWithoutLeak procedure was called.
    I admit I need a better understanding of eval_ and addr_ any suggestions where I can get more information?

    Thanks for your help!
    Wikipedia "VooDoo".
  • SystemAdmin
    SystemAdmin
    3180 Posts

    Re: Stream memory leak

    ‏2013-02-19T14:41:51Z  
    • llandale
    • ‏2013-02-19T14:32:28Z
    Wikipedia "VooDoo".
    > Wikipedia "VooDoo".

    wiktionary ROFL
  • Mathias Mamsch
    Mathias Mamsch
    1970 Posts

    Re: Stream memory leak

    ‏2013-02-19T16:19:37Z  
    Mathias,
    Your code is most appreciated!!! It works great with one tweak. I changed "Stream os = write(...)" to Stream os = append(..) so the text file was not overwritten each time the writeWithoutLeak procedure was called.
    I admit I need a better understanding of eval_ and addr_ any suggestions where I can get more information?

    Thanks for your help!
    I will make a new post for that. Regards, Mathias
  • Mathias Mamsch
    Mathias Mamsch
    1970 Posts

    Re: Stream memory leak

    ‏2013-02-19T18:53:22Z  
    I will make a new post for that. Regards, Mathias
    It is kind of huge, but here it is:

    http://www.ibm.com/developerworks/forums/thread.jspa?threadID=471754&tstart=0

    Sorry for the amount of info, but I came from this into that when explaining the code above. Regards, Mathias

    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS