Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
8 replies Latest Post - ‏2012-11-12T22:10:49Z by llandale
llandale
llandale
2943 Posts
ACCEPTED ANSWER

Pinned topic Deleting structures

‏2012-06-01T18:42:21Z |
Once in a while I'm blind as a bat, but I just realized that there are numerous DXL data structures that should be deleted when no longer needed. I wonder if we could get such a list, including ones that should not.

Its rather obvious to pair up "create" commands with a "delete" command (e.g. create Skip, use it, delete Skip). But maybe not so obvious perhaps a "ModuleProperty".

Some stuff however has a "create" which should not be "deleted", such as an Object. So I wonder if these references take up internal data structures that cannot be released. I think we can delete a "ViewDef" but not a "View".

-Louie
Updated on 2012-11-12T22:10:49Z at 2012-11-12T22:10:49Z by llandale
  • Mathias Mamsch
    Mathias Mamsch
    1946 Posts
    ACCEPTED ANSWER

    Re: Deleting structures

    ‏2012-06-01T20:08:39Z  in response to llandale

    Well the stuff that can be created and deleted is all the internal data structures of DXL (that are not bound to persisting data in DOORS)
     

    create/delete            : Array, Buffer, Skip, DxlObject, Regexp, OleAutoArgs, Stat
    create/oleCloseAutoObject: OleAutoObj 
    create / destroy         : DB
    client, server / delete  : IPC
    Probably also: HTTPBody, HTTPHeader, HTTPHeaderEntry,
    

     


    A special case is Stream, which you can close, but that will not free the Stream, i.e. a Stream is always leaked.

    Then there is stuff that is bound persistently to DOORS Data:

     

     

    Module, Item, Project, View, ...
    



    Since they are bound to a specific DOORS instance, they are managed by DOORS internally and would need no destructor. They will also not create an entry in the allocated objects list. However there are special cases, for which this is not the case. They have no destructors, but will still create an entry in the allocated object list (what I consider a bug):



     

     

     

    Baseline, Column, (possibly more?), ...
    



    ModuleVersion is a very bad case, because this one actually has a destructor (delete (ModuleVersion)) but it also has a copy-assignment operator that will leak ModuleVersions uncontrolled unless you are very careful. Worst thing is, that DXL will call the copy-assignment operator only when the assignment is not part of the declaration. Example:



     

     

     

     

    // targetVersion creates a new ModuleVersion 
    ModuleVersion mv = targetVersion theLink  // assignment in declaration, will not create a copy of the ModuleVersion
    delete mv // free the ModuleVersion, everything is fine 
     
     
    ModuleVersion mv = null 
    // this creates a copy of the targetVersion result, and will leak the targetVersion result
    mv = targetVersion theLink   // calls the ::=(ModuleVersion &, ModuleVersion) copy-assignment operator!
    delete mv // delete only the copy
    



    Regarding the question, which DOORS Structures can be deleted without actually modifying the database is a good question. I did not try many of them, but it would surprise me if it were many. I am not sure if ViewDef creates an allocated object list entry. But ViewDefs are kind of complicated anyway, do you know any other structures that can be deleted this way?

    Regards, Mathias



     

     

     


    Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

     

    Updated on 2014-01-06T21:26:51Z at 2014-01-06T21:26:51Z by iron-man
    • llandale
      llandale
      2943 Posts
      ACCEPTED ANSWER

      Re: Deleting structures

      ‏2012-06-04T21:24:25Z  in response to Mathias Mamsch

      I see ModuleProperties didn't make your lists. Are there just a couple, or perhaps many others that we need to deal with? Can we -- meaning you -- create a "void delete(&ModuleProperties)" function?

      I'm sure you have some evidence of your "ModuleVersion" anomaly, but do you have an explanation and/or some way to deal with it? Does this solve the problem?

      ModuleVersion targetVersionOfLink(Link lnk)
      {  // Get the ModuleVersion of the target of the link
         // Caller should delete the mv when finished with it
         ModuleVersion mv = targetVersion(lnk)
         return(mv)
      }  // end targetVersionOfLink()
       
      ModuleVersion sourceVersionOfLink(Link lnk) ditto
      


      -Louie

      Updated on 2014-01-06T21:27:31Z at 2014-01-06T21:27:31Z by iron-man
      • Mathias Mamsch
        Mathias Mamsch
        1946 Posts
        ACCEPTED ANSWER

        Re: Deleting structures

        ‏2012-06-07T14:16:12Z  in response to llandale

        I answered the ModuleVersion thing here: https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14832712&#14832712

        Regarding the ModuleProperties thing I guess I forgot about that. But I really do not know how to find out what types we need to deal with, other than trying. Using the allocatedObjects function from the above post one can determine if an object needs to be freed. For ViewDef for example the code looks like that:
         

        print "Allocated Objects Before Declaration: " allocatedObjects() "\n"
        ViewDef vd = createPublic() 
        print "Allocated Objects After Declaration: " allocatedObjects() "\n"
        

         


        If you run this you see, that there is no entry in the allocated objects list. So does this mean that it will permanently leak memory? No. There is in fact another category of objects in DXL, the most known one possibly the Attr__ type.

         

         

         

        if (null current Object) error "Please run this script from a module with an object!"
         
        print "Allocated Objects Before Declaration: " allocatedObjects() "\n"
        Attr__ x = (current Object)."Object Text"  // allocate the Attr__ 
        print "Allocated Objects After Declaration: " allocatedObjects() "\n"
        string s = x   // assignment will free the Attr__
        print "Allocated Objects After Assignment: " allocatedObjects() "\n"
        



        My guess is that the same is true for ViewDef objects. Creating a ViewDef only makes sense with saving a view, therefore hopefully DOORS will not leak the ViewDef memory, but free it as soon as you save the view definition. The only way to test this is to try to assign the ViewDef after saving the view, and see if eventually you get a crash (sign of View Definition being freed). The same will happen if you try to assign the Attr__ twice.

        So what types exactly are you worried about?

        Regards, Mathias



         

         

        Updated on 2014-01-06T21:28:58Z at 2014-01-06T21:28:58Z by iron-man
        • llandale
          llandale
          2943 Posts
          ACCEPTED ANSWER

          Re: Deleting structures

          ‏2012-11-10T17:54:57Z  in response to Mathias Mamsch
          I've got a batch script that finds all recently modified modules, opens them Exclusive, then saves and closes without making changes. The idea is to improve database access on the several huge Shared modules we have; periodically saving them Exclusive is supposed to consolidate database files and improve open times.

          I use "ModuleProperties" to query the "Last Modified On" date of the module, figuring to ignore it if more than 8 days old. I notice memory useage increases rather dramatically and I presume it to be the large number of ModuleProperties allocated units.

          Can you create a "void delete(&ModuleProperties)" function?

          -Louie
          • Mathias Mamsch
            Mathias Mamsch
            1946 Posts
            ACCEPTED ANSWER

            Re: Deleting structures

            ‏2012-11-10T18:36:15Z  in response to llandale
            Well since the ModuleProperties go to the allocation list it seems to be the best way instead of a very bad hack to try to call the internal cleanup function to use eval_ to read the ModuleProperties to a buffer. I showed here how to do it:

            http://www.ibm.com/developerworks/forums/click.jspa?searchID=-1&messageID=14849772

            Regards, Mathias

            Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS
            • llandale
              llandale
              2943 Posts
              ACCEPTED ANSWER

              Re: Deleting structures

              ‏2012-11-11T19:34:18Z  in response to Mathias Mamsch

              You know I have an unreasonable resistance to outside-the-box voodoo. Never-the-less I ran your code in that post; wrote my own "string" instead of "Buffer" version; and wrote a test to see how long each takes along with using the normal non-eval version.

              I was rather shocked at the results:

              • all versions take about the same amount of time, 150ms
              • 150ms seems like a whopping amount of time to me, getProperties() seems to be the culprit
              • "eval_" didn't add any significant amount of time.
              • creating a bunch of structures and running the code increased the Buffer and normal versions somewhat.


              I'm inclined to use the String version because I would expect the Buffer version to deteriorate in time in large programs that create a lot of other structures.

              Thoughts? Like perhaps the Buffer version allows distinction between results and errors and that matters.

              -Louie

               

              string    NameMod_Target        = "/Testing/Document1"
              string  Results_Expected        = "Testing_"  // Prefix of module
              pragma runLim, 0
               
              //******************************************
              string  fModAttr_GetValue(ModName_ in_mn, string in_NameAttr) 
              {     // Get the module attr value without opening the module
                      // This is done using ModuleProperties but since there is no delete for
                      //      that structure, this function uses "eval_" to do it
                      //      presuming the structure is removed after eval_() ends
                      if (null in_mn  or null in_NameAttr)            then return("")       // Program input error
               
                      string  ErrMess, EvalResult = ""
                      ModuleVersion   mv = moduleVersion (in_mn) 
                      if (null mv)
                      {  print "fModAttr_GetValue(); bad moduleVersion\n"
                         return("")
                      }
                      string  EvalCode = "
                              ModName_        mn = addr_ " (int (addr_ in_mn)) "
                              ModuleVersion   mv = addr_ " (int (addr_ mv))    "
                              ModuleProperties mp
                              getProperties (mv, mp) 
                              return_(mp.\"" in_NameAttr "\" \"\")
                      "  // end string EvalCode
                      // print "Evaluating EvalCode: " EvalCode "\n\n"
                      ErrMess = checkDXL(EvalCode)
                      if (!null ErrMess)
                      {  print "fModAttr_GetValue(); checkDXL() error [" ErrMess "\t]... for code:" EvalCode "\n"
                      }
                      else
                      {  noError()
                         EvalResult = eval_(EvalCode)
                         ErrMess      = lastError() 
                         if (!null ErrMess)
                         {  print "fModAttr_GetValue() eval_() error [" ErrMess "\t]... for code:" EvalCode "\n"
                            EvalResult        = ""
                         }
                      }
                      delete(mv)
                      // print "EvalResult [" EvalResult "]\n"
                      return EvalResult
              }    // end fModAttr_GetValue()
               
              //******************************************
              Buffer readModuleProperty (ModName_ mn, string aName) {
                      Buffer buf = create() 
                      string sCode = "
                              ModName_ mn = addr_ " ((addr_ mn)  int) "
                              Buffer buf  = addr_ " ((addr_ buf) int) "
                              ModuleVersion mv = moduleVersion mn 
                              ModuleProperties mp
                              getProperties (mv, mp) 
                              buf = mp.\"" aName "\"
                      "
                      // print "Evaluating sCode: " sCode "\n\n"
                      noError(); string result = eval_ sCode; string errMsg = lastError() 
                      if (!null result || !null errMsg) { delete buf; buf = null Buffer; print errMsg }
                      return buf
              }
               
               
              void    DoTest_Buffer(int in_NumLoops)
              {
                      Buffer  x
                      ModName_        mn = module(NameMod_Target)
                      int     i, TimeStart = getTickCount_()
                      for (i=0; i<in_NumLoops; i++)
                      {  x = readModuleProperty(mn, "Prefix") 
                         if (!null x) delete x
                      }
                      print "\t" (getTickCount_() - TimeStart) "ms\tDoTest_Buffer\n" 
                      x = readModuleProperty(mn, "Prefix") 
                      if (!null x) 
                      {  if (tempStringOf(x) != Results_Expected) print "\t\tBAD results [" x "]\n"
                         delete x     
                      }
                      else print "\t\tNull buff results\n"
              }
               
               
              void    DoTest_String(int in_NumLoops)
              {
                      string  x
                      ModName_        mn = module(NameMod_Target)
                      int     i, TimeStart = getTickCount_()
                      for (i=0; i<in_NumLoops; i++)
                      {  x = fModAttr_GetValue(mn, "Prefix") 
                      }
                      print "\t" (getTickCount_() - TimeStart) "ms\tDoTest_String\n" 
                      if (x != Results_Expected) print "\t\tBAD results [" x "]\n"
              }
               
               
              void    DoTest_Normal(int in_NumLoops)
              {
                      string  x
                      ModName_        mn = module(NameMod_Target)
                      ModuleVersion   mv
                      ModuleProperties mp
                              //      mv = moduleVersion mn 
                              //      getProperties (mv, mp) 
                      int     i, TimeStart = getTickCount_()
                      for (i=0; i<in_NumLoops; i++)
                      {  mv = moduleVersion mn 
                         getProperties (mv, mp) 
                         x = mp."Prefix"
                      }
                      print "\t" (getTickCount_() - TimeStart) "ms\tDoTest_Normal\n" 
                      if (x != Results_Expected) print "\t\tBAD results [" x "]\n"
              }
               
              void    DoAllTests(int in_NumLoops)
              {
                      print "Test: #Loops: " in_NumLoops "\n"
                      DoTest_Normal(in_NumLoops)
                      DoTest_Buffer(in_NumLoops)
                      DoTest_String(in_NumLoops)
                      DoTest_Normal(in_NumLoops)
              }
               
              if (false)
              {    // Create a bunch of structures.  100,000 took about 2 minutes
                      int i
                      Buffer x
                      for (i=0; i<30000; i++) x = create()
              }
               
              DoAllTests(100)         // 100 loops
              

               

               

              Results:
               
              Test: #Loops: 100
                  1547ms      DoTest_Normal
                      1453ms  DoTest_Buffer
                      1469ms  DoTest_String
                      1594ms  DoTest_Normal
              

               

              Updated on 2014-01-06T21:30:38Z at 2014-01-06T21:30:38Z by iron-man
              • Mathias Mamsch
                Mathias Mamsch
                1946 Posts
                ACCEPTED ANSWER

                Re: Deleting structures

                ‏2012-11-11T20:41:42Z  in response to llandale
                Well - i am not surprised by 150ms - after all getProperties is a network operation. It will read the module attribute file from the server and evaluate it. Regarding the timing I am really surprised, that there is no caching in the DOORS client. I would expect, that getProperties takes a long time on the first run and no time on the next runs. All in all it seems there is no need to any optimization in DXL as long as you are not dealing with stuff, that takes more than 100ms.

                Of course if you expect DXL programs with lots of allocations the string version will exceed the buffer version in its performance. The error checking could either be done in the eval or you put some string in front of the result and check for that string. So that is no real reason to use the buffer variant. So I guess you will be fine using the string variant, as long as you don't query very long richText values (if that is even possible).

                Regards, Mathias

                Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS