Topic
  • 7 replies
  • Latest Post - ‏2013-08-02T05:27:04Z by DanielJunglas
FANS_Nader_Al_Theeb
FANS_Nader_Al_Theeb
3 Posts

Pinned topic Clear Models and Memory leak

‏2013-06-29T21:47:40Z |

Dear all,

I discussed this topic in the past, but I have some issues. Let's see the following code

--------------------------------------------------------------------------------------------------

Create model // name = model

for i=1:100

IloCplex cplex(env);
cplex.extract(model);

IloModel model2(env);
model2.add(model);
cplex.extract(model2);

modify the model2

solve and save the objective value

        cplex.clearModel();
        cplex.end();
        model2.end();

end for

---------------------------------------------------------------------------------------

 

My problem is that even I clear the model at the end of each iteration, a small portion of memory is used somewhere and not cleared. This cause the iterations to take longer time (5 seconds per iteration for the first 10 iterations, 30 seconds per iteration for the last 10 iterations)

Anyone knows why this happened and where the memory is lost?

Thanks

 

 

 

  • DanielJunglas
    DanielJunglas
    119 Posts

    Re: Clear Models and Memory leak

    ‏2013-07-01T05:08:08Z  

    What version of CPLEX do you use? How do you assess that memory is leaked? Do you have a tool that actually reports the leakage?

    How exactly do you modify model2? Maybe the code that performs the modification leaks memory?

    Could you please print out env.getMemoryUsage() at the beginning of each iteration and check whether this value increases?

    How large is the amount of memory that is leaked? I have a hard time believing that a small memory leak will result in a 25 second slowdown. Do you apply the exact same modifications in each iteration or may the slowdown be due to the fact that you are solving different models?

  • FANS_Nader_Al_Theeb
    FANS_Nader_Al_Theeb
    3 Posts

    Re: Clear Models and Memory leak

    ‏2013-07-01T17:44:00Z  

    What version of CPLEX do you use? How do you assess that memory is leaked? Do you have a tool that actually reports the leakage?

    How exactly do you modify model2? Maybe the code that performs the modification leaks memory?

    Could you please print out env.getMemoryUsage() at the beginning of each iteration and check whether this value increases?

    How large is the amount of memory that is leaked? I have a hard time believing that a small memory leak will result in a 25 second slowdown. Do you apply the exact same modifications in each iteration or may the slowdown be due to the fact that you are solving different models?


    Dear Daniel
    Thanks for your replay

    I'm using CPLEX-Concert 12.2 in Linux machine, I was observing the memory usage in other terminal using top command. Now, I'm using env.getMemoryUsage() as you suggest, I got the following results (iteration number        memory usage)

    ------------------------------------------------------

    0         memory usage =  394496256
    1         memory usage =  826231320
    2         memory usage =  1266370088
    3         memory usage =  1672823352
    4         memory usage =  2146386416
    5         memory usage =  2552839968
    6         memory usage =  2959490992
    7         memory usage =  3365945552
    8         memory usage =  3906879480
    9         memory usage =  4313135200
    10         memory usage =  4719785576
    11         memory usage =  5126437104
    12         memory usage =  5532693328

    ------------------------------------------------------

    I modify model 2 by enforcing some variables to take a value and keep others free, this modifications are the same in all iterations.

    Is there anything else can be done to solve this problem? For example, Do I need to clear variables or change arrangement of extract, clear, and end commands in my code?

    Thanks
     

  • Lessi
    Lessi
    5 Posts

    Re: Clear Models and Memory leak

    ‏2013-07-31T20:10:38Z  


    Dear Daniel
    Thanks for your replay

    I'm using CPLEX-Concert 12.2 in Linux machine, I was observing the memory usage in other terminal using top command. Now, I'm using env.getMemoryUsage() as you suggest, I got the following results (iteration number        memory usage)

    ------------------------------------------------------

    0         memory usage =  394496256
    1         memory usage =  826231320
    2         memory usage =  1266370088
    3         memory usage =  1672823352
    4         memory usage =  2146386416
    5         memory usage =  2552839968
    6         memory usage =  2959490992
    7         memory usage =  3365945552
    8         memory usage =  3906879480
    9         memory usage =  4313135200
    10         memory usage =  4719785576
    11         memory usage =  5126437104
    12         memory usage =  5532693328

    ------------------------------------------------------

    I modify model 2 by enforcing some variables to take a value and keep others free, this modifications are the same in all iterations.

    Is there anything else can be done to solve this problem? For example, Do I need to clear variables or change arrangement of extract, clear, and end commands in my code?

    Thanks
     

    I usually clear variables, constraints, etc.

     

  • DanielJunglas
    DanielJunglas
    119 Posts

    Re: Clear Models and Memory leak

    ‏2013-08-01T06:02:53Z  


    Dear Daniel
    Thanks for your replay

    I'm using CPLEX-Concert 12.2 in Linux machine, I was observing the memory usage in other terminal using top command. Now, I'm using env.getMemoryUsage() as you suggest, I got the following results (iteration number        memory usage)

    ------------------------------------------------------

    0         memory usage =  394496256
    1         memory usage =  826231320
    2         memory usage =  1266370088
    3         memory usage =  1672823352
    4         memory usage =  2146386416
    5         memory usage =  2552839968
    6         memory usage =  2959490992
    7         memory usage =  3365945552
    8         memory usage =  3906879480
    9         memory usage =  4313135200
    10         memory usage =  4719785576
    11         memory usage =  5126437104
    12         memory usage =  5532693328

    ------------------------------------------------------

    I modify model 2 by enforcing some variables to take a value and keep others free, this modifications are the same in all iterations.

    Is there anything else can be done to solve this problem? For example, Do I need to clear variables or change arrangement of extract, clear, and end commands in my code?

    Thanks
     

    Sorry for the late reply. I missed your answer :-(

    It indeed looks like you or CPLEX (or both) are leaking memory here. Does the problem persist if you skip the "modify model2" step in the loop (that, solve the same model in each loop)?

  • FANS_Nader_Al_Theeb
    FANS_Nader_Al_Theeb
    3 Posts

    Re: Clear Models and Memory leak

    ‏2013-08-01T07:28:58Z  

    Sorry for the late reply. I missed your answer :-(

    It indeed looks like you or CPLEX (or both) are leaking memory here. Does the problem persist if you skip the "modify model2" step in the loop (that, solve the same model in each loop)?

    Thanks all for replays

    I found what is the problem,

    originally I use  (IloEnv env;) before the for loop and (env.end();) after the for loop

    Now, I use (IloEnv env;) at the beginning of each iteration (inside for loop) and (env.end();) before the end of for loop

    I did this by chance and I don't know what the idea behind this, if you would explain why

    Thanks
    Nader
     

  • PaulRubin
    PaulRubin
    50 Posts

    Re: Clear Models and Memory leak

    ‏2013-08-01T19:21:44Z  

    Thanks all for replays

    I found what is the problem,

    originally I use  (IloEnv env;) before the for loop and (env.end();) after the for loop

    Now, I use (IloEnv env;) at the beginning of each iteration (inside for loop) and (env.end();) before the end of for loop

    I did this by chance and I don't know what the idea behind this, if you would explain why

    Thanks
    Nader
     

    Some operations can produce unintended persistent byproducts. I think += does this, although I'm not positive about that. Since C++ does not have automatic garbage collection, these byproducts can accumulate (and consume memory). Ending the environment to which they belong clears them out. I don't know if your code produced any byproducts, but it is one possible explanation.

    Paul

  • DanielJunglas
    DanielJunglas
    119 Posts

    Re: Clear Models and Memory leak

    ‏2013-08-02T05:27:04Z  
    • PaulRubin
    • ‏2013-08-01T19:21:44Z

    Some operations can produce unintended persistent byproducts. I think += does this, although I'm not positive about that. Since C++ does not have automatic garbage collection, these byproducts can accumulate (and consume memory). Ending the environment to which they belong clears them out. I don't know if your code produced any byproducts, but it is one possible explanation.

    Paul

    Paul,

    it is usually the other way around. In C++-Concert a statement like 'expr += ...' will usually avoid the creation of "byproducts" while something like 'expr = expr + ...' will leak memory.

    Nader,

    if calling env.end() fixes your problem then almost certainly the old code was missing appropriate end() statements in some places. Like Paul said, it is not always blatantly obvious on which objects on has to call end() :-(