Topic
  • 23 replies
  • Latest Post - ‏2013-02-25T08:57:24Z by SystemAdmin
SystemAdmin
SystemAdmin
1883 Posts

Pinned topic Mem issue, each IloOplModel takes 13 MB

‏2013-02-12T15:00:38Z |
Hi

I'm implementing a column geneartion based algorithm as described in:

https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14939755&#14939755

As described this requires that i have a large number of IloOplModel objects existing at the same time, which gives me a speed up, but it really eats memory. The flow for creating the objects is as:
var subOplArrays = new Array(numVessels);
var subCplexArrays = new Array(numVessels);

...

for(var ves in MasterDataElmts.Vessels) {
subCplexArrays[i] = new IloCplex();
subOplArrays[i] = new IloOplModel(subDef, subCplexArrays[i]);

...
subOplArrays[i].addDataSource(subData);
subOplArrays[i].generate();
subOplArrays[i].settings.mainEndEnabled = true;
subCplexArrays[i].epgap = subProblemGap;
MasterDataElmts.bunkerPlanUniqueId = MasterDataElmts.bunkerPlanUniqueId +1;
writeln("Model object created for vessel: "+i);
i++;
}

this setup restricts me to have at most 100 subproblems due to mem issues, as each model takes about 13 MB. And i'd really like to go much higher.

So my question, what can i do to reduce memery load ? I've tried to make sure that subData only holds the absolutely necessary, and i cant reduce the model it self more than ive done.

Looking forward to any hints!

Christian
Updated on 2013-02-25T08:57:24Z at 2013-02-25T08:57:24Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-12T19:24:03Z  
    Why don't you reuse the same sub model objects?
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T06:56:58Z  
    Why don't you reuse the same sub model objects?
    subData is different for each sub problem and i would have to recreate the whole matrix, objective and rhs. So it would be better to destroy and recreate.
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T09:40:01Z  
    subData is different for each sub problem and i would have to recreate the whole matrix, objective and rhs. So it would be better to destroy and recreate.
    You can call generate() for each sub problem instance (on the same model object), and only update the input data... I think I don't understand what is your problem, could you please clarify
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T10:11:32Z  
    You can call generate() for each sub problem instance (on the same model object), and only update the input data... I think I don't understand what is your problem, could you please clarify
    Sorry i'll elaborate some more.

    Originally i did as you suggested, calling generate for each subproblem in each iteration.

    But with this approach the "generate()" call ends up using 90 % or more of the computational time, as each of my subproblems are usually easily solved. So the whole idea of this different approach is to avoid the generate calls, by simply updating the changed dual values in the subproblems, as all other values are the same from iteration to iteration.

    The problem is now that all these subproblems eat all the mem, so it does scale to sufficiently large instances.
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T11:11:17Z  
    Sorry i'll elaborate some more.

    Originally i did as you suggested, calling generate for each subproblem in each iteration.

    But with this approach the "generate()" call ends up using 90 % or more of the computational time, as each of my subproblems are usually easily solved. So the whole idea of this different approach is to avoid the generate calls, by simply updating the changed dual values in the subproblems, as all other values are the same from iteration to iteration.

    The problem is now that all these subproblems eat all the mem, so it does scale to sufficiently large instances.
    In the code above I see a call to sub problem generate() inside the loop, so how did you reduce the number of calls there?
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T12:02:13Z  
    In the code above I see a call to sub problem generate() inside the loop, so how did you reduce the number of calls there?
    As stated, i've only posted the part of the code creating the object. If you looked at the post of above link you'll get an idea of what the modyfing and solving code looks like.

    Cheers, Chrsitian
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T12:24:29Z  
    As stated, i've only posted the part of the code creating the object. If you looked at the post of above link you'll get an idea of what the modyfing and solving code looks like.

    Cheers, Chrsitian
    I don't understand your question in 5 minutes given all the data. There is a good chance I am not smart enough :), hope someone helps you.
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T14:01:35Z  
    I don't understand your question in 5 minutes given all the data. There is a good chance I am not smart enough :), hope someone helps you.
    Thanks ;) Sorry I cant explain better. But basically i need each IloOplModel to take up much less mem than its doing. So I'm looking for any way to do this ? Maybe it can be stored in sparse form ?
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T14:23:16Z  
    Thanks ;) Sorry I cant explain better. But basically i need each IloOplModel to take up much less mem than its doing. So I'm looking for any way to do this ? Maybe it can be stored in sparse form ?
    May be after completing the sub model run, store the required result somewhere and drop/reuse the big object.
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-13T17:55:45Z  
    May be after completing the sub model run, store the required result somewhere and drop/reuse the big object.
    No. That would resemble the original approach.
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-14T14:23:31Z  
    No. That would resemble the original approach.
    Hi Christian,

    One way could be to decrease the number of CPLEX instance.
    I can see that you create an CPLEX instance for each subproblem.

    
    
    
    for(var ves in MasterDataElmts.Vessels) 
    { ..... subCplexArrays[i] = 
    
    new IloCplex(); ....
    }
    


    You can use only one instance of CPLEX.
    If your model solve fast on only 1 or 2 threads you can have several CPLEX instance with a fix number of thread according the number of core of your machine.

    then you can do something like:
    
    CplexArrays[i] = 
    
    new Array() 
    
    for(var i in 1:nbcore) 
    {CplexArrays[i] = 
    
    new  IloCplex();
    }   
    
    for(var ves in MasterDataElmts.Vessels) 
    { ..... subOplArrays[i] = 
    
    new IloOplModel(subDef, CplexArrays[i%CplexArrays.length]); ....
    }
    


    you can find information about loding Model structure in the CPLEX help here:
    Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/uss_langtut_fc_multiple_search_6.html

    if you are part of IBM Academic Initiative you can access to teaching material about CPLEX and OPL.
    the CPLEX_OPL.zip is for you especially the lesson 18: Performance Tuning
    Linkhttps://www14.software.ibm.com/webapp/iwm/web/reg/download.do?source=ai-course-bao&S_PKG=2VPS0&lang=en#
    I hope it helps

    best regards
    Stephen Hall
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-14T15:10:14Z  
    Hi Christian,

    One way could be to decrease the number of CPLEX instance.
    I can see that you create an CPLEX instance for each subproblem.

    <pre class="jive-pre"> for(var ves in MasterDataElmts.Vessels) { ..... subCplexArrays[i] = new IloCplex(); .... } </pre>

    You can use only one instance of CPLEX.
    If your model solve fast on only 1 or 2 threads you can have several CPLEX instance with a fix number of thread according the number of core of your machine.

    then you can do something like:
    <pre class="jive-pre"> CplexArrays[i] = new Array() for(var i in 1:nbcore) {CplexArrays[i] = new IloCplex(); } for(var ves in MasterDataElmts.Vessels) { ..... subOplArrays[i] = new IloOplModel(subDef, CplexArrays[i%CplexArrays.length]); .... } </pre>

    you can find information about loding Model structure in the CPLEX help here:
    Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/uss_langtut_fc_multiple_search_6.html

    if you are part of IBM Academic Initiative you can access to teaching material about CPLEX and OPL.
    the CPLEX_OPL.zip is for you especially the lesson 18: Performance Tuning
    Linkhttps://www14.software.ibm.com/webapp/iwm/web/reg/download.do?source=ai-course-bao&S_PKG=2VPS0&lang=en#
    I hope it helps

    best regards
    Stephen Hall
    Hi Again,

    you can try to reduce the number of thread of CPLEX. It will reduce the memory usage.

    the second point is to isolate a subproblem in .mod and .dat. After you can run it and analyze it in the console. You would be able to find where the memory is used in the subproblem and reduce it size by modifying the model. Tuning example are:
    • use tuples instead of array
    • use Range instead of Set
    • use internal data instead of external data
    • use integer id key instead of string
    • ..... find other hint in CPLEX_OPL training book from Academic Initiative. see previous post

    after modification you can replicate the changes into your original model.

    let me know what are the results.

    Stephen Hall
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-15T06:25:06Z  
    Hi Christian,

    One way could be to decrease the number of CPLEX instance.
    I can see that you create an CPLEX instance for each subproblem.

    <pre class="jive-pre"> for(var ves in MasterDataElmts.Vessels) { ..... subCplexArrays[i] = new IloCplex(); .... } </pre>

    You can use only one instance of CPLEX.
    If your model solve fast on only 1 or 2 threads you can have several CPLEX instance with a fix number of thread according the number of core of your machine.

    then you can do something like:
    <pre class="jive-pre"> CplexArrays[i] = new Array() for(var i in 1:nbcore) {CplexArrays[i] = new IloCplex(); } for(var ves in MasterDataElmts.Vessels) { ..... subOplArrays[i] = new IloOplModel(subDef, CplexArrays[i%CplexArrays.length]); .... } </pre>

    you can find information about loding Model structure in the CPLEX help here:
    Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/uss_langtut_fc_multiple_search_6.html

    if you are part of IBM Academic Initiative you can access to teaching material about CPLEX and OPL.
    the CPLEX_OPL.zip is for you especially the lesson 18: Performance Tuning
    Linkhttps://www14.software.ibm.com/webapp/iwm/web/reg/download.do?source=ai-course-bao&S_PKG=2VPS0&lang=en#
    I hope it helps

    best regards
    Stephen Hall
    Hi Stephen,

    I'm not sure i can do this, as i update the objective in each cplex subproblem instance, in each iteration, as:

    subCplexArrays[j].setObjCoef( subOplArrays[j].purchaseContract[c], c.price + dualAdjust);

    as i cant use:

    subOplArrays[j].getObjective().setCoef(subOplArrays[j].purchaseContract[c], c.price + dualAdjust);

    as my problem is a MIP (and thus non linear). So i would end up calling Generate() in each iteration again, to ensure that model and cplex matched.
    I have reduced number of threads to 1.
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-15T08:36:14Z  
    Hi Stephen,

    I'm not sure i can do this, as i update the objective in each cplex subproblem instance, in each iteration, as:

    subCplexArrays[j].setObjCoef( subOplArrays[j].purchaseContract[c], c.price + dualAdjust);

    as i cant use:

    subOplArrays[j].getObjective().setCoef(subOplArrays[j].purchaseContract[c], c.price + dualAdjust);

    as my problem is a MIP (and thus non linear). So i would end up calling Generate() in each iteration again, to ensure that model and cplex matched.
    I have reduced number of threads to 1.
    Hi Christian,

    Sorry you are right for the use of only one instance of CPLEX as when you generate you populate the CPLEX instance with one problem.

    But you can still try:
    There is an option to tell CPLEX to limit memory usage if possible: Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.cplex.help/CPLEX/Parameters_reference/topics/MemoryEmphasis.html

    look in the doc for some hint
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-15T12:21:04Z  
    Hi Christian,

    Sorry you are right for the use of only one instance of CPLEX as when you generate you populate the CPLEX instance with one problem.

    But you can still try:
    There is an option to tell CPLEX to limit memory usage if possible: Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r4/topic/ilog.odms.cplex.help/CPLEX/Parameters_reference/topics/MemoryEmphasis.html

    look in the doc for some hint
    Hi Stephen,

    Thnanks for your constructive answers.

    I am already running with 1 thread and i've set the reduce memoty switch (in .opl file, i guess thats sufficient?). These helped a bit.

    But its not my perception that there is a massive memory usage in cplex itself. When stepping through the code, its the "generate()" calls of the ilog model that eats the majority of the mem, so i think the solution should be found in opl somehow.

    Thanks, Chr
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-15T13:53:39Z  
    Hi Stephen,

    Thnanks for your constructive answers.

    I am already running with 1 thread and i've set the reduce memoty switch (in .opl file, i guess thats sufficient?). These helped a bit.

    But its not my perception that there is a massive memory usage in cplex itself. When stepping through the code, its the "generate()" calls of the ilog model that eats the majority of the mem, so i think the solution should be found in opl somehow.

    Thanks, Chr
    Hi Christian,

    I guess you are right and we are back to your 1st assessment and tittle of this thread; minimizing model memory

    For that you need to investigate were the memory is used in your Submodel.
    1. isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)
    2. Run the subproblem
    3. use the profiler to see where the memory is used. Tutorial of the profiler here: Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplide/topics/opl_ide_profil.html

    for example the use of tuples normally decrease the memory usage.
    the internal data of an objet are loaded in memory only if needed, while external data are all loaded from start.

    I hope you will find some solution to decrease the model size.
    keep me posted

    MVH.
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-18T13:37:08Z  
    Hi Christian,

    I guess you are right and we are back to your 1st assessment and tittle of this thread; minimizing model memory

    For that you need to investigate were the memory is used in your Submodel.
    1. isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)
    2. Run the subproblem
    3. use the profiler to see where the memory is used. Tutorial of the profiler here: Linkhttp://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplide/topics/opl_ide_profil.html

    for example the use of tuples normally decrease the memory usage.
    the internal data of an objet are loaded in memory only if needed, while external data are all loaded from start.

    I hope you will find some solution to decrease the model size.
    keep me posted

    MVH.
    Hi Stephen,

    I've added a profiler dump of the full problem (master and subproblem), but it lacks details to really unxderstand the issue.

    So you're right that I need to: "isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)", but how to ? I've looked through the help and documentation, but hasnt been able to find any ?

    Thanks, Christian
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-18T14:18:51Z  
    Hi Stephen,

    I've added a profiler dump of the full problem (master and subproblem), but it lacks details to really unxderstand the issue.

    So you're right that I need to: "isolate one iteration of a subproblem in .mod and .dat (in your master run you can extract the internal and external data in a .dat)", but how to ? I've looked through the help and documentation, but hasnt been able to find any ?

    Thanks, Christian
    Hi Christian,
    You can have a look at this topic:Performance and memory usage
    http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/opl_languser_perf.html

    You have already the .mod of your subproblem in your project.
    the only thing that you need is the .dat. below is how to do it:
    CPLEX Help:http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/refjsopl/html/IloOplModel.html

    
    
    
    if (
    
    true) 
    { 
    //dump OPL data source var dumpFileName = 
    "pscextdata.dat"; writeln(
    "dumping data to : ", dumpFileName); var edf= 
    
    new IloOplOutputFile(dumpFileName); var ed=thisOplModel.printExternalData(
    "PSC data"); edf.write(ed); edf.close(); 
    }
    


    you have the same for internal Data : "printExternalData("title")"
    Def of Internal and External Data http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplinterfaces/topics/opl_interfaces_work_memory.html

    I have seen another way to control save the entire model:
    "exportModel(path)" Exports the CPLEX model to a file.
    "importModel(path)" Import the CPLEX model from a file.
    CPLEX help: http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/refjsopl/html/IloCplex.html
    I don't know if it would be faster then generate() but it is an option.
    good luck
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-20T09:58:28Z  
    Hi Christian,
    You can have a look at this topic:Performance and memory usage
    http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/opllanguser/topics/opl_languser_perf.html

    You have already the .mod of your subproblem in your project.
    the only thing that you need is the .dat. below is how to do it:
    CPLEX Help:http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/refjsopl/html/IloOplModel.html

    <pre class="jive-pre"> if ( true) { //dump OPL data source var dumpFileName = "pscextdata.dat"; writeln( "dumping data to : ", dumpFileName); var edf= new IloOplOutputFile(dumpFileName); var ed=thisOplModel.printExternalData( "PSC data"); edf.write(ed); edf.close(); } </pre>

    you have the same for internal Data : "printExternalData("title")"
    Def of Internal and External Data http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplinterfaces/topics/opl_interfaces_work_memory.html

    I have seen another way to control save the entire model:
    "exportModel(path)" Exports the CPLEX model to a file.
    "importModel(path)" Import the CPLEX model from a file.
    CPLEX help: http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/refjsopl/html/IloCplex.html
    I don't know if it would be faster then generate() but it is an option.
    good luck
    Hi Stephen,

    Thanks, It worked. I succesfully exported the internal and external data to .Dat files. Following i've been able to run the subproblems alone and review the profiling, pls refer to attached for details.

    But im not able to get much value out of this, the Mem usage is reported at 6.7 MB, a bit less than my experience from the whole program execution, but comparable. Going through the high mem users of these 6.7 i see: "CPLEX MIP Optimization", EXTRACT and LOAD_MODEL posts. But none of these seem to indicate that my model is mem-inefficiently constructed or I'm putting too much (unnneeded) data into it ? So am i looking at it the right way ? I cant find the insight that should help me lower the mem usage.

    I still think it should be possible to reduce mem usage, if you look at the tab "Engine Log", you'll see that the model is quite small, only 3151 non-zeros, i've cant see why this should take up 6.7 MB ?

    Cheers, Christian
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-20T11:18:08Z  
    Hi Stephen,

    Thanks, It worked. I succesfully exported the internal and external data to .Dat files. Following i've been able to run the subproblems alone and review the profiling, pls refer to attached for details.

    But im not able to get much value out of this, the Mem usage is reported at 6.7 MB, a bit less than my experience from the whole program execution, but comparable. Going through the high mem users of these 6.7 i see: "CPLEX MIP Optimization", EXTRACT and LOAD_MODEL posts. But none of these seem to indicate that my model is mem-inefficiently constructed or I'm putting too much (unnneeded) data into it ? So am i looking at it the right way ? I cant find the insight that should help me lower the mem usage.

    I still think it should be possible to reduce mem usage, if you look at the tab "Engine Log", you'll see that the model is quite small, only 3151 non-zeros, i've cant see why this should take up 6.7 MB ?

    Cheers, Christian
    Hi Christian,

    good new you were able to run the subproblem alone.

    But you missed to attache the file, please send it
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-20T16:24:17Z  
    Hi Christian,

    good new you were able to run the subproblem alone.

    But you missed to attache the file, please send it
    Hi Again,

    from your file I can see that some memory are allocated to EXTRACT, and I am suppose is to extract some constraints.

    you can avoid that by:
    1: not labeling the constraint
    2: not extracting the data for the profiler http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplide/topics/opl_ide_profil_examin_turn.html
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-21T08:54:36Z  
    Hi Again,

    from your file I can see that some memory are allocated to EXTRACT, and I am suppose is to extract some constraints.

    you can avoid that by:
    1: not labeling the constraint
    2: not extracting the data for the profiler http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplide/topics/opl_ide_profil_examin_turn.html
    Hi Stephen,

    I've tried this. Did very little to change mem consumption. Or actually nothing ;) I can scale to the same problem sizes with and without the profiling, although the mem consumption is a little bit less

    Thanks, Christian
  • SystemAdmin
    SystemAdmin
    1883 Posts

    Re: Mem issue, each IloOplModel takes 13 MB

    ‏2013-02-25T08:57:24Z  
    Hi Again,

    from your file I can see that some memory are allocated to EXTRACT, and I am suppose is to extract some constraints.

    you can avoid that by:
    1: not labeling the constraint
    2: not extracting the data for the profiler http://pic.dhe.ibm.com/infocenter/cosinfoc/v12r5/topic/ilog.odms.ide.help/OPL_Studio/usroplide/topics/opl_ide_profil_examin_turn.html
    I've talked with Stephen and Alex Fleischer and they've resolved my issue. I ended up:

    Having one subproblem at a time, creating the models when needed, solving and destroying afterwards. With a lean model and using the cmd version as "oplrun -deploy" the generate() calls are fast and each iteration runs smoothly without taking up much mem.
    Thanks for the help.

    Christian