Topic
  • 17 replies
  • Latest Post - ‏2013-03-06T06:02:58Z by SystemAdmin
SystemAdmin
SystemAdmin
7929 Posts

Pinned topic The difference between using concert technology of C++ and Cplex directly.

‏2013-02-25T04:35:10Z |
Hello, this is from a beginner of Cplex.

I am using concert technology for C++ of Cplex to solve my MIP model.
And it returns out of memory after solving it by this C++ collaborate with Cplex.

However, I tried once to read the .lp file directly by using Cplex 12.4 as well. It can be solved without any memory problem in this way.

So my question is, what's the difference between this two methods? I use the same computer and identical MIP model...

Need help to figure out the differences, please.
Thank you! ^^
Hong
Updated on 2013-03-06T06:02:58Z at 2013-03-06T06:02:58Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-25T06:09:31Z  
    The Concert C++ layer keeps a copy of the model. For large models this may result in a significant increase of memory requirements.
    When you say your "read in the .lp directly", does that mean that you also read the LP file into Concert or did you build the model in Concert, export it to an LP file and then read that LP file? In the latter case things are expected to be different. Export/import to/from an LP file may change the ordering of columns and rows and may introduce floating point round off. You should repeat the experiment using SAV files instead (just change extension .lp to .sav).
    Since you say that the solution process runs out of memory during the solve (could you show us a log file) this probably means that the tree got too large. You can instruct CPLEX to swap out parts of the tree to disk. To do this you have to change the following parameters:
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-25T09:52:40Z  
    The Concert C++ layer keeps a copy of the model. For large models this may result in a significant increase of memory requirements.
    When you say your "read in the .lp directly", does that mean that you also read the LP file into Concert or did you build the model in Concert, export it to an LP file and then read that LP file? In the latter case things are expected to be different. Export/import to/from an LP file may change the ordering of columns and rows and may introduce floating point round off. You should repeat the experiment using SAV files instead (just change extension .lp to .sav).
    Since you say that the solution process runs out of memory during the solve (could you show us a log file) this probably means that the tree got too large. You can instruct CPLEX to swap out parts of the tree to disk. To do this you have to change the following parameters:
    Thank you DanielJunglas!

    First, read .lp means the latter case that I export the model as a .lp file which is built by Cplex concert technology for C++. Then I use Cplex optimizer to read this model(.lp file) and input command "opt" to solve this directly.

    As for the memory problem, you are right, it's because of the tree size. And I will change the parameters soon.

    Attached is my MIP model. Do you think this model size could leads to the different results?
    Did you mean if I export the model into a SAV file instead of lp file by Concert C++, and do "read" and "opt" by Cplex optimizer 12.4 directly, the solution result will be out of memory (same with Concert C++)?
  • T_O
    T_O
    448 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-25T10:53:02Z  
    Thank you DanielJunglas!

    First, read .lp means the latter case that I export the model as a .lp file which is built by Cplex concert technology for C++. Then I use Cplex optimizer to read this model(.lp file) and input command "opt" to solve this directly.

    As for the memory problem, you are right, it's because of the tree size. And I will change the parameters soon.

    Attached is my MIP model. Do you think this model size could leads to the different results?
    Did you mean if I export the model into a SAV file instead of lp file by Concert C++, and do "read" and "opt" by Cplex optimizer 12.4 directly, the solution result will be out of memory (same with Concert C++)?
    Could you, as Daniel proposed, repeat your experiments with .sav-files. Some coefficients in your .lp-file might cause numerical roundoffs (e.g. 9.15478038787842).

    How much RAM do you have?

    Best regards,
    Thomas
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-25T15:49:38Z  
    Thank you DanielJunglas!

    First, read .lp means the latter case that I export the model as a .lp file which is built by Cplex concert technology for C++. Then I use Cplex optimizer to read this model(.lp file) and input command "opt" to solve this directly.

    As for the memory problem, you are right, it's because of the tree size. And I will change the parameters soon.

    Attached is my MIP model. Do you think this model size could leads to the different results?
    Did you mean if I export the model into a SAV file instead of lp file by Concert C++, and do "read" and "opt" by Cplex optimizer 12.4 directly, the solution result will be out of memory (same with Concert C++)?
    Sorry, I was unclear about the issue with the .lp file.
    As I said, exporting the model to an .lp file may reorder the rows/columns of the problem matrix and may also result in numerical round off (very likely in your case as Thomas noted).
    If there is numerical round off in the coefficients then you are actually solving a different problem in the interactive optimizer than you did in C++. So it is no surprise that results/performance are different.
    Even if there is no round off for the coefficients and only the rows/columns are reordered, then you are still solving a different problem in the interactive and again different results/performance are expected.
    The SAV file format avoids all these issues and when using a SAV file you should see the same behavior in the interactive optimizer as you saw in C++.

    >Attached is my MIP model. Do you think this model size could leads to the different results?
    >
    It is not the size of the model. As Thomas said, your .lp file contains numbers like 2.24722051620483. That is most probably not exactly what was in the C++ model but was instead rounded/truncated when output to the .lp file.
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-26T03:40:37Z  
    • T_O
    • ‏2013-02-25T10:53:02Z
    Could you, as Daniel proposed, repeat your experiments with .sav-files. Some coefficients in your .lp-file might cause numerical roundoffs (e.g. 9.15478038787842).

    How much RAM do you have?

    Best regards,
    Thomas
    Thank you Thomas!
    I repeat my experiment with .sav file instead of .lp file.
    However, the problem is still solved in this way.
    My RAM is 8.00G, r there any other possibilities to explain this kind of difference?

    Sincerely,
    Tarcy
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-26T03:51:59Z  
    Sorry, I was unclear about the issue with the .lp file.
    As I said, exporting the model to an .lp file may reorder the rows/columns of the problem matrix and may also result in numerical round off (very likely in your case as Thomas noted).
    If there is numerical round off in the coefficients then you are actually solving a different problem in the interactive optimizer than you did in C++. So it is no surprise that results/performance are different.
    Even if there is no round off for the coefficients and only the rows/columns are reordered, then you are still solving a different problem in the interactive and again different results/performance are expected.
    The SAV file format avoids all these issues and when using a SAV file you should see the same behavior in the interactive optimizer as you saw in C++.

    >Attached is my MIP model. Do you think this model size could leads to the different results?
    >
    It is not the size of the model. As Thomas said, your .lp file contains numbers like 2.24722051620483. That is most probably not exactly what was in the C++ model but was instead rounded/truncated when output to the .lp file.
    To DanielJunglas, I'm sorry to made u confused.
    As I replied to Thomas, I repeat the procedure but with .sav file.
    The result is still different with concert C++.
    But for the coefficients, you are correct that round off coefficients are exist.

    Thank you Daniel!^^
  • T_O
    T_O
    448 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-26T07:45:38Z  
    Thank you Thomas!
    I repeat my experiment with .sav file instead of .lp file.
    However, the problem is still solved in this way.
    My RAM is 8.00G, r there any other possibilities to explain this kind of difference?

    Sincerely,
    Tarcy
    Well, I can now only guess where the problem comes from. Memory might be the problem. Could you post your C++-code? Maybe there is a memory leak in your code? You might also try to use the C callable library instead of concert.

    Best regards,
    Thomas
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-27T06:18:14Z  
    • T_O
    • ‏2013-02-26T07:45:38Z
    Well, I can now only guess where the problem comes from. Memory might be the problem. Could you post your C++-code? Maybe there is a memory leak in your code? You might also try to use the C callable library instead of concert.

    Best regards,
    Thomas
    Thank you Thomas!
    Did u mean that the experiment result should be no difference between using Cplex optimizer 12.4 directly and Cplex collaborate with C++?
    If so, I will check my code as u suggested to figure it out.
    I'm really appreciate for your help!
    Once I figure it out, I will post my experienc here.

    Best regards,
    Tracy
  • T_O
    T_O
    448 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-27T06:53:27Z  
    Thank you Thomas!
    Did u mean that the experiment result should be no difference between using Cplex optimizer 12.4 directly and Cplex collaborate with C++?
    If so, I will check my code as u suggested to figure it out.
    I'm really appreciate for your help!
    Once I figure it out, I will post my experienc here.

    Best regards,
    Tracy
    I mean that I would not expect any remarkable differences betwwen using the C callable library and the interactive optimizier if your code does not use a lot of memory for your own data and if it does not have any memory leaks.

    Let me give you an example in pseudocode:

    create_and_fill_data_vectors
    create_cplex
    copy_data_from_vectors_to_cplex
    solve_with_cplex

    can use a lot more memory than

    create_cplex
    {
    create_and_fill_data_vectors
    copy_data_from_vectors_to_cplex
    }
    solve_with_cplex

    You see the difference?

    Best regards,
    Thomas
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-27T13:56:52Z  
    Thank you Thomas!
    Did u mean that the experiment result should be no difference between using Cplex optimizer 12.4 directly and Cplex collaborate with C++?
    If so, I will check my code as u suggested to figure it out.
    I'm really appreciate for your help!
    Once I figure it out, I will post my experienc here.

    Best regards,
    Tracy
    Could you please post log files for the two different solves?
    Maybe then we can spot what the difference is between your two runs.
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-03-05T01:56:04Z  
    Could you please post log files for the two different solves?
    Maybe then we can spot what the difference is between your two runs.
    Sorry for the late reply.
    Attached are the log files. It look like the memory sizes for saving the tree are different in C++ program and Cplex solver 12.4. But I just use the default parameters settings of Cplex 12.4 in C++ code (I didn't change any parameters in my code).

    Also, the tree sizes are different in "cplexlog-SAV" and "cplexlog-LP" these two files. Do u think the differences are because of round off coefficients and column changes?

    Sincerely,
    Tracy
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-03-05T01:59:05Z  
    Could you please post log files for the two different solves?
    Maybe then we can spot what the difference is between your two runs.
    Sorry for previous miss attach.
    Correct one is attached here.

    Thank you,
    Tracy
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-03-05T02:07:47Z  
    • T_O
    • ‏2013-02-27T06:53:27Z
    I mean that I would not expect any remarkable differences betwwen using the C callable library and the interactive optimizier if your code does not use a lot of memory for your own data and if it does not have any memory leaks.

    Let me give you an example in pseudocode:

    create_and_fill_data_vectors
    create_cplex
    copy_data_from_vectors_to_cplex
    solve_with_cplex

    can use a lot more memory than

    create_cplex
    {
    create_and_fill_data_vectors
    copy_data_from_vectors_to_cplex
    }
    solve_with_cplex

    You see the difference?

    Best regards,
    Thomas
    Thank u for your reply Thomas.
    Actually, I think my code is the first case of your two pseudo code models.
    I attached a part of my code here. I think it can briefly explain my code structure.

    Could please explain a little bit why the first pseudo code will use much more memory than the second one?

    Sincerely,
    Tracy

    Attachments

  • T_O
    T_O
    448 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-03-05T06:59:34Z  
    Thank u for your reply Thomas.
    Actually, I think my code is the first case of your two pseudo code models.
    I attached a part of my code here. I think it can briefly explain my code structure.

    Could please explain a little bit why the first pseudo code will use much more memory than the second one?

    Sincerely,
    Tracy
    If your data consumes a lot of memory, then this is freed in the second example, so CPLEX can use this memory for solving the problem. But I don't really think this is the root of your problem as your model is quite small. Nevertheless, I don't know how large stop_list and OD are. Maybe you allocated unnecessary memory?

    As the log files differ, it might be that one instance needs a lot more memory than the other. So I think we should try to find out why they differ. Maybe one of the CPLEX-People could have a look at it.

    Best regards,
    Thomas
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-03-05T07:56:37Z  
    • T_O
    • ‏2013-03-05T06:59:34Z
    If your data consumes a lot of memory, then this is freed in the second example, so CPLEX can use this memory for solving the problem. But I don't really think this is the root of your problem as your model is quite small. Nevertheless, I don't know how large stop_list and OD are. Maybe you allocated unnecessary memory?

    As the log files differ, it might be that one instance needs a lot more memory than the other. So I think we should try to find out why they differ. Maybe one of the CPLEX-People could have a look at it.

    Best regards,
    Thomas
    For the problem I attached, there are 13 stops in stop_list.
    Consequently, OD is a 13*13 matrix.
    If there is no other possibilities, I guess I will check my code again to in case of any inappropriate memory allocate.

    Thank you so much Thomas!^^
    Tracy
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-03-05T08:23:22Z  
    Sorry for previous miss attach.
    Correct one is attached here.

    Thank you,
    Tracy
    OK, it is hard to tell what is going different here without having the C++ code at hands. If you compare the SAV log and the C++ log then you can see that things are exactly the same until node 35080. In particular the statistics for the presolved model etc. are all the same. This is a strong indication that the in-memory models that are solved are the same.
    One thing that can change the solution paths in low-memory situations are optional sub-algorithm that fail due to out of memory. For example, if CPLEX runs a heuristic then this heuristic may run out of memory in one run but not in the other. Since heuristics are optional, out of memory in heuristics is usually not considered an error and optimization continues. To make sure that sporadic out of memory is not the issue here could you repeat the experiment (SAV file and C++ solve) using node files in both cases? Set WorkMem to 200 and NodeFileInd to 2 ("set workmem 200" and "set mip strat file 2" in the interactive).
    Another thing to try would be to run without any node files (NodeFileInd set to 0).
  • SystemAdmin
    SystemAdmin
    7929 Posts

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-03-06T06:02:58Z  
    OK, it is hard to tell what is going different here without having the C++ code at hands. If you compare the SAV log and the C++ log then you can see that things are exactly the same until node 35080. In particular the statistics for the presolved model etc. are all the same. This is a strong indication that the in-memory models that are solved are the same.
    One thing that can change the solution paths in low-memory situations are optional sub-algorithm that fail due to out of memory. For example, if CPLEX runs a heuristic then this heuristic may run out of memory in one run but not in the other. Since heuristics are optional, out of memory in heuristics is usually not considered an error and optimization continues. To make sure that sporadic out of memory is not the issue here could you repeat the experiment (SAV file and C++ solve) using node files in both cases? Set WorkMem to 200 and NodeFileInd to 2 ("set workmem 200" and "set mip strat file 2" in the interactive).
    Another thing to try would be to run without any node files (NodeFileInd set to 0).
    Thank you DanielJunglas!

    Attached are log files of new experiments with settings “WorkMem 200Mb and NodeFileInd type 2”.
    MIP is solved in both way this time. However, the node information is not exactly same.
    I also set mip strat file to 0 and cplex returned memory error this time.

    The code of this MIP is included, hope it could help.

    Sincerely,
    Tracy