Topic
17 replies Latest Post - ‏2013-03-06T06:02:58Z by SystemAdmin
SystemAdmin
SystemAdmin
7929 Posts
ACCEPTED ANSWER

Pinned topic The difference between using concert technology of C++ and Cplex directly.

‏2013-02-25T04:35:10Z |
Hello, this is from a beginner of Cplex.

I am using concert technology for C++ of Cplex to solve my MIP model.
And it returns out of memory after solving it by this C++ collaborate with Cplex.

However, I tried once to read the .lp file directly by using Cplex 12.4 as well. It can be solved without any memory problem in this way.

So my question is, what's the difference between this two methods? I use the same computer and identical MIP model...

Need help to figure out the differences, please.
Thank you! ^^
Hong
Updated on 2013-03-06T06:02:58Z at 2013-03-06T06:02:58Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    7929 Posts
    ACCEPTED ANSWER

    Re: The difference between using concert technology of C++ and Cplex directly.

    ‏2013-02-25T06:09:31Z  in response to SystemAdmin
    The Concert C++ layer keeps a copy of the model. For large models this may result in a significant increase of memory requirements.
    When you say your "read in the .lp directly", does that mean that you also read the LP file into Concert or did you build the model in Concert, export it to an LP file and then read that LP file? In the latter case things are expected to be different. Export/import to/from an LP file may change the ordering of columns and rows and may introduce floating point round off. You should repeat the experiment using SAV files instead (just change extension .lp to .sav).
    Since you say that the solution process runs out of memory during the solve (could you show us a log file) this probably means that the tree got too large. You can instruct CPLEX to swap out parts of the tree to disk. To do this you have to change the following parameters:
    • SystemAdmin
      SystemAdmin
      7929 Posts
      ACCEPTED ANSWER

      Re: The difference between using concert technology of C++ and Cplex directly.

      ‏2013-02-25T09:52:40Z  in response to SystemAdmin
      Thank you DanielJunglas!

      First, read .lp means the latter case that I export the model as a .lp file which is built by Cplex concert technology for C++. Then I use Cplex optimizer to read this model(.lp file) and input command "opt" to solve this directly.

      As for the memory problem, you are right, it's because of the tree size. And I will change the parameters soon.

      Attached is my MIP model. Do you think this model size could leads to the different results?
      Did you mean if I export the model into a SAV file instead of lp file by Concert C++, and do "read" and "opt" by Cplex optimizer 12.4 directly, the solution result will be out of memory (same with Concert C++)?
      • T_O
        T_O
        438 Posts
        ACCEPTED ANSWER

        Re: The difference between using concert technology of C++ and Cplex directly.

        ‏2013-02-25T10:53:02Z  in response to SystemAdmin
        Could you, as Daniel proposed, repeat your experiments with .sav-files. Some coefficients in your .lp-file might cause numerical roundoffs (e.g. 9.15478038787842).

        How much RAM do you have?

        Best regards,
        Thomas
        • SystemAdmin
          SystemAdmin
          7929 Posts
          ACCEPTED ANSWER

          Re: The difference between using concert technology of C++ and Cplex directly.

          ‏2013-02-26T03:40:37Z  in response to T_O
          Thank you Thomas!
          I repeat my experiment with .sav file instead of .lp file.
          However, the problem is still solved in this way.
          My RAM is 8.00G, r there any other possibilities to explain this kind of difference?

          Sincerely,
          Tarcy
          • T_O
            T_O
            438 Posts
            ACCEPTED ANSWER

            Re: The difference between using concert technology of C++ and Cplex directly.

            ‏2013-02-26T07:45:38Z  in response to SystemAdmin
            Well, I can now only guess where the problem comes from. Memory might be the problem. Could you post your C++-code? Maybe there is a memory leak in your code? You might also try to use the C callable library instead of concert.

            Best regards,
            Thomas
            • SystemAdmin
              SystemAdmin
              7929 Posts
              ACCEPTED ANSWER

              Re: The difference between using concert technology of C++ and Cplex directly.

              ‏2013-02-27T06:18:14Z  in response to T_O
              Thank you Thomas!
              Did u mean that the experiment result should be no difference between using Cplex optimizer 12.4 directly and Cplex collaborate with C++?
              If so, I will check my code as u suggested to figure it out.
              I'm really appreciate for your help!
              Once I figure it out, I will post my experienc here.

              Best regards,
              Tracy
              • T_O
                T_O
                438 Posts
                ACCEPTED ANSWER

                Re: The difference between using concert technology of C++ and Cplex directly.

                ‏2013-02-27T06:53:27Z  in response to SystemAdmin
                I mean that I would not expect any remarkable differences betwwen using the C callable library and the interactive optimizier if your code does not use a lot of memory for your own data and if it does not have any memory leaks.

                Let me give you an example in pseudocode:

                create_and_fill_data_vectors
                create_cplex
                copy_data_from_vectors_to_cplex
                solve_with_cplex

                can use a lot more memory than

                create_cplex
                {
                create_and_fill_data_vectors
                copy_data_from_vectors_to_cplex
                }
                solve_with_cplex

                You see the difference?

                Best regards,
                Thomas
                • SystemAdmin
                  SystemAdmin
                  7929 Posts
                  ACCEPTED ANSWER

                  Re: The difference between using concert technology of C++ and Cplex directly.

                  ‏2013-03-05T02:07:47Z  in response to T_O
                  Thank u for your reply Thomas.
                  Actually, I think my code is the first case of your two pseudo code models.
                  I attached a part of my code here. I think it can briefly explain my code structure.

                  Could please explain a little bit why the first pseudo code will use much more memory than the second one?

                  Sincerely,
                  Tracy

                  Attachments

                  • T_O
                    T_O
                    438 Posts
                    ACCEPTED ANSWER

                    Re: The difference between using concert technology of C++ and Cplex directly.

                    ‏2013-03-05T06:59:34Z  in response to SystemAdmin
                    If your data consumes a lot of memory, then this is freed in the second example, so CPLEX can use this memory for solving the problem. But I don't really think this is the root of your problem as your model is quite small. Nevertheless, I don't know how large stop_list and OD are. Maybe you allocated unnecessary memory?

                    As the log files differ, it might be that one instance needs a lot more memory than the other. So I think we should try to find out why they differ. Maybe one of the CPLEX-People could have a look at it.

                    Best regards,
                    Thomas
                    • SystemAdmin
                      SystemAdmin
                      7929 Posts
                      ACCEPTED ANSWER

                      Re: The difference between using concert technology of C++ and Cplex directly.

                      ‏2013-03-05T07:56:37Z  in response to T_O
                      For the problem I attached, there are 13 stops in stop_list.
                      Consequently, OD is a 13*13 matrix.
                      If there is no other possibilities, I guess I will check my code again to in case of any inappropriate memory allocate.

                      Thank you so much Thomas!^^
                      Tracy
              • SystemAdmin
                SystemAdmin
                7929 Posts
                ACCEPTED ANSWER

                Re: The difference between using concert technology of C++ and Cplex directly.

                ‏2013-02-27T13:56:52Z  in response to SystemAdmin
                Could you please post log files for the two different solves?
                Maybe then we can spot what the difference is between your two runs.
                • SystemAdmin
                  SystemAdmin
                  7929 Posts
                  ACCEPTED ANSWER

                  Re: The difference between using concert technology of C++ and Cplex directly.

                  ‏2013-03-05T01:56:04Z  in response to SystemAdmin
                  Sorry for the late reply.
                  Attached are the log files. It look like the memory sizes for saving the tree are different in C++ program and Cplex solver 12.4. But I just use the default parameters settings of Cplex 12.4 in C++ code (I didn't change any parameters in my code).

                  Also, the tree sizes are different in "cplexlog-SAV" and "cplexlog-LP" these two files. Do u think the differences are because of round off coefficients and column changes?

                  Sincerely,
                  Tracy
                • SystemAdmin
                  SystemAdmin
                  7929 Posts
                  ACCEPTED ANSWER

                  Re: The difference between using concert technology of C++ and Cplex directly.

                  ‏2013-03-05T01:59:05Z  in response to SystemAdmin
                  Sorry for previous miss attach.
                  Correct one is attached here.

                  Thank you,
                  Tracy
                  • SystemAdmin
                    SystemAdmin
                    7929 Posts
                    ACCEPTED ANSWER

                    Re: The difference between using concert technology of C++ and Cplex directly.

                    ‏2013-03-05T08:23:22Z  in response to SystemAdmin
                    OK, it is hard to tell what is going different here without having the C++ code at hands. If you compare the SAV log and the C++ log then you can see that things are exactly the same until node 35080. In particular the statistics for the presolved model etc. are all the same. This is a strong indication that the in-memory models that are solved are the same.
                    One thing that can change the solution paths in low-memory situations are optional sub-algorithm that fail due to out of memory. For example, if CPLEX runs a heuristic then this heuristic may run out of memory in one run but not in the other. Since heuristics are optional, out of memory in heuristics is usually not considered an error and optimization continues. To make sure that sporadic out of memory is not the issue here could you repeat the experiment (SAV file and C++ solve) using node files in both cases? Set WorkMem to 200 and NodeFileInd to 2 ("set workmem 200" and "set mip strat file 2" in the interactive).
                    Another thing to try would be to run without any node files (NodeFileInd set to 0).
                    • SystemAdmin
                      SystemAdmin
                      7929 Posts
                      ACCEPTED ANSWER

                      Re: The difference between using concert technology of C++ and Cplex directly.

                      ‏2013-03-06T06:02:58Z  in response to SystemAdmin
                      Thank you DanielJunglas!

                      Attached are log files of new experiments with settings “WorkMem 200Mb and NodeFileInd type 2”.
                      MIP is solved in both way this time. However, the node information is not exactly same.
                      I also set mip strat file to 0 and cplex returned memory error this time.

                      The code of this MIP is included, hope it could help.

                      Sincerely,
                      Tracy
      • SystemAdmin
        SystemAdmin
        7929 Posts
        ACCEPTED ANSWER

        Re: The difference between using concert technology of C++ and Cplex directly.

        ‏2013-02-25T15:49:38Z  in response to SystemAdmin
        Sorry, I was unclear about the issue with the .lp file.
        As I said, exporting the model to an .lp file may reorder the rows/columns of the problem matrix and may also result in numerical round off (very likely in your case as Thomas noted).
        If there is numerical round off in the coefficients then you are actually solving a different problem in the interactive optimizer than you did in C++. So it is no surprise that results/performance are different.
        Even if there is no round off for the coefficients and only the rows/columns are reordered, then you are still solving a different problem in the interactive and again different results/performance are expected.
        The SAV file format avoids all these issues and when using a SAV file you should see the same behavior in the interactive optimizer as you saw in C++.

        >Attached is my MIP model. Do you think this model size could leads to the different results?
        >
        It is not the size of the model. As Thomas said, your .lp file contains numbers like 2.24722051620483. That is most probably not exactly what was in the C++ model but was instead rounded/truncated when output to the .lp file.
        • SystemAdmin
          SystemAdmin
          7929 Posts
          ACCEPTED ANSWER

          Re: The difference between using concert technology of C++ and Cplex directly.

          ‏2013-02-26T03:51:59Z  in response to SystemAdmin
          To DanielJunglas, I'm sorry to made u confused.
          As I replied to Thomas, I repeat the procedure but with .sav file.
          The result is still different with concert C++.
          But for the coefficients, you are correct that round off coefficients are exist.

          Thank you Daniel!^^