Solving Optimization Problems On The Cloud With Python
JeanFrancoisPuget 2700028FGP Visits (11746)
The availability of an api for solving optimization problems via DOcloud is opening new ways to develop an application: you can use the implementation language of your choice. Granted, at the time of writing, we only provide a client api for Java. But we also provide a REST api, which is what matters here.
As an example, we present a very simple client api for Python that wraps the REST api. All you need to do is to provide as input a list of files that define the problem to be solved. These files can be created as you wish. Of course, you can use our tools: CPLEX optimizer, OPL, and CP optimizer. You can also use the mathematical optimization modeling languages of your taste. You can even use a competing optimization solver! Last but not least you can use one of the Python packages for mathematical optimization modeling, eg Pulp or Pyomo. All what matters is their ability to save your problem in one of the formats DOcloud accepts. At the time of writing, these file type combinations are supported by the DOcloud api:
LP, MPS, and SAV files can be compressed as a BZ2 or a GZIP archive. JSON is the default format for data files that accompany OPL models.
Before using our Python api, we must have a valid subscription to DOcloud api. Instructions to get a key are available on the DOcloud page. Look for the developer drop down menu. If you are logged in with an existing IBM id, then you should select the Get API & Base URL item. If you are not logged in then you should go to the developer community, and register for the free trial.
Once you have completed the onboarding step you must paste your key and the base url in the code below.
key = 'Paste your DOcloud key here'
Our client api is made of one class named Docloud. We must create one instance of that class with our credentials
doc = DOcloud(base_url, key, verbose=True)
Given we are discovering the class here, we create it in verbose mode. we will therefore get messages about what is going on. We could have created a DOcloud instance without the verbose argument if we wanted to get rid of these messages.
We can then launch the execution of a job by providing a list of the relevant files in a single function call. We can optionally state where to put the solution files and the log file.
Here is an example with a lp file stored in our /resources directory. Results for CPLEX optimizers are provided in the same format as the CPLE
test.lp upload succeeded
We can then inspect the content of the solution as follow.
We can also only print some of the lines of the the file, for instance we print the last 10 lines of the log:
We can provide more than one input file. For instance, the Sudoku problem we solved in our previous post can also be solved with a single function call. Assume we have stored the OPL data in a file named sudoku.dat, and the OPL model in a file named sudoku.mod. A one liner does all the job:
If we want to process the results before writing to a file, we can decompose the solve() function into its constituents. Here is an example with the lp file stored in our /resources directory. We start the job, then we wait for the results and process them, then we print the CPLEX running time from the log, then we clean the job on DOcloud server.
job1 = doc.
test.lp upload succeeded
Let us look again at the Sudoku problem . We can solve the problem and print the result with this code. It is much shorter and easier to read than what we had in our previous post.
The input and optimization model are the same as in my previous post.
input = """
We can solve it and get the result in one call. This time we pass the data directly instead of passing file paths. Note that we must provide file names for each input data. the names are arbitrary, but the file extension is not. DOcloud will route the job to the right worker (CPLEX or OPL) based on the file extensions.
We can also use the matplotlib code we had to get a nice display of the result
import numpy as np
Note that you can handle several jobs at a time. For instance, this code runs perfectly well:
job2 = doc.
result2 = doc.
We can also run the solve function as a standalone script. The usage is to provide the list of input files on the command line. There are two options to specify the solution and the log files. If they are both omitted then no result is stored, which is a bit useless... There is also a verbose option if we want details on what's going on.
Here are two examples corresponding to the ones we used above:
$ python docloud.py --solution resources/test.sol --log resources/test.log --verbose resources/test.lp
$ python docloud.py --solution reso
The code for our class and the exampels above is available as a notebook here.
The standalone Python script is available here.
We will provide both in Github in the near future. Stay tuned as we will evolve it in order to meet additional needs. Suggestions for improvement are welcome. Among the ones we have in mind is exception handling. We also want to avoid going through files when data is already in memory.
Note that we do not look at the models that are submitted to DOcloud. We don't keep them either. What is being submitted is treated as private data and protected as such.
Let me conclude with the traditional forward looking warning:
This client api is provided as a mere example. We may provide a client api for Python in the future that differs substantially from the one I present here.
Updated on April 15, 2015. Added the solve method, as well as the possibility to run it as a standalone script.
Update on April 25, 2015. Added the ability to pass streamed data rather than files.
Update on Oct, 3, 2015. We announced a python client and a python modeling API for Docloud, see here.