Is Optimization Really Part Of Analytics?
JeanFrancoisPuget 2700028FGP Comment (1) Visits (13454)
I looked at this question not so long ago but it is worth revisiting using the new analytics definition proposed by INFORMS. The proposed definition reads:
"Analytics -- the scientific process of transforming data into insight for making better decisions."
I like it. It is concise, well worded. Optimization seems covered by the "making better decisions" part.
However, the new definition seems very (too much?) data centric as stated by Ehsan Nikbakhsh on Or Exchange. Indeed, it clearly base decision making on insight derived from data. It implies you first need insight from data before you can use optimization.
I discussed it with colleagues at IBM, given we are also folding optimization under the (Smarter) Analytics umbrella. This is consistent with the above definition. According to this view, everything starts with data, optimization being no exception.
Are you OK with that? Are you, really?
Think about it. Do you really believe that we always need insight before optimization can be used?
I don't think so. I believe that there are two ways of using optimization, a data centric one, and an expert centric one.
The data centric way is captured by the proposed INFORMS definition. The good news here is that optimization can provide more value than just insight. As I wrote in Is Optimization Part Of Analytics? :
Optimization can be used to make better decisions instead of just showing insights derived from data . For instance, when we have sales trends, we can compute sales forecast, then we can plan production or replenishment against these forecasts. The value of Optimization is that it is using science to compute decisions. This is usually better than what humans can do .
The expert centric way is when optimization is applied to business problems that are well understood. It typically starts with interviews with experts. From them you get business constraints and objectives that you cast into a mathematical model. It is not uncommon to write the model before any data is available. And even when data is available, it is used as is, to feed actual constraints and costs into the model. I'll use a classical example to illustrate that point. The much celebrated Traveling Salesman Problem (TSP) is to find the shortest way to visit all cities once. The data is the distance between all city pairs. There is no need to derive insight from this data before solving the problem. Truth is that there are many real problems where optimization is used without any data analysis.
I hope I made it clear that the expert centric way of using optimization is not covered by the new definition of analytics proposed by INFORMS. Hence optimization is not (always) part of analytics!
Is it a problem? I am not so sure. Indeed, there are reasons to believe that the data centric way will be more and more common. First, analytics is a buzzword, it is getting lots of traction. Jumping on the bandwagon will lead to more exposure for optimization, hence it will lead to more use of it. Second, progress on data analysis is uncovering new opportunities for optimization. With better sales forecast retailers can optimize replenishment. With better buying pattern insight companies can optimize marketing spend, etc. I'll stop here but the more you think about it, the more examples come to mind. Third, business processes are evolving quickly, and there are more and more cases where no one is experienced enough to become an expert. In those case the only way to go is the data centric one.
Therefore, optimization will become more and more part of analytics as defined above.
There is much more to say about the relationship between analytics and optimization. Here are few blog entries or papers I found quite interesting:
I am interested in hearing about others, send me email or add comment and I'll add them to this list.
Let me conclude by saying that you can express your view on the new INFORMS analytics definition in this poll.
[edited on June 24 to add the analytics journey reference, suggested by Irv Lustig]
[edited on June 25 to add Laura McLay and Dualnoise posts]