Why Users Cannot Help You Improve Your Products
JeanFrancoisPuget 2700028FGP Visits (9169)
Making decision based on data seems a good idea, doesn't it? After all, this is the value promised by all Big Data promoters out there. Let's look at a real use case to understand better what might go right or wrong. I will focus on the decisions product managers must make when they think of the next version of their product. Should they base product evolutions on customer feedback?
Let's first address the case of disruptive technologies. It is (now) (well) known that the answer to the above question is "no" in that case. For instance, Henry Ford reportedly said:
"If I had asked people what they wanted, they would have said faster horses"
He did not see people asking for automobiles before these existed. For a more recent example, it is well documented that people were not asking for software defined phones when the first iPhone went out. Yet it was a success.
Let's look back at our question. Granted, customer feedback is not a good source of data for product revolutions. But isn't it a good source for product evolutions? Doesn't it seem reasonable to rely on customer feedback in that case? Addressing their needs should lead to a better product, shouldn't it?
Before answering these questions let me tell you a story that is well known in some circle (description below is inspired from Surv
During world war II, a statistician named Abraham Wald was asked to help military to reduce bomber losses. The military wanted to add armor to bombers, but they could not add armor everywhere, or the plane would not take off. Therefore, they wanted to know where to put some very limited extra armor. They looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Considering this information, where would you put the extra armor? Naturally, the commanders wanted to put the thicker protection where they could clearly see the most damage, where the holes clustered.
But Wald said no, that would be precisely the wrong decision. Putting the armor there wouldn’t improve their chances at all. After all, the planes that returned did return. One should worry about those who did not return. These were the ones in need of more protection. Then, assuming bullet impact are evenly distributed, missing planes were hit exactly where the returning planes were not hit. It means that the holes in the surviving planes actually revealed the locations that needed the least additional armor. Look at where the survivors are unharmed, he said, and that’s where these bombers are most vulnerable; that’s where the planes that didn’t make it back were hit. Wald convinced the military to do the exact opposite of what they were about to do.
What does Wald story tells us in general? Wald message was that military should base their decisions on all planes, those who came back, but also those who didn't. The military only looked at those who returned, which is a sampling bias as it ignores a whole set of planes (the ones that did not return). I'll stop discussing Wald here and refer readers to Survivorship Bias, to Sele
We can now come back to our product evolution question. Should we base them on user feedback? Well, the bombers case tells us that we should also get feedback from those people who don't use our product (people = airplane for this analogy). We should get feedback from all the people who could be using our product, whether they actually use it or not.
Getting feedback from non users is much harder than getting feedback from users. It is much harder, but it is very important; if we don't, then we are like the world war II military, we may may end up doing the exact opposite to what is really useful.
So, what should we be doing? Our answer at IBM is to now use a design methodology based on sponsor users. These sponsor users may not be current users, they are rather selected to be representative of the set of users we target. With representative users we avoid most of the sampling bias drawbacks. Interestingly enough, other companies, e.g .Facebook, are using a similar approach to escape the sampling bias. Note that there is more to IBM Design than sponsor users, see http
We've seen two examples of how sampling bias can lead to bad decisions. This is a general statement. As a mater of fact, sampling bias is one of the major issue that Big Data approaches faces. It is good to base decisions on data, provided your data is representative. If your data isn't representative, then you may end up doing the wrong things, as world war II military almost did.