We’ve all seen the press releases. A vendor announces to the world that one of their customers achieved some kind of significant saving or productivity enhancement by using their products.
It got me thinking. What if the improvement was due to random chance? What if the improvements were just some sort of Hawthorne Effect?
What if it didn’t matter which vendor or product was chosen, and that the improvement happens simply because something was changed?
What if a different choice of vendor or product would have resulted in an even bigger improvement?
Do any of these ‘studies’ control for that? Or are the numbers mostly marketing enhanced interpretations of managers’ self-appraisals?
These reports often present a ‘do-nothing’ comparison scenario on which the cost savings are based. Simply stated, they measure what would have happened if a company continued doing things the way they were already doing them. The status quo is then compared to what happened when they used the new WhippleSoft brand ProjektMangler.
Somehow it always turns out to have been a good idea.
The analysis is rarely detailed, is littered with assumptions, and I’ve yet to see one that includes confidence intervals or error bars.
What we don’t see is analysis of alternate Do Something plans.
We see a Do Nothing plan, and the successful plan, but no others.
Let’s say you have $10,000 to invest. It’s currently sitting in your bank transaction account, earning 3% p.a. You put it in a term deposit with SponsorBank and earn 6% (a 100% increase!).
SponsorBank commissions a study by Consulticorp to measure how wise and noble you are. To everyone’s delight, they find that you did indeed make a wise and noble decision to invest with SponsorBank. The Consulticorp report is published to the world with an accompanying press release trumpeting the news.
Everyone has a jolly good lunch and congratulates each other on a job well done. Sometimes there is cake.
What is missing from this picture is the knowledge that OtherBank was offering 7% on their term deposits, and that RiskyBank was offering as much as 8%.
How wise and noble are you now?
A Good Start
These projects probably did improve things for the customer, and the vendors’ technology probably did provide that improvement.
But would things have gotten better with any old change? Was it really the special magic provided by the choice of vendor that provided all of that improvement? What if it was only some of the improvement? How much?
What if you could have done better, for the same money, if only you’d made a different choice?
The PR title of this post is made up, but do you have a real example of an over-hyped vendor report? Share it in the comments!