We’ve all seen the press releases. A vendor announces to the world that one of their customers achieved some kind of significant saving or productivity enhancement by using their products.
It got me thinking. What if the improvement was due to random chance? What if the improvements were just some sort of Hawthorne Effect?
What if it didn’t matter which vendor or product was chosen, and that the improvement happens simply because something was changed?
What if a different choice of vendor or product would have resulted in an even bigger improvement?
Do any of these ‘studies’ control for that? Or are the numbers mostly marketing enhanced interpretations of managers’ self-appraisals?
Opportunity Costs
These reports often present a ‘do-nothing’ comparison scenario on which the cost savings are based. Simply stated, they measure what would have happened if a company continued doing things the way they were already doing them. The status quo is then compared to what happened when they used the new WhippleSoft brand ProjektMangler.
Somehow it always turns out to have been a good idea.
The analysis is rarely detailed, is littered with assumptions, and I’ve yet to see one that includes confidence intervals or error bars.
What we don’t see is analysis of alternate Do Something plans.
We see a Do Nothing plan, and the successful plan, but no others.
Financial Analogy
Analogy time!
Let’s say you have $10,000 to invest. It’s currently sitting in your bank transaction account, earning 3% p.a. You put it in a term deposit with SponsorBank and earn 6% (a 100% increase!).
SponsorBank commissions a study by Consulticorp to measure how wise and noble you are. To everyone’s delight, they find that you did indeed make a wise and noble decision to invest with SponsorBank. The Consulticorp report is published to the world with an accompanying press release trumpeting the news.
Everyone has a jolly good lunch and congratulates each other on a job well done. Sometimes there is cake.
What is missing from this picture is the knowledge that OtherBank was offering 7% on their term deposits, and that RiskyBank was offering as much as 8%.
How wise and noble are you now?
A Good Start
These projects probably did improve things for the customer, and the vendors’ technology probably did provide that improvement.
But would things have gotten better with any old change? Was it really the special magic provided by the choice of vendor that provided all of that improvement? What if it was only some of the improvement? How much?
What if you could have done better, for the same money, if only you’d made a different choice?
The PR title of this post is made up, but do you have a real example of an over-hyped vendor report? Share it in the comments!
Another good read.
The thing is though that when “most” companies decide to implement a new system, they also investigate the alternatives in the marketplace and choose the best solution to fit their needs. To follow your financial analogy, “most” people with $10k to invest will research which institutions offer the best rates and security, and then make their choice. Your post seems slanted towards a situation where the company/investor randomly chooses a product without consideration for alternatives and their associated outcomes.
Thanks Shane!
What I was trying to say with my analogy (and didn’t do well enough, it appears) is that in the analyst reports trumpeting success, we don’t see the evidence of the research you mention here. All we see is the chosen path, and a comparison with the status quo.
What if a company is doing a really poor job today? Changing just about anything is likely to provide them a benefit. A 37% cost saving by a company that leaks cash like a poorly maintained sieve is a very different result than a 37% saving by a well run company.
Consider the saving in printer toner expenses Arts Victoria could claim next year.