Would you buy something that doesn’t have demonstrable Return on Investment? Of course you would! Whether you realize it or not, you do this every time you buy software. There is no proven ROI.

Maybe this is just a pet peeve of mine. Maybe I care too much about cause and effect. Maybe I care too much about logic and scientific method. Maybe I care too much about what it means to prove something. Or maybe I’m just another egomaniacal jerk with a blog that thinks he is entitled to raise a stir about nothing. I am, therefore I will.

Every software vendor touts the ROI you will achieve by buying their products. 10%, 100%, 1000% ROI – the claims are everywhere, and part of every sales cycle. And it isn’t just the vendors claiming ROI. It is also de rigueur for the customer’s internal project sponsor or champion to make claims of anticipated ROI -- in order to secure the funding to buy the software.

ROI is a sham??? Wow!

Think about it – how would you ever prove that implementing software X caused effect Y (where Y is increased revenue, reduced costs, higher profits, etc.). Just because one thing happens along with something else, does not prove there is any relationship between the two whatsoever – let alone a “causal” relationship. The problem is that things change, they vary over time. (Perhaps this is why they are called variables?) The interrelationships among variables can be subtle and mysterious, and we don’t really know for sure what mechanism (if any) is driving things.

We often (and wrongly) attribute causality in these sorts of situations. The geniuses in the financial news media, for example, can provide brilliant explanations for the cause of every market fluctuation. Nassim Nicholas Taleb has a great example of this in his book The Black Swan:

"One day in December 2003, when Saddam Hussein was captured, Bloomberg News flashed the following headline at 13:01: U.S. TREASURIES RISE; HUSSEIN CAPTURE MAY NOT CURB TERRORISM." Bond prices then fell. "At 13:31 they issued the next bulletin: U.S. TREASURIES FALL; HUSSEIN CAPTURE BOOSTS ALLURE OF RISKY ASSETS."

So, per Bloomberg, the capture of Saddam caused bond prices to both rise and fall? (I guess if Saddam were not captured, bond prices would both fall and rise?) Foolish me, I thought only Jim Cramer could come up with gold like this!

In some situations purported causality has been reasonably well established through controlled experimentation. For example, I accept that downing 400mg of ibuprofen will cause my headache to go away. But claims of causality from forecasting software are not subject to such rigorous oversight. There is not yet a Federal Forecasting Administration to vet our ROI claims.

When the economy is good and profits are rising, a positive ROI can be attributed to just about anything an organization does. (True Story: Within weeks after interviewing and accepting the position of forecasting manager at Iomega in 1996, its stock price rose from $28 to $110. Wall Street apparently heard I was coming!) Any vendor is sure to jump on this bandwagon, and attribute profit gains to implementation of its software. This makes a good story, but fails to consider what else has changed. Perhaps your company introduced a hot new product that generated tremendous revenues and profits. Or perhaps your customers were just spending more freely in the booming economy and you would have increased revenue and profits without the new software. There needs to be much better evidence to demonstrate the cause and effect.

If a vendor uses this argument in good faith, and accepts the “logic” that its software caused these profit gains at its customers, then what happens next? What about its customers that implemented the software in late 2007 or the first half of 2008? Is the vendor willing to apply the same “logic” and admit that its software “caused” these customers to have falling revenues, huge losses, and perhaps even go out of business in the greatest financial collapse since the Great Depression? Who would ever admit that?

This is my gripe – we are fast to apply causality when the “effect” is good, but can always come up with a creative explanation when the “effect” is bad. Yet without a rigorous (and probably impractical) proof of cause and effect, we are just talking a bunch of nonsense.

Don’t talk to me about ROI.

Another time I'll discuss more appropriate ways to evaluate the value of forecasting software.