With apologies to Johnnie Cochran and Joyce Kilmer : “If the model do fit, it don’t prove ****”
This was the warning from Trick #1. As a forecaster your job is to produce forecasts – as good as they can reasonably be expected to be – not to fit models to history. Just because some highfalutin new software can fit your history like a glove doesn’t mean you should buy it. As the Microsoft Excel demo inadvertently illustrates, having too close a fit to history is often a very bad thing for forecasting!
Trick #2: Assert you can handle events
Another favorite trick of the snake-oil/turnip salesman is to say “Yes, we can handle events.” An event is holiday, promotion, price change, disaster, etc…something planned (or unplanned) that has an impact on demand. As a forecaster you realize these sorts of things have happened in the past, and will probably continue to happen in the future. You want to be able to account for events with your forecasting software, and have it tell you, for example, that if you drop your price X%, there will be a Y% increase in demand.
When you solicit proposals from various forecasting vendors, every vendor is going to check “Yes” in the RFP box about handling events. Every vendor is going to respond emphatically that “Of course, you can handle events with our software.” But what does this really mean for most vendors? There are two situations:
• Does the software actually analyze historical events to determine the likely impact of the new event, and then reflect it in the forecast?
• Do you, the user, come up with some estimate of the impact of the event, and then enter the impact (e.g. increase demand 10%) so the software can do the arithmetic and show you the new forecast?
Most likely, the software you now have (or are considering in an RFP) does the latter. The software is smart enough to do the addition or subtraction (or maybe even multiplication and division -- woohoo!) as long as YOU tell it the impact of the event. This important detail is buried in the sales and marketing verbiage, usually along the lines of being able to ‘what if’ the outcome of various scenarios. This works great – as long as you realize it is YOU making up the outcomes out of thin air, and not based on any sort of analytical assessment of the data.
How do you figure out the impact of an event? That happens to be a difficult problem, and you may never really know for sure even with the best statistical analysis. However, an analytical approach combined with judgment is better than relying on judgment alone.
When reviewing software and soliciting proposals make sure to ask the right questions, and understand the answers. It isn’t actually a lie to say “Yes we handle events” but just realize -- this probably means that you still have to do all the difficult work of figuring out the event impact by yourself.
