"Why are forecasts so wrong?" Q&A (part 2 of 2)


Continuation of Q&A from the September 19, 2018 ASA web lecture "Why Are Forecasts So Wrong? What Management Must Know About Forecasting."

Why Are Forecasts So Wrong? Q&A (Part 2)ASA Logo

Q: Should we make a distinction between business as usual forecasts and major change forecasts and do FVA for these separately? Management reaction can differ with these.

A: Segmenting your time series based on their characteristics is often helpful in both modeling and forecasting, and in evaluating forecast performance. Many companies have long running, stable products, that can be forecast quite accurately with simple methods (perfect for automatic forecasting software).  Other products may have much more volatile demand due to promotions or other reasons.

When you do FVA analysis, the “stairstep” report is a good way to see the results. You can start with an overall report, aggregating results across the full product line. Then you can start segmenting the data, perhaps reporting products by brand or category, or by geography or distribution center, or even by the planner responsible for generating the forecasts.  This can help identify areas that may need further investigation by management.

You can also segment all the way down to the individual time series. Particularly problematic / non-value adding time series can easily be identified and the issues addressed.

Q: Is there any research that estimates the degree to which automated forecasting processes end up with over-fit models?

A: I’m not aware of such research, but this would be valuable information. If the automatic model selection is done properly, there should be protections against overfitting. For example, if there is sufficient history a hold-out sample can be used to evaluate performance of the model over the hold out period. There are also metrics like the Akiake Information Criterion (AIC) which deals with the trade-off between goodness of fit and simplicity of the model.

Q: I just want to confirm that moving averages would fall under the naive approach, correct?

A: Forecasts obtained with a minimal amount of effort and data manipulation and based solely on the most recent information available are referred to as “naïve” forecasts. (Definition taken from Makridakis, et al, Forecasting: Methods and Applications (3rd edition, 1998).) Computationally simple methods like moving average or single exponential smoothing are suitable benchmarks for comparing performance of more elaborate models and can be considered naïve (or at least “simple”) models.

However, I would suggest always comparing performance to the random walk (or seasonal random walk), as this lets you know what the simplest model would achieve.

Q: What top two things can we as the statistician do to improve forecasts?

A: Perhaps the first thing is to always remember that your job is to create the best possible forecasts, not just fit models to history. Along with this, remembering that it is always better to start simple, and only add complexity when it can be justified by improved forecasting performance. We cannot become enamored with our favorite pet models and apply them in all circumstances. The simplest approaches often work the best, even if they aren’t new, exciting, or glamorous.

Second, recognize that composite models generally perform better than any single model. So rather than go to heroic efforts tuning and tweaking a single model, you may get better results (and with much less effort) simply averaging the forecasts from several standard models. For example, in the recently completed M4 Forecasting Competition, only 17 of 50 entries performed at least as well as the composite benchmark. And 12 of those 17 were themselves combinations of mostly statistical approaches.


About Author

Mike Gilliland

Product Marketing Manager

Michael Gilliland is author of The Business Forecasting Deal (the book), and editor of Business Forecasting: Practical Problems and Solutions. He is a longtime business forecasting practitioner, and currently Product Marketing Manager for SAS Forecasting software. Mike serves on the Board of Directors for the International Institute of Forecasters, and received the 2017 Lifetime Achievement in Business Forecast award from the Institute of Business Forecasting. He initiated The Business Forecasting Deal (the blog) to help expose the seamy underbelly of forecasting practice, and to provide practical solutions to its most vexing problems.

Leave A Reply

Back to Top