Blogs

Blogs

More on forecasting benchmarks

1
By

The Perils Revisited

A few posts ago I warned of the perils of forecasting benchmarks, and why they should not be used to set your forecasting performance objectives:

  1. Can you trust the data?
  2. Is measurement consistent across the respondents?
  3. Is the comparison relevant?

In addition to a general suspicion about unaudited survey responses, my biggest concern is the relevance of such comparisons. If company A has smooth, stable, and easy-to-forecast demand, and company B has wild, erratic, and difficult-to-forecast demand, then the forecasters at these two companies should be held to different standards of performance. It makes no sense to hold them to some "industry benchmark" which may be trivial for company A to achieve, and impossible for B.

Perhaps the only reasonable standard is to compare an organization's forecasting performance against what a naive or other simple model would be able to achieve with their data. Thus, if a random walk model can forecast with a MAPE of 50%, then I should expect the organization's forecasting process to do no worse than this.

If the process consistently forecasted worse than a random walk, we know there must be something terribly wrong with it!

Benchmark Study on Forecasting Blog

One of the forecasting blogs I enjoy is the aptly named Forecasting Blog, published by Mark Chockalingam's Demand Planning, LLC. Last week it reported on results from a forecasting benchmark survey covering (among other things) the forecast error metric used, and forecast error results.

Unsurprisingly, they found that 67% of respondents used MAPE or weighted MAPE (WMAPE) as their error metric. Less commonly used error metrics were % of Forecasts Within +/- x% of Actuals, Forecast Bias, and Forecast Attainment (Actual/Forecast).

The blog also reported Average of Forecasting Error by Industry (e.g. 39% in CPG, and 36% in Chemicals). However, it was unclear how this average error was computed, and I suspect Peril #2 (Is the measurement consistent across respondents?) may be violated.

It is well known that the same data can give very different results even for metrics as similar sounding as MAPE and WMAPE. If different companies are using different metrics to compute their forecast error, I'm not sure how you would combine them into an industry average.

Take a look at the blog post for yourself, and spend a few minutes to take their Forecast Error Benchmark Survey.

Tags error metrics forecasting benchmarks Forecasting Blog forecasting survey Mark Chockalingam
Share

About Author

Mike Gilliland

Product Marketing Manager

Michael Gilliland is a longtime business forecasting practitioner and formerly a Product Marketing Manager for SAS Forecasting. He is on the Board of Directors of the International Institute of Forecasters, and is Associate Editor of their practitioner journal Foresight: The International Journal of Applied Forecasting. Mike is author of The Business Forecasting Deal (Wiley, 2010) and former editor of the free e-book Forecasting with SAS: Special Collection (SAS Press, 2020). He is principal editor of Business Forecasting: Practical Problems and Solutions (Wiley, 2015) and Business Forecasting: The Emerging Role of Artificial Intelligence and Machine Learning (Wiley, 2021). In 2017 Mike received the Institute of Business Forecasting's Lifetime Achievement Award. In 2021 his paper "FVA: A Reality Check on Forecasting Practices" was inducted into the Foresight Hall of Fame. Mike initiated The Business Forecasting Deal blog in 2009 to help expose the seamy underbelly of forecasting practice, and to provide practical solutions to its most vexing problems.

Back to Top