Benchmarks of forecasting performance are available from several sources, including professional organizations and journals, academic research, and private consulting/benchmarking organizations. But there are several reasons why industry forecasting benchmarks should not be used for setting your own forecasting performance objectives.
1) Can you trust the data?
Are the numbers based on rigorous audits of company data or responses to a survey? If they are based on unaudited survey responses, do the respondents actually know the answers or are they just guessing?
2) Is measurement consistent across the respondents?
Are all organizations forecasting at the same level of granularity, such as by product, customer or region? Are they forecasting in the same time interval, such as weekly or monthly? Are they forecasting by the same lead time offset, such as three weeks or three months in advance? Are they using the same metric? It is important to note that even metrics as similar sounding as MAPE, weighted MAPE, and symmetric MAPE can deliver very different values from the same data.
3) Finally, and most important, is the comparison relevant?
Does the benchmark company have equally forecastable data?
Consider this worst-case example:
Suppose a benchmark study shows that Company X has the lowest forecast error. Consultants and academics then converge on Company X to study its forecasting process and publish reports touting Company X’s best practices. You read these reports and begin to copy Company X’s best practices at your own organization.
However, upon further review using FVA analysis, it is discovered that Company X had very easy-to-forecast demand, and it would have had even lower error if it had just used a naive forecast. In other words, Company X’s so-called best practices just made the forecast worse.
This example is not far-fetched. Organizations at the top of the benchmark lists are probably there because they have the easiest-to-forecast demand. Many organizational practices, even purported best practices, may only make the forecast worse.
Benchmarks tell you the accuracy that best-in-class companies are able to achieve. But...they do not tell you whether their forecasting environment is similar to yours or worthy of your admiration. Without that information, industry benchmarks are largely irrelevant and should not be used to evaluate your performance or set performance objectives.
11 Comments
I would correct that final statement to be, "Benchmarks tell you the accuracy that best-in-class companies are WILLING to report."
There is also always the question of what forecast misses are scrubbed out of the metric. One can imagine that there usually is a reasonable business explanation like: the customer cancelled the promotion at the last minute, the customer changed their mind and wanted a later shipment, we had the stuff in stock, but the plant messed up the shipment, sure we didn't have enough product on hand for the launch, but we plan for shortages to generate marketing buzz, and so on. One would probably find that the things scrubbed out of the metric are not equally balanced between exceptionally good forecasts and "justifiable" bad forecasts so the forecast accuracy number reported is probably somewhat higher than the unvarnished truth.
Somebody would do such a thing?!?!?!? Somebody would purposely scrub a metric to make it look better?!?!?!?
Oh the humanity...
Ladies and gentlemen, we've found the one guy in forecasting who is more skeptical than I am.
For more of Sean's misanthropic ravings, come see his presentation "Forecastability: Driving Improvement in Forecasting" next week (June 25) at the International Symposium on Forecasting in Boston.
After looking for the most recent benchmarking data for demand forecasting I have not been able to find any multi-industry accuracy percentages since 2006-7. What journals/research/organizations would have the most up to date information?
Michael,
The Institute of Business Forcasting periodically publishes results of their surveys of conference attendees. These appear in the Journal of Business Forecasting (e.g. Winter 2007-08 issue) or in separate publications. Find more at http://www.ibf.org -- you may need to become a member to access their reports.
Other consulting or academic organizations may have this kind of information available (perhaps only data from their clients, and perhaps only for a fee). For example, The Hackett Group provides business benchmarking services, and the University of Tennessee Supply Chain Forum does work in the forecasting area. And there are other forecasting research hotspots, such as Lancaster University's Centre for Forecasting.
Just be aware of the perils when viewing and interpreting benchmark data.
Your response is much appreciated and your advice/recommendations on the perils will be considered when evaluating the information. Thank you
Pingback: More on forecasting benchmarks - The Business Forecasting Deal
Pingback: 5 steps to setting forecasting performance objectives (Part 2) - The Business Forecasting Deal
Pingback: 5 steps to setting forecasting performance objectives (Part 2) supplychain.com
Pingback: Forecasting research project ideas - supplychain.com
Pingback: Show your forecasting skills in Good Judgment Open - The Business Forecasting Deal
Pingback: Forecasting research project ideas - The Business Forecasting Deal