M4 Forecasting Competition: Results and Commentary


The International Journal of Forecasting has published its 2020-Q1 issue, guest edited by Spyros Makridakis and Fotios Petropoulos, and dedicated entirely to results and commentary on the M4 Forecasting Competition. This issue should be of great interest and value to business forecasting practitioners, and you get online access to it now as a member of the International Institute of Forecasters. (See membership promotional information below.)

M4 participants forecasted 100,000 time series. The 49 competitors included eight that used either pure Machine Learning approaches, or ML in conjunction with traditional time series methods. The six pure (or combination of pure) ML methods fared poorly, with all of them falling below a benchmark combination of three simple time series methods. However, utilizing ML either in combination with statistical methods (and for selecting weightings), or in a hybrid model with exponential smoothing not only exceeded the benchmark, but performed at the top.

With all the hype about ML methods in forecasting, the M4 provides an important source of research data on the performance of various traditional and emerging forecasting approaches. We've seen in prior research and competitions (such as the M3 in 1998), simpler methods and combination methods tend to perform well. Again, in M4, combinations of traditional time series methods ranked near the top. But the two best performers used statistical + ML methods, showing that highly complex methods can potentially forecast better than simpler methods...but at a cost.

A hugely important contribution of the M4 was in replicating (most of) the competing methods on a standard hardware configuration. This allowed us to see the "cost" of each method (in terms of computational time). For the benchmark Naive1 ("no change") model, it took 12 seconds to forecast the 100,000 series. The top performing statistical + ML methods took 6 days and 32 days respectively!!!

This issue of the IJF is a must-read for anyone interested in improving their forecasting performance. It contains 35 articles, including a foreword by Nassim Nicholas Taleb (author of The Black Swan), analyses of the results, detailed descriptions of several of the competing methods, and many discussions and commentaries (including my own on "The value added by machine learning approaches in forecasting").

IIF Membership Promotion Expires January 15

Join the International Institute of Forecasters and receive both hardcopy subscriptions and online access to two journals:

  • International Journal of Forecasting
  • Foresight: The International Journal of Applied Forecasting

Through January 15, new IIF members receive a bonus Foresight Guidebook (choices below), plus free access to the IJF M4 issue. Current members already have online access to the IJF M4 issue, and get extra savings (of $20 or $30) on 2- or 3-year renewals.

* Foresight Guidebooks are collections of previously published articles on specific topics. Choices are:

  • Artificial Intelligence
  • Forecast Accuracy Measurement
  • Forecasting Product and Temporal Hierarchies
  • Forecasting Method Tutorials
  • Forecasting Process: Guiding Principles
  • Improve Your Forecasting Process

About Author

Mike Gilliland

Product Marketing Manager

Michael Gilliland is a longtime business forecasting practitioner and currently Product Marketing Manager for SAS Forecasting. He is on the Board of Directors of the International Institute of Forecasters, and is Associate Editor of their practitioner journal Foresight: The International Journal of Applied Forecasting. Mike is author of The Business Forecasting Deal (Wiley, 2010) and editor of the free e-book Forecasting with SAS: Special Collection (SAS Press, 2020). He is principal editor of Business Forecasting: Practical Problems and Solutions (Wiley, 2015) and Business Forecasting: The Emerging Role of Artificial Intelligence and Machine Learning (Wiley, 2021). In 2017 Mike receive the Institute of Business Forecasting's Lifetime Achievement Award. In 2021 his paper "FVA: A Reality Check on Forecasting Practices" was inducted into the Foresight Hall of Fame. Mike initiated The Business Forecasting Deal blog in 2009 to help expose the seamy underbelly of forecasting practice, and to provide practical solutions to its most vexing problems.

Leave A Reply

Back to Top