Q&A with Steve Morlidge of CatchBull (Part 3)


Q: ­How important is it to recognize real trend change in noisy data?­

A: It is very important. In fact the job of any forecast algorithm is to predict the signal – whether it is trending or not – and to ignore the noise.

Unfortuantely this is not easy to do because the trend can be unstable and the noise confuses the situation. In fact one of the most common problems with forecasting algorithms is that they ‘overfit’; the data which means that they mistake noise for a signal.

This is the reason why experts recommend that you do not use R2 or other ‘in sample’ fitting statistics as a way of selecting which algorithm to use. The only way to know whether you have made the right is choice is by tracking performance after the event, ideally using statistic like RAE which allows for forecastability and helps you make meaningful comparisons and judgements about the level of forecast quality.

Q: ­From what you've seen, does RAE tend to be stable over time?­

A: No it does not.

In my experience performance can fluctuate over time. This might be the result of a change to the behaviour of the data series but the most common cause in my experience is a change in the quality of judgemental interventions. In particular I commonly see changes to the level of bias – systematic over or underforecasting.

A simple average will not pick this up however. You need to plot a moving average of RAE or even better an exponentially smoothed average as this takes in all available data but gives more weighting to the most recent.

Q: ­All of this assumes that there are no outside attributes available upon which to base a forecast. This all applies then only to situation in which the forecaster has only past data on the item itself to be forecasted. Is this correct?­

A: The main technique used by supply chain forecasters is time series forecasting whereby an algorithm is used to try to identify the signal in demand history which is then extrapolated into he future – on the assumption that the pattern will continue. The implicit assumption here is that the forecast (the dependent variable) is a product of the signal (the independent variable) and this there is only one this approach is termed ‘univariate’.

There are other types of forecasting which use other variables in addition to or instead of the history of the time series. These are known as ‘multivariate’ techniques since there is more than one dependant variable.

Irrespective of the approach used, the limits of forecastability still apply. There is no good reason why any forecasting method should consistently fail to beat the naïve forecast (have an RAE in excess of 1.0). The limits of what is forecastable is a product of the level of noise (which can’t be forecast by any method) compared to the level and nature of change in the signal.

Different techniques – univariate or multivariate – will be more or less successful in forecasting the signal but the same constraints apply to all.


About Author

Mike Gilliland

Product Marketing Manager

Michael Gilliland is author of The Business Forecasting Deal (the book), and editor of Business Forecasting: Practical Problems and Solutions. He is a longtime business forecasting practitioner, and currently Product Marketing Manager for SAS Forecasting software. Mike serves on the Board of Directors for the International Institute of Forecasters, and received the 2017 Lifetime Achievement in Business Forecast award from the Institute of Business Forecasting. He initiated The Business Forecasting Deal (the blog) to help expose the seamy underbelly of forecasting practice, and to provide practical solutions to its most vexing problems.

Leave A Reply

Back to Top