Guest Blogger: Udo Sglavo on Cross-validation using SAS Forecast Server (Part 2 of 2)

1

In Part 1, Udo provided SAS code to replicate the example in Hyndman's blog.  Below, he shows the results of out-of-sample testing, and draws some conclusions on the computational efficiency of this approach.

Out-of-sample Testing

In addition to the example shared by Hyndman, out-of-sample data was used to illustrate the final performance of the winning model. The last 12 month of A10 were used as out-of-sample data, which changes the MAE plot above, as less data is available for cross-validation (again, ARIMA is the preferred model):

The new cross validation plot:

Forecast plot:

Forecast region only plot:

 

 Conclusion

Time series cross-validation as suggested by Hyndman can be implemented using SAS High-Performance Forecasting today. Cross-validation is a computationally intensive approach to assessing forecast model performance, so this needs to be taken into account when trying to apply it on large scale data. Nevertheless, each iteration of cross-validation (for each lead time) can be run independently, so a multi-threaded approach might improve the overall run-time (by running on several processors in parallel). It should also be noted that in this example the forecasting models candidates are already known, which might not be the case in large-scale environments for all series at hand.

Tags
Share

About Author

Mike Gilliland

Product Marketing Manager

Michael Gilliland is author of The Business Forecasting Deal (the book), and editor of Business Forecasting: Practical Problems and Solutions. He is a longtime business forecasting practitioner, and currently Product Marketing Manager for SAS Forecasting software. Mike serves on the Board of Directors for the International Institute of Forecasters, and received the 2017 Lifetime Achievement in Business Forecast award from the Institute of Business Forecasting. He initiated The Business Forecasting Deal (the blog) to help expose the seamy underbelly of forecasting practice, and to provide practical solutions to its most vexing problems.

Back to Top