Come on Irene: Time series cross-validation


The hurricane didn't get me, but Monday night's thunderstorm sure played a dirty trick. After leaving my car windows cracked open over night, I drove to work Tuesday morning feeling a little soggier by the minute.  Upon arrival at SAS, I was aghast to find the seat of my pants was soaked (very visibly so).

What to do next?

My first reaction was to cut and run, i.e., drive back home and change into some dry clothes.  I hadn't been in a situation this dire since '93, when I came to work at Oscar Mayer Foods wearing one black shoe and one brown shoe.  (Back then I had no choice but to return home for a change -- you know how poorly the meat packing industry takes to sartorial faux pas.) At least both shoes were mine.

Thankfully, my oversized European Man-Purse saved the day.  Thinking quickly, I adjusted the straps until it hung low behind me, fully outing the damned spot from view.  I then cantered to my office, seemingly unobserved, and spent the next 30-minutes squatting over an air vent to dry.

Time Series Cross-Validation

Tuesday turned out a little better than it started, when my colleague Udo Sglavo pointed me to a Research Tips blog about cross validation for time series by Rob J Hyndman. Time series cross-validation describes a method for forecast evaluation "with a rolling origin," analogous to a leave-one-out cross-validation for cross-sectional data. Last week Hyndman blogged about implementing time-series cross validation in R.  Udo wrote:

Currently SAS Forecast Server does not provide access to cross-validation by default, as it implements a more classical methodology for hold-out sampling, which is designed for large-scale automatic forecasting challenges.

A holdout sample is a subset of actual time series used for model selection. For each candidate model specified, the holdout sample is excluded from the initial model fit and forecasts are made within the holdout sample time range. Then, for each candidate model specified, the statistic of fit specified is computed by using only the observations in the holdout sample. Finally, the candidate model, which performs best in the holdout sample, based on this statistic, is selected to forecast the actual time series.

The cross-validation approach described by Hyndman can be implemented by directly access the batch engine of SAS Forecast Server called SAS High-Performance Forecasting. As certain default functionality of this forecasting engine needs to be “turned off,” some SAS coding is required to replicate the example provided by Hyndman. (I can provide the full code upon request to  Also, SAS High-Performance Forecasting does not provide access to Hyndman’s ETS method; instead a multiplicative Winter’s method is used (for sake for the example).

Over the next two installments of The BFD, Udo will guest-blog his solution to conducting cross validation using SAS Forecast Server.


About Author

Mike Gilliland

Product Marketing Manager

Michael Gilliland is a longtime business forecasting practitioner and formerly a Product Marketing Manager for SAS Forecasting. He is on the Board of Directors of the International Institute of Forecasters, and is Associate Editor of their practitioner journal Foresight: The International Journal of Applied Forecasting. Mike is author of The Business Forecasting Deal (Wiley, 2010) and former editor of the free e-book Forecasting with SAS: Special Collection (SAS Press, 2020). He is principal editor of Business Forecasting: Practical Problems and Solutions (Wiley, 2015) and Business Forecasting: The Emerging Role of Artificial Intelligence and Machine Learning (Wiley, 2021). In 2017 Mike received the Institute of Business Forecasting's Lifetime Achievement Award. In 2021 his paper "FVA: A Reality Check on Forecasting Practices" was inducted into the Foresight Hall of Fame. Mike initiated The Business Forecasting Deal blog in 2009 to help expose the seamy underbelly of forecasting practice, and to provide practical solutions to its most vexing problems.

Comments are closed.

Back to Top