Hello Michael, I completely agree with your view points. In fact it is a very standard practice in data mining community to see the model performance on a hold out sample. It is really tricky to do it for forecasting problems as we cannot randomly select the sample (as would be the case with normal data mining exercise). One approach you outlined above (point 2) is really interesting and worth exploring (assuming one has sufficient time points). Thanks for the post!