Forecasting across a time hierarchy with temporal reconciliation

The BFD blog probably won't make you the best forecaster you can be. There are plenty of books and articles and world class forecasting researchers (many of them meeting next week in Rotterdam at the International Symposium on Forecasting), that can teach you the most sophisticated techniques to squeeze every last percent of accuracy out of your forecasts. Think of these as your offense in the game of forecasting.

The purpose of The BFD, in this thankless game, is to play defense. To identify the stupid stuff, the bad practices, the things we do that sound like good ideas -- but may just make the forecast worse. In short, to protect you from becoming the worst forecaster you can be.

But sometimes the best defense is a good offense, so let's examine a method known as "temporal reconciliation" that has been getting some recent attention.

Hierarchical Forecasting

Every business forecaster is familiar with forecasting hierarchies. For example, for products:

ITEM ==> PRODUCT GROUP ==> BRAND ==> TOTAL

For manufacturing and distribution:

CUSTOMER ==> DISTRIBUTION CENTER ==> MANUFACTURER ==> TOTAL

For sales management:

CUSTOMER ==> TERRITORY ==> SALES REP ==> SALES MANAGER ==> TOTAL

Some forecasting software packages maintain the hierarchies separately, while others (like SAS Forecast Server), combine the product and geographical / organizational levels into a single hierarchy, such as:

ITEM ==> PRODUCT GROUP ==> SALES REGION ==> DISTRIBUTION CENTER ==> TOTAL

Forecasts at the most granular level of detail (e.g., ITEM) can then be aggregated to any higher level. SAS Forecast Server automatically builds customized models and generates forecasts for each node, at each level of the hierarchy. Using "hierarchical reconciliation" you choose the forecasts at one level and have them cascade down and up the hierarchy, apportioning the numbers so that the whole hierarchy adds up properly.

The value of reconciliation is that it allows you to take advantage of information at each level. For example, there may be consistent seasonal trends that are observable at higher levels, but are difficult to discern at the most granular levels where the data is very noisy. You'll often find that models at the most granular level are flat line -- simply forecasting the average -- since there is too much randomness to detect a seasonal signal. But by applying a seasonal model at a higher level, the seasonality can be pushed down to the granular level through reconciliation. This can give you a better forecast at the granular level.

Forecasting Across a Time Hierarchy

"Temporal reconciliation" is a less familiar approach, utilizing forecasts created in different time buckets. It has been getting a lot of attention lately, with two articles in the forthcoming issue of Foresight. Per Editor Len Tashman's preview:

The two articles in this section address temporal aggregation. In his own piece, Forecasting by Temporal Aggregation, Aris provides a primer on the key issues in deciding which data frequencies to forecast and how to aggregate/disaggregate from these to meet organizational requirements. Then, in Improving Forecasting via Multiple Temporal Aggregation, Fotios Petropoulos and Nikolaos Kourentzes offer a provocative new way to achieve reconciliation of daily through yearly forecasts while increasing the accuracy with which each frequency is forecast.

Petropoulos and Kourentzes provide a summary version in the Lancaster Centre for Forecasting's latest newsletter, with the assertion, "The key idea is that by combining various views/aggregation levels of the series we augment different features of it, helping forecasting models to better capture its structure. The same simple trick can be used to reduce or even remove intermittency in slow moving items making them easier to forecast."

If you are ready to try this, note that SAS Forecast Server has included temporal reconciliation since version 4.1 (released in July 2011). As explained in The BFD at its release:

Temporal Reconciliation: A powerful new tool for combining and reconciling forecasts generated at different time frequencies (e.g. hourly, daily, weekly). Consider demand forecasting in the electric utility industry, where consumption varies by hour within the day (less demand in the middle of the night), by day within the week (business days compared to weekends), and by week within the year (higher peak demand in summer to drive air conditioners). With the SAS Forecast Server Procedures engine inside SAS Forecast Server 4.1, you can model the behavior at each time frequency, and then combine the results to incorporate the various seasonal cycles.