The miracle of combining forecasts

Life gifts us very few miracles. So when a miracle happens, we must be prepared to embrace it, and appreciate its worth.
Dogs in snow

Winter Storm Pax

In 1947, in New York City, there was the Miracle on 34th Street.

In 1980, at the Winter Olympics, there was the miracle on ice.

In 1992, at the Academy Awards, there was the miracle of Marisa Tomei winning the Best Supporting Actress Oscar.

And in 2014, on Wednesday afternoon this week, there was the miracle of getting off the SAS campus in the middle of winter storm Pax.

There are also those "officially recognized" miracles that can land a person in sainthood. These frequently involve images burned into pancakes or grown into fruits and vegetables (e.g. the Richard Nixon eggplant). While I have little chance of becoming a saint, I have witnessed a miracle in the realm of business forecasting: the miracle of combining forecasts.

A Miracle of Business Forecasting

Last week's installment of The BFD highlighted an interview with Greg Fishel, Chief Meteorologist at WRAL, on the topic of combined or "ensemble" models in weather forecasting. In this application, multiple perturbations of initial conditions (minor changes to temperature, humidity, etc.) are fed through the same forecasting model. If the various perturbations deliver wildly different results, this indicates a high level of uncertainty in the forecast. If the various perturbations deliver very similar results, the weather scientists consider this reason for good confidence in the forecast.

In Fishel's weather forecasting example, they create the ensemble forecast by passing multiple variations of the input data through the same forecasting model. This is different from typical business forecasting, where we feed the same initial conditions (e.g. a time series of historical sales) into multiple models. We then take a composite (e.g. an average) of the resulting forecasts, and that becomes our combined or ensemble forecast.

In 2001, J. Scott Armstrong published a valuable summary of the literature in "Combining Forecasts" in his Principles of Forecasting. Armstrong's work is referenced heavily in a recent piece by Graefe, Armstrong, Jones, and Cuzan in the International Journal of Forecasting (30 (2014) 43-54). Graefe et. al. remind us of the conditions under which combining is most valuable, and illustrate with an application to election forecasting. Since I am not much fond of politics or politicians, we'll skip the elections part, but look at the conditions where combining can help:

  • "Combining is applicable to many estimation and forecasting problems. The only exception is when strong prior evidence exists that one method is best and the likelihood of bracketing is low" (p.44). ["Bracketing" occurs when one forecast was higher than the actual, and one was lower.] This suggests that combining forecasts should be our default method. We should only select one particular model when there is strong evidence it is best. However in most real-world forecasting situations, we cannot know in advance which forecast will be most accurate.
  • Combine forecasts from several methods. Armstrong recommended using at least five forecasts. These forecasts should be generated using methods that adhere to accepted forecasting procedures for the given situation. (That is, don't just make up a bunch of forecasts willy-nilly.)
  • "Combining forecasts is most valuable when the individual forecasts are diverse in the methods used and the theories and data upon which they are based" (p.45). Such forecasts are likely to include different biases and random errors -- that we expect would help cancel each other out.
  • The larger the difference in the underlying theories or methods of component forecasts, the greater the extent and probability of error reduction through combining.
  • Weight the forecasts equally when you combine them. "A large body of analytical and empirical evidence supports the use of equal weights" (p.46). There is no guarantee that equal weights will produce the best results, but this is simple to do, easy to explain, and a fancier weighting method is probably not worth the effort.
  • "While combining is useful under all conditions, it is especially valuable in situations involving high levels of uncertainty" (p.51).

So forget about achieving sainthood the hard way. (If burning a caricature of Winston Churchill in a grilled cheese sandwich were easy, I'd be Pope by now). Instead, deliver a miracle to your organization the easy way -- by combining forecasts.

[For further discussion of combining forecasts in SAS forecasting software, see the 2012 SAS Global Forum paper "Combined Forecasts: What to Do When One Model Isn't Good Enough" by my colleagues Ed Blair, Michael Leonard, and Bruce Elsheimer.]

Post a Comment

WRAL weather forecaster more than a pretty face

I've always thought of TV weather forecasters as just talking heads. Sure they look pretty, waving hands in front of fancy green-screen graphics, reading poetically off the teleprompters, and standing fearlessly in the midst of the worst storm conditions. But could we expect man candy as tart as Al Roker and Willard Scott to actually know anything about science and math?

Well, maybe not Al and Willard. But Greg Fishel, Chief Meteorologist at WRAL in Raleigh, is bringing the goods.

In a recent post on the WRAL WeatherCenter Blog by Nate Johnson, Fishel is interviewed on the topic of ensemble forecasting. This 10 minute video is worth a look.

Deterministic vs. Ensemble Weather Forecasting Models

First Fishel describes the traditional "deterministic" weather model. In this approach, observered initial conditions (temperature, pressure, etc., from various observation points) are fed into a computer model. These initial conditions provide the current state of the atmosphere, from which the model derives the state of the atmosphere at some point (e.g. 7 days) in the future.

Everyone realizes that we can't expect a perfect prediction of next week's weather, which is what the deterministic model purports to deliver. In fact, we don't even have perfect knowledge of the current state of the atmosphere, since we have only a finite number of weather monitors reporting conditions at particular locations.

The ensemble approach, as Fishel explains, takes the initial condition data, and perturbs the data points (e.g. slightly changing the temperature and pressure at each point, in various ways), creating an ensemble of perhaps 50 sets of initial condition data. Each variation of initial conditions is run through the same model, and the resulting solutions are compared.

If all versions give essentially the same result a week out, this would imply that the atmosphere is not overly sensitive to small variations in initial conditions, and this would merit more confidence in the forecast.

If the different versions of input data resulted in wildy different forecasts, we might have much less confidence in our weather prediction.

Application to Business Forecasting

What is happening here, which is a very good lesson for business forecasters as well, is acknowledgment that it is impossible to make a perfectly accurate forecast. As Fishel puts it, "I don't think there is anything wrong, in a highly uncertain situation, to be honest with the public and say 'we don't know how this will play out, but here are the most likely scenarios.'" Then as a consumer of the weather forecast, you can plan accordingly.

An indication of uncertainty is a valuable addition to the typical "point forecast" that just tells us one number. For example, telling your inventory planner that the demand forecast is for "100 +/- 100 units" might lead to a different inventory position than a forecast of "100 +/- 10 units."

So forget about Al, Willard, and the long list of celebrities* who served up the weather at some point in their careers. Not only is Greg Fishel good looking enough to land a nightly gig on a mid-market CBS affiliate, he can teach us a thing or two about forecasting!


*Including David Letterman, Pat Sajak, and Raquel Welch.

Post a Comment

IBF Scottsdale: FVA at Cardinal Health

IBF Conference Brochure CoverWhere is global warming when you need it?

Throughout much of the southeast, life has been at a standstill since midday yesterday, when 2" of snow and 20oF temperatures brought civilization to its knees. If your life, or at least your forecasting career, is at a similar standstill, make plans to join us February 23-25 for the Institute of Business Forecasting's Supply Chain Forecasting Conference in Scottsdale, AZ.

February is a great time to be in Arizona, with beautiful weather and the rattlesnakes still in hibernation. The IBF event offers a full day Fundamentals of Demand Planning & Forecasting Tutorial by Mark Lawless on Sunday the 23rd, with three tracks of regular sessions Monday through Tuesday morning.

On Tuesday, 9:00-9:55am, join me and Scott Finley, Manager - Advanced Analytics at Cardinal Health, for a look at Forecast Value Added (FVA) analysis. From our abstract:

Forecast Value Added (FVA) is a metric for evaluating the performance of each step and each participant in the forecasting process. FVA compares process performance to essentially “doing nothing”—telling you whether your efforts are adding value by making the forecast better, or whether you are just making things worse! This presentation provides an overview of the FVA approach, showing how to collect the data and analyze results. It includes a case study on how the Advanced Analytics group at Cardinal Health is using FVA analysis to evaluate and improve their forecasting process. You will learn:

  • What data is needed and how to calculate the forecast value added metric
  • How to use FVA to evaluate each step of process performance, identify non-value adding activities, and eliminate process waste
  • How Cardinal Health is using FVA analysis to evaluate their forecasting efforts and guide the evolution of their forecasting process

 Our Gifts to You

As an event sponsor, SAS will be showing SAS Forecasting for Desktop, and will announce the release of a new module in the Demand-Driven Forecasting component of our Supply Chain Intelligence suite of offerings. My colleagues Charlie Chase and Ed Katz will demonstrate the new module on Tuesday 10:00-10:30am.

Business Forecasting Deal book coverDemand-Driven Forecasting book coverEarly risers and the terminally hungry should stop by the SAS booth during Tuesday's breakfast hour (7:00-8:00am) to score a signed copy of The Business Forecasting Deal (the book). Then attend the demo session at 10:00 where Charlie will be signing copies of his latest book, Demand-Driven Forecasting (2nd Edition).

You'll have plenty of good reading for your flight home.

Post a Comment

SAS/Foresight Q1 webinar: The forecasting mantra

Martin Joseph

Martin Joseph

Alec Finney

Alec Finney

In this quarter's installment of the SAS/Foresight Webinar Series, Martin Joseph and Alec Finney of Rivershill Consultancy  discuss "The Forecasting Mantra." Based on their article in the Winter 2009 issue of Foresight: The International Journal of Applied Forecasting, the webinar provides a template that identifies all the elements required to achieve sustained, world-class forecasting and planning excellence.

As previously highlighted in The BFD, Martin and Alec recently published a neat new application of statistical process control methods in their article  “Using Process Behaviour Charts to Improve Forecasting and Decision Making” (Foresight, Fall 2103). This webinar focuses on broader issues of how to:

  • Bring together all elements of the forecasting process.
  • Assess and benchmark your forecasting performance.
  • Design a sustainable, world-class forecasting function.
  • Integrate forecasting with key decision-making business processes.

Watch this brief video with Martin providing a sneak preview of The Forecasting Mantra.

The live webinar is Thursday, February 20, 11:00 am ET. Register for The Forecasting Mantra.

For those who can't attend live, an on-demand recording will be made available soon after the live event.

Submit Your Questions Now!

As a new feature of the SAS/Foresight Webinar Series, we are encouraging attendees to submit questions prior to the event. Submitted questions will be forwarded to the presenters, to help shape the material to audience interests. Send your questions by February 11 to Nancy Rudolph (

As usual, questions can also be submitted during the live webinar, and as many as possible will be answered during the event.

Previous SAS/Foresight Webinars Available for On-Demand Review

Post a Comment

Guest blogger: Len Tashman previews Winter 2014 issue of Foresight

High in the mountains of Colorado, Foresight editor-in-chief Len Tashman previews the new issue:

Len TashmanWhat proficiencies are essential for today’s business forecasters and planners? Sujit Singh offers a detailed and quite formidable list in Critical Skills for the Business Forecaster, our feature article in this 32nd issue of Foresight. While forecasters may not be required to understand all the various forecasting methods at an expert level, it is very important that they know the levers to pull to control the output from their forecasting solution. As Sujit notes, “Proper analysis by the forecaster will often show a clear separation between situations that should be forecast statistically and those that require manual input.”

A major contributor to Foresight in the past year, Sujit is also the subject of this issue’s Forecaster in the Field interview.

Forecasting Support Systems Editor Stavros Asimakopoulos then teams up with George Boretos and Constantinos Mourlas on an article that projects significant potential benefits to forecasters and planners from our many mobile devices. Forecasting “In the Pocket”: Mobile Devices Can Improve Collaboration points out that smartphones, tablets, and such offer a boon to forecasters by simplifying information flow and enabling more timely adaptations to new information. The onus is on vendors to design m-forecasting applications that optimize our mobile experience.

We continue our series of Forecasting Methods Tutorials with Geoff Allen’s primer on Regression Modeling for Business Forecasting. Regression is the principal statistical approach to incorporating business drivers into the forecasting model. Geoff takes us through the key attributes for understanding the structure of a regression model, the conditions for obtaining reliable regression forecasts, and the diagnostic tests that help improve model specification.

Our section on Forecasting Principles and Practices begins with Do Forecasting Methods Reduce Avoidable Error? Evidence from the Forecasting Competitions, Steve Morlidge’s careful and insightful look at what these so-called “competitions” tell us about choosing appropriate forecasting methods.

Nobel economist Paul Krugman wrote in 2009 that “[t]he economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth.” David Orrell, author of Truth or Beauty: Science and the Quest for Order (2012), now asks whether this same hubris has applied to forecasters in general. In The Beauty of Forecasting, David concludes that because “living systems…resist the tidiness of mathematical laws,” it is a risky business indeed to assume that these systems we seek to analyze are either easily depicted or predictable through elegant equations.

Tao Hong concludes this section with an engaging historical overview of the objectives of energy forecasting and the evolution of the methodologies applied to this task – from simply counting lightbulbs in the earliest days of projecting load demands to the demand-response forecasting and renewable-generation forecasting required in the smart grid era. See his Energy Forecasting: Past, Present, and Future.

Post a Comment

Forecasting new products (Part 5): Model, forecast, override

We ended last time having selected a cluster of surrogate products -- a subset of the original selection of like-items that had the same attributes as the new product. Judgment has been used throughout the process so far, in specification of the relevant attributes, filtering the original candidate pool of like-items, and selecting a cluster to best represent the new product.

In the Model step, we specify a statistical model that fits the surrogate cluster. Again, we are using judgment to select a type of model and set its parameters. Below, we see a model that has been fit to the surrogate data, and the Forecast step provides the weekly forecast along with upper and lower limits to represent the most likely range of outcomes.

Model fitted to surrogate cluster

In the final Override step, judgment is used to manually adjust the model's forecasts. Once any overrides are made, the new DVD forecasts can be exported to downstream planning systems.

Applying the Structured Analogy Approach

The structured analogy approach can be useful in many (but not all) new product forecasting situations. It augments human judgment by automating the historical data handling and extraction, by incorporating statistical analysis, and by providing visualization of the range of historical outcomes.

Software makes it possible to quickly extract candidate products based on the user-specified attribute criteria. It aligns, scales, and clusters the historical patterns automatically, making it easier to visualize the behavior of past new products. This visualization helps the forecaster realize the risks, uncertainties, and variability in new product behavior.

Expectations for the accuracy of new product forecasts should be modest, and acknowledgment of this uncertainty should be at the forefront. The structured analogy approach allows the organization to both statistically and visually assess the likely range of new product demand, so that it can manage accordingly. Rather than lock in elaborate sales and supply plans based on a point forecast that is likely to be wrong (and possibly very wrong), the organization can use the structured analogy process to assess alternative demand scenarios and mitigate risk.

Judgment is always going to be a big part of new product forecasting. A computer will never be able to tell us whether Lime Green or Day-Glo Orange is going to be the hot new fashion color, but judgment needs assistance to expose biases and keep it as objective as possible.

While the structured analogy approach can be used to generate new product forecasts, I find its greatest value in assessing the reasonableness of forecasts that are provided from elsewhere in the organization. The role of structured analogy software is to do the heavy computational work and provide guidance—making the NPF process as automated, efficient, and objective as possible.

Think of it as your BS detector for new product forecasting.

[The structured analogy approach to new product forecasting was developed by my colleagues Michael Leonard, Tom Dickey, Michele Trovero, and Sam Guseman. For more information see the downloadable white paper "New Product Forecasting Using Structured Analogies," or the article "Forecasting New Products by Structured Analogy " in the Winter 2009-10  issue of Journal of Business Forecasting.]



Post a Comment

SAS #2 in Fortune Best Companies to Work For

Seems like we've been here before.

It is January, so time again to announce the Fortune 100 Best Companies to Work For in the US. This year SAS is at #2, our fifth straight year in the top 3, over which our average rank has been 1.8.

We've covered this topic a couple of times before (2011: Best Company to Work For -- Again and 2013: What good is being a "great place to work"?), so you can refer to these previous blog posts for details. But I would like to point out a nice video that appeared on NBC's Today show this morning. (My only disappointment is that, just like when they shot Iron Man 3 in my building in 2012, I wasn't invited to represent the face of SAS. Apparently I have a face better suited to blogging.)

The best part of being a great place to work is that it provides a great environment for providing value to customers. None of this matters and none of this will last -- none of the perks or anything else -- unless SAS employees can deliver software and services that make a difference. That's what we are trying to do, and shown we can do, through 38 consecutive years of profitable growth.

Post a Comment

Forecasting new products (Part 4): Query, filter, and cluster

The Query step begins by selecting like-items based on the appropriate product attributes, then reviewing historical sales of past new product introductions.

Continuing with the DVD example, suppose the new release is an R-rated horror movie. For like-items, we would query our database and pull the history of all prior DVD releases with the two attributes: Genre = "Horror" and MPAA Rating = "R".

Note that we are using judgment to determine which attributes are most relevant -- the system doesn't tell us this. But the system does automate the work of extracting R-rated horror movies from the data set of all previous DVD releases, and aligns the history on a common timeline (weeks after release). These extracted items form our pool of candidate products.

The output from the Query step is a graphical overlay of the weekly sales of all candidate products (shown below for the first 20 weeks after release).

Profile graph candidate DVDs

Judgment again comes into play in the Filter step. The graph of candidate products points out a peculiar bump in week 5 sales for the movie Dawn of the Dead. If we decide that Dawn of the Dead is an outlier, we can filter it out of the candidate pool.

The Filter step lets us remove candidates we judged as inappropriate for whatever reasons. For example, if the week 5 bump can be explained by a special promotion we ran for Dawn of the Dead, that we don't plan to run for the new release, then it may make sense to remove it.

After filtering is applied, the Cluster step groups the remaining candidates according to similarity of their sales pattern. Below is one of the three main clusters that was formed (with the vertical axis showing % of sales rather than units sold).

Cluster of DVDs

Note that the system does not tell us which cluster is most appropriate for forecasting the new product (or whether we should cluster at all or just use every candidate). This step simply provides the option of reviewing clusters, and again using judgment. Whatever we decide to do (use a particular cluster, or instead use all the candidates), we refer to these as our surrogate products.

Next time we'll examine how you might model and generate the new product forecast from your surrogates.

Post a Comment

Forecasting new products (Part 3): By structured analogy

In the previous installment we were reminded of the potential abuses of forecasting by analogy. People are naturally reluctant to forecast that their new product idea is going to flop. Therefore, there is an inclination to ignore similar items that failed in the marketplace, or apply less weight to the failures than to the successes in calculating the new product forecast.

The structured analogy approach tries to address these abuses. It utilizes data visualization and analytical software to combine the use of analogies with structured judgment, forcing you to make conscious (and hopefully less biased) decisions along the way. The approach begins with two types of data: attributes (of prior products and new products) and historical sales (of prior products).

Attributes can be just about anything relating to the product, such as its type, price point, target market, function or purpose, as well as physical characteristics like color or size. You need to have a file of all previously released products with their attributes, as well as know the attributes of forthcoming new products.

Historical sales of past new product introductions provides both the volume of sales and the shape or profile of the sales pattern during the introductory period. For example, you might capture weekly sales for the first X number of weeks.  As we saw last time, it is helpful to display scaled thumbnails of the historical sales patterns, to get a sense of how similar (or variable) these patterns can be.

Profiles of 100 DVD introductory sales

These scaled profiles of the first eight weeks sales for 100 movie DVDs follows a pattern that is unlike most other new product introductions. For movie DVDs, peak sales usually occur in the first week of release, and quickly taper off after that. The distinctiveness of this pattern makes it a good example for illustrating the structured analogy approach.

Six Steps of the Structured Analogy Process

  1. Query: Identify a set of candidate products that have similar attributes to the new product
  2. Filter: Option to manually remove inappropriate or outlier products from the set of candidate products.
  3. Cluster: Cluster the candidate products according to their sales pattern and use judgment to manually select the most appropriate cluster to serve as the surrogate products.
  4. Model: Select the most appropriate statistical model for the cluster of surrogate products.
  5. Forecast: Use the statistical model to generate a forecast for the new product.
  6. Override: Option to make manual adjustments to the statistical model's forecast.

It is obvious that judgment still plays a major role in this process. In each step some kind of decision has to be made (specifying the relevant attributes in Step 1, removing any inappropriate or outlier products in Step 2, etc.). But each decision is now transparent, documented, and subject to management scrutiny.

In the next installment we'll work through these six steps with the DVD sales data.


Post a Comment

Forecasting new products (Part 2): By analogy

Image of house for saleThe real estate market provides a good example of the use of analogies.

To determine a reasonable listing price for a property (such as this dump on the right) that is new on the market, the sales agent will prepare a list of "comps"  (comparable homes) that are currently on the market or have recently sold. The comparable homes will have similar attributes, such as square footage, number of bedrooms and baths, and lot size. The prices of these comps (which serve as "like-items") are used to suggest an appropriate listing price for the new property -- and a forecast of what the property should sell for.

Forecasting by analogy is, perhaps, the second most common new product forecasting (NPF) practice. (I believe the most common NPF practice is making up whatever number you want.) Forecasting by analogy assumes that demand for a new product will be similar to demand for like-items from the past.

Like-items are choosen based on product attributes (such as style, color, size, or purpose) or other criteria, and the forecast for the new item is based on a composite of the sales history of the like-items. Like all NPF approaches, there is a degree of judgment involved. There is judgment in the specification of like-item attributes. And there is judgment in how to compute the composite forecast from the histories of the like-items (simply average them, or use another method).

One of the "worst practices" that can occur in this approach is the selective choice ("cherry picking") of past new products, only using those that were successful. Like-items that failed in the marketplace may be forgotten (or purposely ignored), leading to overly optimistic expectations for the new product.

Data Requirements

To utilize analogies, you must first pull the history of past new product introductions, and then align the profiles of their sales during the introductory period. This requires a fair amount of data extraction and manipulation to create a chart like the one below, which shows the first eight weeks of sales for 100 new movie releases on DVD.

First 8 weeks of sales for 100 DVD releases

In the next installment we'll discuss new product forecasting by structured analogy, which begins with the extraction of these past new product profiles.


Post a Comment
  • About this blog

    Michael Gilliland is a longtime business forecasting practitioner and currently Product Marketing Manager for SAS Forecasting. He initiated The Business Forecasting Deal to help expose the seamy underbelly of the forecasting practice, and to provide practical solutions to its most vexing problems.
  • Subscribe to this blog

    Enter your email address:

    Other subscription options

  • Archives