When do you stop trying to improve forecast accuracy? (Part 1)

Mike Gilliland
Mike Gilliland|May 24, 2013

If the popularity of one's blog can be measured by the number of comments received, then The BFD has become quite popular.

Many of the comments are quite flattering, such as:

Hello, I check your blog like еvery week. Үour writing style is wittу, keep doing ωhat you're doing!

Vеry shortly this site will be famοus аmong all blogging and ѕitе-building users, ԁue to it's pleasant posts

Oh my goodness! Incredible article dude! Many thanks

Some are more tempered in their enthusiasm:

Looking at this article reminds me of my previous roommate!

The idea reminds myself a small amount of lipstick removal that's another account wholly.

A few are quite harsh in their critique:

I used to be able to find good information from your blog posts.

And one was just difficult to interpret:

I have read not one article on your blog. You’re a big lad.

(Any idea what that is supposed to mean?) I just hope our newly installed spam filtering software doesn't intercept all these...I'll miss them.

Q:When Do You Stop Trying to Improve Forecast Accuracy?

Despite the popular exhortations to never quit, never give up, and never stop improving, there may be some good reasons to stop trying to improve your forecast, and focus resources elsewhere. Some rules of thumb:

1. Is your forecast accuracy good enough to meet your business needs? If so, don't waste resources building fancier models or developing a more elaborate process. If forecast accuracy is not constraining your overall performance, move on to the next problem.

2. Have you considered the consequences of a less-than-perfect forecast? If the costs and consequences are small, why waste time trying to get great forecasts? Or at least focus any improvement efforts on those forecasts that have the most impact on your business.

On the other hand, if you've conducted a rudimentary FVA analysis and determined that you are forecasting worse than a naive model, then this is no time to quit trying. The most fundamental objective of the forecaster is "First, do no harm." If all your people and software and elaborate processes are performing worse than a naive model, then there is room for improvement.

More rules of thumb in the next installment...

Post a Comment

Forecasting webinars

Mike Gilliland
Mike Gilliland|May 6, 2013

"Why Should I Trust Your Forecasts?" now available on-demand

The SAS / Foresight webinar series had a rousing kickoff on April 24, with Paul Goodwin asking (and answering) the question, "Why Should I Trust Your Forecasts?"

The webinar is now available for free on-demand review . Be sure to stick around for the engaging Q&A session after Paul's presentation.

"FVA: A Reality Check on Forecasting Practices" on June 20

Please join us again at 11:00 am EDT on June 20 for the next webinar in the SAS / Foresight series, "FVA: A Reality Check on Forecasting Practices." I'll be delivering the content, based on an article in the Spring 2013 issue of Foresight.  Here is the abstract:

Forecast Value Added (FVA) is the change in a forecasting performance metric that can be attributed to a particular step or participant in the forecasting process. The concept turns attention away from the end result (forecast accuracy) to focus on the overall effectiveness of the forecasting process. FVA has caught on in many companies as an aid in eliminating unnecessary and even harmful actions. This webinar shows how to gather the data and conduct FVA analysis, with examples from companies that have applied this approach.

"Get to Know the IBF" on May 8

Anish Jain, managing director of the Institute of Business Forecasting, speaks on  how to "Get to Know the IBF" at 10:30 am EDT on May 8. Anish will cover some of the offerings provided by the IBF, including:

  • Benchmarking research
  • Certification
  • Training
  • Events
  • Journal of Business Forecasting
  • Books
  • Membership
  • Recognition Awards

Registration is free.

 

Post a Comment

Guest Blogger: Len Tashman previews Spring 2013 issue of Foresight

Mike Gilliland
Mike Gilliland|May 6, 2013

Editor Len Tashman's Preview of Foresight

For a look at articles in the Spring issue of Foresight: The International Journal of Applied Forecasting, here is editor Len Tashman's preview:

Kevin Foley is an IIF–Certified Forecaster with over 15 years of consultant experience in defense and aerospace companies. Drawing on this background and experience, Kevin guides us through the maze of Forecasting Revenue in Professional Service Companies. The key is understanding the contractual pipeline from beginning to end – from the identification of new opportunities to client billing – and making careful estimates for the probability, timing, and value of new and existing contracts.

Mike Gilliland [yes, that is me] has long advocated the use of forecast value added (FVA) to determine if each component aspect of the forecasting process is working to add value (improve accuracy) – or actually worsening it. In FVA: A Reality Check on Forecasting Practices, Mike reviews the logic behind FVA and tells us how it is being applied in four companies.

Our section on S&OP this issue has two superb discussions. In S&OP and Financial Planning, John Dougherty and Chris Gray, the authors of S&OP: Best Practices (2006), lend their insight and expertise to show how constant comparison and reconciliation between operating and financial plans in the S&OP process can help keep each functional area attuned to contribute to a firm’s overall objectives.

Then, for businesses with an effective S&OP process already in place, John Mello’s article Collaborative Forecasting: Beyond S&OP examines the benefits a firm can reap by collaborating with supply-chain partners to share forecasts and replenishment plans, as well as manufacturing and merchandising strategies. But John notes that Collaborative Planning, Forecasting, and Replenishment (CPFR) isn’t a free lunch, and he lays bare key obstacles in its implementation.

We read over and over again about "black swans", an extreme form of rare events that, while very unlikely to occur, can cause great upheaval if they do. In Rare Events: Limiting Their Damage Through Advances in Modeling, Gloria González-Rivera summarizes the state of the art in the forecasting of rare events and recommends strategies to mitigate the damages if they recur.

We asked Tom Willemain to have a look at a new forecasting text, Practical Time Series Forecasting: A Hands-On Guide, by Galit Shmueli of the Indian School of Business, Hyderabad. Tom did so, and found that the book is “a little gem” suitable not just for the classroom; he calls it a wonderful “leave behind” to help corporate forecasters-in-training understand the basics of time-series forecasting. Get all the details in Tom’s review.

The Spring 2013 issue concludes with Ira Sohn’s assessment of the latest National Intelligence Council report, “Global Trends 2030: Alternative Worlds.” The report identifies four megatrends and six game changers that, in various combinations, create four possible or potential worlds for the year 2030. For those of you who plan to be around 17 years from now, it’s compelling stuff.

Post a Comment

See SAS (=Stark Industries) in Iron Man 3

Mike Gilliland
Mike Gilliland|May 3, 2013

SAS World HeadquartersWhen you work at headquarters of the leader in advanced analytics software, you never know who you'll encounter in the lobby. It might be celebrity statistician (and New York Times FiveThirtyEight blogger) Nate Silver, of The Signal and the Noise and election forecasting fame. It might be Donald Wheeler, giant in the quality control field, Deming Medal winner, and author of one of my favorite books, Understanding Variation.

Or, it might even be People magazine's most beautiful woman in the world, Gwyneth Paltrow, on the set of Iron Man 3.

In the movie, SAS serves as headquarters for Stark Industries, and last June we enjoyed a day of filming with Gwyneth, Robert Downey Jr., Jon Favreau, and the whole IM3 gang. There was even a casting call for SAS employees to serve as extras.

Despite my multiple efforts, encouraged by someone who called himself "the casting director," it turned out you couldn't just earn a part the old fashioned way. Instead, there were three requirements:

  1. You had to have a luxury vehicle
  2. You had to be well dressed and well groomed
  3. You had to be under 50 years old.

Unfortunately, I failed miserably on all three. But at least several of my younger, richer, and better looking colleagues made the final cut.

Iron Man 3 opens today at a theater near you.

[See this full account of How SAS became Stark Industries for a day.]

Post a Comment

SAS / Foresight webinar series debuts April 24

Mike Gilliland
Mike Gilliland|April 12, 2013

This week Nate Silver, renowned election forecaster (fivethirtyeight blog) and top selling author (of the excellent The Signal and the Noise), spoke at an event here in my building on the SAS campus. Unfortunately, I wasn't considered a B enough of a FD to land an invite to Nate's presentation. However, I will provide links to the recording of his comments when they become available.

SAS / Foresight Webinar Series

April 24 marks the debut of the new SAS/Foresight webinar series, produced by SAS along with Foresight: The International Journal of Applied Forecasting. On that date Paul Goodwin, Professor of Management Science at the University of Bath and a regular columnist in Foresight, will speak on his recent article "Why Should I Trust Your Forecasts?" The webinar begins at 11am EDT and will be recorded for later on-demand review. Registration is free.

Goodwin's article (co-authored with M. Sinan Gönül and Dilek Önkal) examines factors that can build (or impede) trust in forecasting. Foremost, we should expect the forecast to be a "competent and honest expectation" of future demand (or whatever else it is we are trying to forecast).

The Determinants of Trust

The authors identify several "determinants of trust" that Goodwin will discuss in his webinar. These include:

  • Perception of the goodwill of the forecast provider
  • Perceived competence or ability of the forecast provider
  • Providing an explanation of the forecast

One benefit of increased trust in a forecast is a reduced tendency to make adjustments. Manual adjustments consume management time (that might be spent on more productive activities), and may contaminate a forecast with biases and personal agendas.

Note, of course, that trust should not be confused with accuracy. A trustworthy forecast, meeting all of Goodwin's guidelines, is not necessarily going to be accurate. (Inaccuracy may be due to chance, or because the phenomena (e.g. product sales) may not be forecastable to the degree of accuracy desired.)

Look for additional installments of the SAS / Foresight webinar series about once a quarter. Topics and registration links will be provided here on The BFD.

 

 

Post a Comment

Is one-number forecasting a new worst practice?

Mike Gilliland
Mike Gilliland|April 2, 2013

The one-number forecasting concept has been debated for years.

Advocates argue that having different groups within the same organization working to different forecasts is insane. You can't have the supply chain building to X, the sales force selling to Y, and the financial folks counting on revenue of Z. This is an invitation for disaster.

But is the imposition of a single forecast number across the whole organization any less insane?

In a SupplyChainBrain blog "There Is No Magic Number for Demand Forecasting," Robert Bowman chronicles discussion of this topic at the recent Institute of Business Forecasting conference. Citing presentations from Nestle, Combe Inc., Syncro Distribution, and Merck, Bowman concludes we should "forget about" fixed one-number forecasting, but we still need to achieve internal and external alignment through the planning process. This is the right conclusion.

The forecast should be an unbiased best guess of what is really going to happen in the future, but we need to recognize (and take into account) the uncertainty in that one number. As Patrick Bower, senior director of corporate planning and customer service at Combe Inc. (and winner of IBF's 2012 award for Excellence in Business Forecasting and Planning) pointed out, what's needed is "banding" around the one number, to indicate the likely range of possible outcomes. (This same point was made by Charles ReCorr in his article "What Demand Planners Can Learn From the Stock Market" as discussed in the previous The BFD blog.)

So the one-number forecasting concept can be rejected because we really should be using at least three numbers: the point forecast, and the upper and lower bounds. This is simple recognition of the reality that our forecasts aren't going to be perfect, and we need to be prepared. As Jonathon Karelse, president of Syncro Distribution, hammered home the point, "You need to plan appropriately for the high side and the low side."

Rejection of one-number forecasting is not the first step toward anarchy. It is instead the rejection of, as Bowman puts it, "A simple answer to a complex problem."

 

Post a Comment

Lessons from forecasting the stock market

Mike Gilliland
Mike Gilliland|March 11, 2013

There is a well recognized phenomenon that combining forecasts, derived from different methods using different sources of information, can improve forecast accuracy. This approach, sometimes called "ensemble forecasting," is available in SAS Forecast Server.

Per Scott Armstrong's review of 57 studies on combining forecasts, "the combined forecast can be better than the best but no worse than the average" of the forecasts being combined.* So when you have a situation where it isn't clear what particular forecasting method is most appropriate, a simple average of several competing methods may be the way to go.

Gathering information from different perspectives is valuable in many other ways. Henry Ford preferred to have an outsider head each new division, because such a person "wasn't already familiar with the impossible."

An outsider provides fresh eyes on the problem. An outsider's viewpoint is not encumbered by the dogmas that go unquestioned by the insiders. We can usually learn a thing or two by observing how the outsider thinks about our problem. So can a demand planner, concerned about filling orders and managing inventories, learn a thing or two from someone who forecasts the price of stocks?

What Demand Planners Can Learn From the Stock Market

The Fall 2012 issue of Journal of Business Forecasting provides us the stock market forecaster's perspective in an article, "What Demand Planners Can Learn from the Stock Market," by Charles ReCorr.

ReCorr begins with the important (but sometimes overlooked) reason why we forecast:

...because all decisions we make require some expectation about the future. Accurate forecasts improve our chances of making the right decision.

He also notes a clear distinction in the decisions to be made and the ability to react to a changing forecast:

The response time for a company to react to, for example, the hint of slowing sales is not the same as that of an institutional investor receiving the same information. A company has to work around production issues, inventory levels, human resource polices and the like before it can respond to changing markets.

The investor, on the other hand, can respond instantaneously with a buy or sell order.

ReCorr identifies seven characteristics that make a forecast useful for his investor clients:

Timeframe - The date or period being forecast (e.g. closing price of the S&P500 on December 31, 2013)

Direction - Is the forecast up or down (compared to today's price or other baseline)

Magnitude - The specific amount or "point forecast" (e.g. S&P500 will be at 1625)

Probability - The distribution of possible outcomes around the point forecast (e.g. 50% chance it will be above 1625, 75% chance it will be above 1575, etc.)

Range - The high and low value for possible outcomes (e.g. 1450 to 1750)

Confidence - A statistically based or subjective "prediction interval" (e.g. 95% confident that it will be between 1550 and 1700)

Historical Forecast Error - Accuracy and bias of previous forecasts

A forecast is useful when it provides enough information to improve decisions under conditions of uncertainty. These characteristics that make a useful stock market forecast would also improve the usefulness of demand forecasts for supply chain planning (where we usually lack anything beyond the point forecast).

Perhaps the least we demand planners can do, in bringing our forecasts to management decision makers, is to fairly present the likely range (and probablility) of possible outcomes. Point forecasts, by themselves, can lead to overconfidence and the taking-on of unnecessary risk. Before making a decision, business managers (like stock market investors) need to know whether (and how) they could live with an outcome that may be far from the point forecast.

---------------

*For a thorough discussion, see Scott Armstrong's chapter on "Combining Forecasts" in his Principles of Forecasting.

Post a Comment

Graduate students available for forecasting projects

Mike Gilliland
Mike Gilliland|February 28, 2013

Back in 2012 I advocated man-dog love and introduced you to my foster dog Mikey. While an endearing little fellow, Mikey did have a bit of a fabric fetish, so his new family has neither curtains nor tablecloths.

Mikey's fetish was discovered while still in foster, when he tore the skirt off a new sofa, and nearly landed himself in the glue factory -- or an IKEA meatball. (Shockingly, the sofa actually looked much better without the skirt. His interior design sense is fabulous!)

Mikey also has a predilection for leather (as in chewing shoes), collars and harnesses (he chewed through three), and latex bones (as seen here). Not that there is anything wrong with any of that.

Host a Master Student Project in Forecasting

Dr. Sven Crone, Director of the Research Centre for Forecasting at Lancaster University, made this announcement on the Institute of Business Forecasting's LinkedIn discussion group:

Need support in a forecasting project? Develop new forecasting algorithms? Setup a forecasting software system? Audit your current forecasting models or processes?

The Lancaster Centre for Forecasting is offering a range of effective Master student projects in forecasting for logistics & supply chain management, government services, call centres, utilities ... and you can determine the topics!

These projects offer a cost efficient way to carry out work for which a company cannot find the internal analytical resources, to buy in external know-how for a project, or to recruit new team members.

All of our Students are well qualified, and trained in forecasting and analytics. They pursue a degree in Management Science, MSc in Logistics & Supply Chain Management or MSc in Marketing Analytics at the esteemed Lancaster University Management School, a triple-accredited, world-ranked management school, consistently ranked among the UK's top.

Contact us today at projects@forecasting-centre.com to develop a master student project with us!

Sven F. Crone
Director, Research Centre for Forecasting
Assistant Professor in Management Science

More information can be found at: http://www.lums.lancs.ac.uk/research/centres/Forecasting/master-student-projects-2013/

Meet Sven Crone at Analytics2013 in London

Sven has been a frequent participant (and past-chairman) of the Analytics, M (data mining) series, and F (forecasting) series events. This June 19-20, Sven will be presenting on "Beyond Forecasting: Time Series Data Mining for New Business Applications " at Analytics2013 in London.

Find more information in the Analytics2013 conference video series.

 

Post a Comment

Role of the sales force in forecasting

Mike Gilliland
Mike Gilliland|February 22, 2013

The war of business forecasting ideas is being waged in the trenches of the online discussion groups. Where else can great disagreement be exacerbated (and sometimes even resolved) by often civilized discussion, with participants from across the globe?

One of the popular groups for business forecasting practitioners is Demand Planning, Sales Forecasting, IBP and Supply Chain Optimization on LinkedIn . There has been a long running discussion about whether the sales team should own forecast accuracy, which has evolved into the broader question of the role of the sales force in forecasting.

[Sidenote: I was thrilled to hear from LinkedIn last week that I have one of the top 100% most viewed LinkedIn profiles for 2012! That is, I was thrilled until a colleague pointed out to me the mathematics of being in the top 100%. Then I became despondent.]

The Role of the Sales Force in Forecasting

It is good to be wary of any inputs into the forecasting process, and this naturally includes inputs from sales. Forecasting process participants have personal agendas, perhaps to drive inventory higher or lower, or to drive sales / revenue expectations higher (as motivational targets) or lower (to make them easier to beat). When we ask someone for a forecast, we shouldn't expect an honest answer.

We can't entirely discount the role of the sales force in forecasting, or as part of the S&OP process. They definitely deserve a seat at the table. But I do want to encourage a healthy skepticism about their input. And let’s be efficient about it – only utilizing their input when they are demonstrably providing value by making the forecast more accurate and less biased.

In some situations, it is more likely that input from the sales force will be valuable. For example, if sales for an item are dominated by one or a small number of customers, insight into the intentions of those dominant customers could dramatically impact our forecast. However, if each item is sold to hundreds or thousands of customers, each accounting for a small percentage of total volume, we probably don’t need the sales force to tell us what individual customers are doing because it just doesn’t matter. Errors in individual customer forecasts just cancel each other out with no net improvement in the forecast for that item.

I would not trust sales to “own” the forecast (and then blindly plan operations around their number). But there may be ways to harvest their information and get some value out of it. As in an example provided by Mark Chockalingam, don’t just give the sales force a blank spreadsheet and tell them to fill in their forecasts – this would be a complete waste of time.

Instead, consider the approach suggested by Eric Wilson of Tempur-Pedic, and appeal to the competitive nature of sales people. Provide them with the forecast generated by your forecasting software, and ask for input where they can improve upon the computer’s forecast. This should limit the adjustments they make (to only those forecasts they are confident they can improve upon), and provides you with the data to determine the effectiveness of their adjustments.

 

Post a Comment

Guest Blogger: Len Tashman previews Winter 2013 issue of Foresight

Mike Gilliland
Mike Gilliland|February 8, 2013

Editor Len Tashman's Preview of Foresight

Foresight has always presented its methods-based articles as either tutorials, which introduce and illustrate a methodology in nontechnical language, or as case studies, with a focus on the practical issues and challenges in generating forecasts. We lead off this issue with two practical issues articles. First, Stephan Kolassa and Roland Martin team up again to examine major challenges of Forecasting to Meet Demand. Then Christian Schäfer presents a case study in the use of simulation and scenarios to forecast the market success of new pharmaceutical products. His article shows How to Separate Risk from Uncertainty in Strategic Forecasting.

Foresight’s feature article in the Fall 2012 issue, “Why Should I Trust Your Forecasts?,” examined the main determinants of trust, including “open communication between forecast users and providers.” In this issue, Joe and Simon Sez expands on the theme with our dynamic duo’s ingredients for Fostering Communication that Builds Trust.

Two very special articles appear in our S&OP section. Jane Lee offers a good dose of reality therapy in her discussion of The Role of S&OP in a Sluggish Economy. Jane argues that, rather than maintaining the pretense of “business as usual,” firms should face the necessity of rapid downward adjustments and utilize the S&OP process for gaining buy-in on new forecasts, new plans, and new production and inventory targets.

Jason Boorman then takes us through the key Five Steps to Gaining Necessary and Appropriate Buy-In for a new S&OP process. It’s a formidable task and success, he says, requires the gaining of commitment from top management and the execution of a successful pilot project.

With so many software solutions out there, it’s hard to imagine room for yet another –but here’s one that really is different. Jeff Greer makes a motivating case for adding Geographic Information Systems software to the supply-chain toolbox in GIS: The Missing Tool for Supply-Chain Design.

The United States has just emerged from a presidential election in which the populace was besieged – nay, bludgeoned – with predictions of the outcome from polls, betting markets, expert opinions, regression models, aggregators, and blogs such as Nate Silver’s FiveThirtyEight for the New York Times. Many polls were wrong beyond statistical expectation (e.g. Gallup), but one clear winner was the aggregator approach, which is based on the averaging of different methods. Andreas Graefe and his team at Pollyvote.com had it consistently right throughout the campaign; how this was accomplished is the subject of Combined Forecasts of the 2012 Election: ThePollyVote.

And speaking of Nate Silver – who outperformed Polly and almost every other predictor in the final two weeks before the elections and has achieved superstar status for his campaign-outcome acumen – he tells all in his new book, The Signal and the Noise: Why So Many Predictions Fail -- But Some Don't. In David Orrell’s review of the book in this issue, he shows the breadth of Silver’s forecasting approach reaching well beyond politics to economics, athletics, and climate.

Post a Comment

  • About this blog

    Michael Gilliland is a longtime business forecasting practitioner and currently Product Marketing Manager for SAS Forecasting. He initiated The Business Forecasting Deal to help expose the seamy underbelly of the forecasting practice, and to provide practical solutions to its most vexing problems.

  • Subscribe to this blog

    Enter your email address:

    Other subscription options

  • Archives