Alfred Hitchcock and a classic forecasting scam

The Forecasting Savant

Suppose you received an email from a self-proclaimed forecasting savant, advising you of a big upset in the upcoming mayoral election...and it turns out to be correct.

You then get an email picking the underdog in the next championship boxing match...which is right again.

Over the course of a few weeks you receive four more such emails, predicting outcomes that turn out to be true.

After six correct picks in a row, this savant has proven his ability, wouldn't you think? And he certainly deserves a cut of earnings on the wagers you've started placing on these forecasts, as well as a healthy gratuity for the forthcoming 7th prediction, a stock pick which is promised to make you rich.

But has the savant established his skills as a forecaster, and does he deserve a big advance for his 7th prediction? Does it make sense to take a large position in the recommended stock?

What Would Alfred Hitchcock Do?

The above scenario (substituting mail for email) plays out in Season 3, Episode 2 of the 1950's television show, "Alfred Hitchcock Presents." Entitled "Mail Order Prophet," E.G. Marshall is the gullible (but so far money winning) recipient of the prophet's prophecies, while Jack Klugman is the skeptical office mate.

Klugman warns Marshall early on, "Don't be an idiot! Nobody can predict the future -- it's impossible." But after failing to act on the first two letters, Marshall is ahead over $1000 placing wagers on the last four, winning them all. He seeks out Klugman's advice on the latest letter, in which the supposed savant requests remuneration for his correct predictions, and promises his next tip is for a stock that will grow 10-fold.

After commenting on Marshall's natty new suit ("Seersucker no doubt? For every seer there's gotta be a sucker") Klugman continues his rant against forecasting, "Predicting the future is a scientific impossibility!" and warns that this is some kind of scam. Yet Marshall remains convinced of his savant's infallible prescience, and "borrows" $15,000 of company funds to buy $30,000 worth of Athabasca Mines on margin.

How You Can Become a Forecasting Savant

Have you figured it out (or seen this scam before)? It goes like this:

1) Send out thousands of emails projecting one outcome to half the recipients, the opposite outcome to the other half. (E.g., incumbent wins or challenger wins.)

2) To the half who received the correct prediction, send another email, again projecting one outcome to half the recipients, and the opposite outcome to the other half.

3) Repeat...each time you'll shrink your pool of "suckers" in half, but they'll be all the more convinced of your future-seeing powers by being correct every time.

4) Monetize the process, requesting a gratuity for all the correct predictions you've made so far, with the offer of an additional forecast that will make them huge money.

5) Pocket your gratuities and flee.

Jack Klugman's Critique of Forecasting

While I'm sympathetic to Klugman's critique ("Any intelligent man knows you can't predict the future"), it needs to be tempered just a bit. As forecasters we cannot expect to predict the future with 100% accuracy. But we can still deliver value to our organizations by assessing the uncertainty of the future, and providing guidance that can lead to improved decision making.

Paul Goodwin's recent Foresight/SAS Webinar provides many good ways of expressing forecast uncertainty, including ranges, prediction intervals, fan charts, and probability density charts. The recorded webinar "Getting Real About Uncertainty," and all previous webinars in the Foresight/SAS Webinar Series, are available for on-demand review.

 

Post a Comment

SAS forecasting and econometrics "tip of the week"

SAS Support Communities provide a forum in which to engage and share with your fellow SAS experts, now in over 20 topic areas including Forecasting and Econometrics. This is the go-to website for your hard core modeling questions, such as "Holdouts in PROC ARIMA," "2-stage Heckman (1979) procedure," and the perennially popular "Moving average forecast."

Ken Sanford

Ken Sanford explaining something complicated

My colleague Ken Sanford just announced a new "tip of the week" series for SAS forecasting and econometrics users on the Communities page. Ken provided a tip on controlling your output in the SAS Output Delivery System (ODS).

This week Bobby Gutierrez of the SAS/ETS development team provided a tip on Fixed vs. Random Effects in Panel Data.

If you get the idea that SAS software can help you build just about any kind of model you want, with data as big as you can get, you are correct.

But if you don't have the time, inclination, or skill set to create your own custom models, let the software do it for you. SAS Forecast Server provides large-scale automatic forecasting, and can combine with SAS Grid Manager to spread the workload and handle the biggest forecast challenges. (See how Wyndham Exchange & Rentals utilizes grid computing to generate 1.4 billion forecasts.)

For businesses operating on a more modest scale, SAS Forecasting for Desktop provides a cost-effective entry point to automatic forecasting.

Here is a 5-minute demonstration of the user interface shared by Forecast Server and Forecasting for Desktop.

If you are already a SAS forecasting or econometrics user, you are encouraged to join the community, post your questions, and share your expertise. And if you aren't already a SAS user, come see what you've been missing.

Post a Comment

Foresight/SAS webinar December 3: Getting real about uncertainty

Paul Goodwin

Paul Goodwin

Just a reminder that on Wednesday, December 3, 10:00am ET, Paul Goodwin will be delivering the next Foresight/SAS Webinar on "Getting Real About Uncertainty." From the description:

Many forecasts only tell us about things like probable events, expected sales and prospective demand. Often they tell us nothing about the level of uncertainty we face.

This webinar will discuss recent research suggesting that people make better decisions when forecasts contain information on uncertainty. It will demonstrate how to:

  • Estimate uncertainty.
  • Use forecasting uncertainty to your advantage.
  • Present forecasts in ways that are credible, understandable and useful.

Register for the live webinar, or check back later this week to view the on-demand recording.

For more discussion of Paul's topic, see this BFD post from last month.

Post a Comment

IBF blog series on Forecast Value Added

Calling All Forecasters

Have you tried Forecast Value Added analysis? What did you find out? Are you willing to share your learnings (at least those that can be revealed publicly)?Would you like to be featured in a new blog series on FVA, published by the Institute of Business Forecasting?

The IBF was an early advocate of FVA analysis, from including "FVA" in their Glossary of forecasting terms, to providing a platform for me and other industry forecasters to speak on the subject at IBF events. The blog series will include interviews with forecasting practitioners and provide case studies on the application of the FVA mindset at their organizations.

FVA Stairste Report

FVA Stairstep Report

In the forthcoming December interview, Jonathon Karelse (co-founder of NorthFind Partners) will discuss his use of FVA starting at Yokohama Tire Canada, and now with his consulting clientele.

How to Participate

Interested in sharing your story? Please email me (mike.gilliland@sas.com) to discuss.

For anyone who is publicity shy (due to company PR restrictions, outstanding arrest warrants, debt collectors, etc.), I'm happy to keep your identity (and your company) confidential.

More News from IBF: 2014 Recognition Award Recipients

While visiting the IBF blog site, don't miss announcement of IBF's 2014 Recognition Awards, to Mark Covas (Lifetime Achievement) and John Gallucci (Forecasting & Planning Excellence).

Mark is Group Director of the Planning Center of Excellence at Coca Cola, and a longtime friend of The BFD.* He was recognized for 20 years of contributions to the IBF as a speaker, journal contributor, and advisory board member. As a practitioner, he has led demand planning initiatives across the globe at Levi-Strauss, Johnson & Johnson, Starbucks, Gillette, Proctor & Gamble, and Georgia-Pacific.

John is Senior Director of Product Planning at Pinnacle foods, and was recognized for his use of lean techniques to improve forecasting process efficiency and effectiveness, and for managing ground up implementations of demand planning and S&OP processes. He is also a frequent speaker at IBF events, and contributor to the Journal of Business Forecasting.

Congratulations to this year's recipients!

-----------------

*I have personally made tens of dollars in royalties from copies of The Business Forecasting Deal Mark purchased for his staff.

 

 

Post a Comment

Lancaster Centre for Forecasting survey

LancasterCan you spare a few minutes to assist researchers at the Lancaster Centre for Forecasting?

We have been commissioned by the European Journal of Operational Research to write a review article on Supply Chain Forecasting. In undertaking this task, we would like to ensure that the topics covered reflect the priorities of the forecasting community as a whole. To that end, we have prepared an (anonymous) questionnaire that aims to identify the most important areas that should be addressed in our paper, both from academic and practitioner perspectives.

To participate in this study, visit the Supply Chain Questionnaire. It will take no more than 5-10 minutes of your time. Please indicate if you are an academic or a practitioner in the last section of the questionnaire. We are working to very tight deadlines, so your timely help and participation is appreciated. Should you have any questions or comments on this work please contact Aris Syntetos. Thank you for your time and co-operation,

A.A. Syntetos, M.Z. Babai, J.E. Boylan, S. Kolassa & K. Nikolopoulos

You will recognize lead author Aris Syntetos, who delivered the October Foresight/SAS Webinar on "Forecasting by Temporal Aggregation." (The recorded webinar is now available for on-demand review.) And Stefan Kolassa co-authored a clever Foresight article "Percentage Errors Can Ruin Your Day" which was discussed in the BFD post Tumbling Dice in 2011.

Post a Comment

Gaming the forecast

Business forecasting is a highly politicized process, subject to the biases and personal agendas of all forecasting process participants. This is why many -- perhaps most -- human adjustments to the forecast fail to make it better. And this is why relative metrics, such as FVA, are so helpful in properly evaluating process performance.

The Games People Play

John Mello

John Mello

In his article "The Impact of Sales Forecast Game Playing on Supply Chains" (Foresight, Spring 2009), John Mello provides a useful taxonomy of seven such games that bias the forecasting process and degrade forecast accuracy.

Enforcing: Maintaining a higher forecast than actually anticipated sales, in order to keep forecasts in line with the organization's sales or financial goals.

When the voice of the marketplace is screaming that you aren't going to hit your sales targets, this should be considered a blessing! Perhaps now, knowing about this in advance, you can do something to address the demand shortfall. Unfortunately, many organizations ignore the screaming, and simply adjust their forecast to meet the objective. It is easy to understand why this happens, because as a practical matter, as long as your forecast still meets your target, your boss won't scream at you!

Filtering: Changing forecasts to reflect the amount of product actually available for sale in a given period.

When you are capacity constrained or otherwise experiencing demand greater than supply, a great way to show 100% forecast accuracy is to forecast sales equal to the expected supply.

Hedging: Overestimating sales in order to secure additional product or production capacity.

No sales person wants to explain unfilled orders to his or her customers. How to avoid this? Simply forecast a lot more than you really expect to need. (Just make sure your heightened forecasts go only to the supply planning folks, not to your own sales management. See next.)

Sandbagging: Underestimating sales in order to set expectations lower than actually anticipated demand.

For your sales management, send in low forecasts. That'll help keep your quotas within reach, and win you that award trip to Hawaii.

Second Guessing: Changing forecasts based on instinct or intuition about future sales.

Sometimes there is no data (new products), or the data may be otherwise inappropriate or insufficient. Sometimes you have to make adjustments based on experience and knowledge of the business. Just make sure to track the performance of such adjustments to make sure they are actually improving accuracy and reducing bias.

Spinning: Manipulating forecasts to obtain the most favorable reaction from individuals or departments in the organization.

Control the data, and control the means of measurement, and you can control the message.

Withholding: Refusing to share current sales information with other members of the organization.

Who wants to get yelled at twice? Why tell your boss you aren't going to hit this quarter's numbers (and get yelled at now), when you can just keep forecasting the hockey stick and only get yelled at quarter end after the sales don't materialize.

How many games are going on at your organization?

Post a Comment

Getting real about uncertainty

Paul Goodwin

Paul Goodwin

In his Spring 2014 article in Foresight, Paul Goodwin addressed the important issue of point vs. probabilistic forecasts.

A point forecast is a single number (e.g., the forecast for item XYZ in December is 635 units). We are all familiar with point forecasts, as these are what's commonly produced (either by the software, or judgment) in our forecasting processes.

The problem is that point forecasts provide no indication of the uncertainty in the number, and uncertainty is an important consideration in business planning. Knowing that the forecasting is 635 +/- 50 units may lead to dramatically different decisions than if the forecast were 635 +/- 500 units.

There are a number of ways to provide a probabilistic view of the forecast. With prediction intervals, the forecast is presented as a range of values with an associated probability. For example, "the 90% prediction interval for item XYZ in December is 635 +/- 500 units." This would indicate much less certainty in the forecast than if the 90% prediction interval were 635 +/- 50 units.

A fan chart provides a visual expansion of a prediction interval over multiple time periods. In Goodwin's example, the darkest band represents the 50% prediction interval, while the wider ranges show the 75% and 95% intervals.

Fan Chart

Fan Chart

A probability density chart can provide even more granular detail on the forecast for a single time period. In this example, the most probably sales are around 500 units, but will almost certainly be between 200 and 1200.

Density Forecast

Density Forecast

Goodwin reviews recent literature on the value of communicating uncertainty, and suggests that it can lead to improved decisions. However, research also showed that if interval forecasts are too wide, they were judged to be uninformative and less credible. So even if 635 +/- 500 is the appropriate width of the 90% prediction interval, decision makers may simply ignore the forecast and doubt the competence of the forecaster who produced it!

Estimating the level of uncertainty may be non-trivial, particularly when forecasts are based on human judgment. Research has repeatedly shown that people produce intervals that are far too narrow. Goodwin cites a 2013 study of financial executives providing 80% prediction intervals for one-year-ahead stock prices. Actual returns fell within the 80% intervals only 36% of the time!

A simple way to address inappropriately narrow intervals was suggested by Makridakis, Hogarth, & Gaba in one of my favorite forecasting-related books, Dance With Chance. They suggest taking your estimated prediction interval and doubling it. (I love a quick and dirty solution to a complex problem.)

Learn more in the next Foresight / SAS Webinar

On December 3, 10:00am ET, Paul Goodwin will present his finding in the next installment of the Foresight / SAS Webinar Series.

This webinar will discuss recent research suggesting that people make better decisions when forecasts contain information on uncertainty. It will demonstrate how to:

  • Estimate uncertainty.
  • Use forecasting uncertainty to your advantage.
  • Present forecasts in ways that are credible, understandable and useful.

Register for "Getting Real About Uncertainty in Forecasting" and watch the 30-second video preview.

 

Post a Comment

SAS Summer Fellowships in Forecasting and Econometrics

The Advanced Analytics division of SAS Research & Development has announced three Summer Fellowships in the areas of Forecasting and Econometrics.

The SAS forecasting fellowships are open to doctoral candidates in mathematics, statistics, computer science, and related graduate departments in the United States. They offer the opportunity to work closely with professional statisticians and computer scientists who develop software used throughout the world.

SAS Short-Life-Cycle Time Series Forecasting Fellowship

The forecasting fellow will contribute to research, develop, and document state-space analysis techniques for panels of short-life-cycle time series. Duties include working with SAS statistical forecasting developers to aid statistical computing research initiatives. The program provides an excellent opportunity to explore software development as a career choice.

SAS Energy Forecasting Fellowship

The forecasting fellow will contribute to research, develop, and document for energy forecasting using high-frequency data. Duties include working with SAS statistical forecasting developers to aid statistical computing research initiatives. The program provides an excellent opportunity to explore software development as a career choice.

SAS Econometrics Fellowship

Open to doctoral candidates in economics, statistics, finance and related graduate programs in the United States, this fellowship offers the opportunity to work closely with professional econometricians and statisticians who develop SAS software used throughout the world. The Econometrics Fellow will contribute to activities such as research, numerical validation and testing, documentation, creating examples of applying SAS econometric software, and assisting with software development work. The specific projects assigned may be adjusted to the skills and interests of the Fellow selected. The program provides an excellent opportunity to explore software development as a career choice.

Apply for these (or other SAS jobs and advanced analytics fellowships) at the SAS Careers website.

Deadline for fellowship applications is January 23, 2015.

Post a Comment

Guest blogger: Len Tashman previews Fall 2014 issue of Foresight

Len TashmanIn 2015 Foresight: The International Journal of Applied Forecasting will celebrate 10 years of publication. From high in his aerie in the Colorado Rockies, here is Editor-in-Chief Len Tashman's preview of the current issue:

In this 35th issue of Foresight, we revisit a topic that always generates lively and entertaining discourse, one where business experience has been far more enlightening than academic research: the question of the proper Role of the Sales Force in Sales Forecasting. Our feature article by Mike Gilliland formulates the key aspects of the issue with three questions: Do salespeople have the ability to accurately predict their customers’ future buying behavior, as many assume they do? Will salespeople provide an honest forecast? And does improving customer-level forecasts improve company performance? Incisive commentaries follow Mike’s piece, contributed by forecast directors at three companies.

As Foresight Editor, I welcome continued discussion from our readers on your experiences and lessons learned at your own organizations.

Paul Goodwin’s Hot New Research column addresses a promising new method for properly representing the uncertainty behind a forecast. Called SPIES (Subjective Probability Interval Estimates), it offers a more intuitive way (than standard statistical approaches) for forecasters to determine and present the probability distribution of their forecast errors. I think you’ll find it provocative.

Our section on Forecasting Support Systems features the article Data-Cube Forecasting for the Forecasting Support System. Noted Russian consultant Igor Gusakov draws on his many years at CPG companies to show how we can achieve the best of what are now two distinct worlds, by synthesizing statistical forecasting capabilities with the OLAP (online analytical processing) tools now commonly used for business intelligence and reporting. Data cubes provide the requisite infrastructure.

Igor is also the subject of our Forecaster in the Field interview.

Our Summer 2014 issue included the first part of a feature section on Forecasting by Aggregation. Two articles there examined “temporal aggregation” opportunities, which deal with the choices of time dimension (daily, weekly, monthly, etc.) for forecasting demands.* Now we present Part Two on Forecasting by Cross-Section Aggregation within a product hierarchy. Giulio Zotteri, Matteo Kalchschmidt, and Nicola Saccani question the usual belief that the level of aggregation for forecasting is specified by the operational requirements of the company. Rather, they argue – quite convincingly – that the best level of aggregation for forecasting should be chosen by the forecasters in an attempt to balance the errors from forecasting with data at too granular a level with those at too aggregate a level.

Rob J. Hyndman and George Athanasopoulos extend the discussion by presenting a way for Optimally Reconciling Forecasts in a Hierarchy. Rarely will the sum of forecasts at a granular level equal the forecast at the group level; hence reconciliation is necessary. The authors argue that traditional reconciliation methods – bottom-up, top-down, and middle-out – fail to make the best use of available data. Their optimal reconciliation is based on a weighted average of forecasts made at all different levels of the hierarchy.

----------------------

*Aris Syntetos will discuss "Forecasting by Temporal Aggregation" in the Foresight/SAS Webinar Series on October 30, 11:00am ET.

Post a Comment

Foresight/SAS webinar October 30 on temporal aggregation

On Thursday, October 30, 11 am ET, Aris Syntetos delivers the next installment of the Foresight/SAS Webinar Series, "Forecasting by Temporal Aggregation." Based on his article in the Summer 2014 issue of Foresight, Aris provided this preview:

When we attempt to improve forecast performance we usually consider new or alternative forecasting methods. However, how about not changing the forecasting methods but rather changing our approach to forecasting altogether (by keeping the forecasting methods the same)?

Have you heard about, or ever considered, forecasting by aggregating demand in lower frequency time units (say weekly into monthly demand)? Aggregating demand will almost always reduce demand uncertainty. It also helps to see the linkages between the output of the forecasting process (forecasts needed to support specific decisions) and input (data available to produce the forecasts).

Best of all, it is highly likely that temporal aggregation will improve forecast performance. Ready to see how?  Join us in this on-demand webinar to understand:

  • How forecasting by temporal aggregation works;
  • The different types of temporal aggregation (overlapping and non-overlapping);
  • The data requirements for supporting forecasting by temporal aggregation;
  • The linkage between decision making requirements and the time periods in which demand is recorded.

Aris Syntetos

Aris is Professor of Operational Research and Operations Management at Cardiff University, UK. His research interests relate to supply chain forecasting and its interface with inventory management and he has advised many commercial organizations in this area. In addition, forecasting, demand classification and stock control algorithms co-developed by him have been implemented or are currently considered for implementation by commercial software packages.

Aris is the supply-chain forecasting editor for Foresight. He serves at the Executive Committee of the International Society for Inventory Research (ISIR) and at the Board of Directors of the International Institute of Forecasters (IIF).

Check out the video preview of the presentation, and register for free.

For more discussion of this topic, and to learn how to apply the approach in SAS Forecast Server software, see the BFD post "Forecasting across a time hierarchy with temporal reconciliation."

Post a Comment
  • About this blog

    Michael Gilliland is a longtime business forecasting practitioner and currently Product Marketing Manager for SAS Forecasting. He initiated The Business Forecasting Deal to help expose the seamy underbelly of the forecasting practice, and to provide practical solutions to its most vexing problems.
  • Subscribe to this blog

    Enter your email address:

    Other subscription options

  • Archives