FVA Interview with Jonathon Karelse

Jonathon Karelse

Jonathon Karelse

In December the Institute of Business Forecasting published the first of a new blog series on Forecast Value Added. Each month I will be interviewing an industry forecasting practitioner (or consultant/vendor) about their use of FVA analysis.

The December interview featured Jonathon Karelse, co-founder of NorthFind Partners. Among his key points:

  • Utilize metrics weighted by profit. With limited time and resources, this focuses your attention on actions that impact earnings.

You may not always have reliable data on margin / profit. But if you can trust the numbers, this is a great way to direct your improvement efforts to those products that make the most difference. (Extremely low volume / low revenue / low margin items may not be worth spending any effort on.)

  • Don't overlook measuring forecast bias.

Company forecasts are often overly optimistic. But Jonathon points out the situation where chronic supply shortages have led Sales forecasters to chronically under-forecast (not wanting their targets tied to numbers they don't trust can be built). This can potentially perpetuate the shortages.

  • Compare performance to a naïve model.

The traditional random walk may be considered too simplistic, so Jonathon suggests using a seasonal random walk, simple exponential smoothing, or a moving average. While I suggest always utilizing the random walk as the ultimate point of comparison, I agree that other extremely simple models are appropriate to use for comparison (and they often forecast reasonably well). Early in my career, in a very stable low-growth business, we compared our forecasts to a 52-week moving average.

  • The FVA metric resonates with management.

FVA is easy to understand, and can be a key metric for root cause analysis and corrective action.

Jonathon uses a deseasonalized CV to reduce the risk of false positives. (While high CV generally implies lower forecastability, a highly seasonal item will have high CV but can be quite forecastable if its patterns are consistent and repeating.)

  • Discourage arbitrary performance goals (such as MAPE < 20%).

Focus on improvement. (Here's how to set performance objectives.)

Find the full interview at www.demand-planning.com, and don't miss the money quote:

FVA is easy! If you aren’t using it, you are missing a critical indicator of your organization’s forecasting performance.

Meet Jonathon Karelse at IBF Conference

You can meet Jonathon in person next month, February 22-24, at IBF's Supply Chain Forecasting Conference in Scottsdale, AZ. He will be co-presenting The Art and Science of Forecasting: When to Use Judgment and Forecast Value Add (FVA) Analysis with his client Finning South America.

Coming soon, IBF will post the January FVA interview with Shaun Snapp of SCM Focus.

Post a Comment

Forecasting research project ideas

There are some things every company should know about the nature of its business. Yet many organizations don't know these fundamentals -- either because they are short on resources, or their resources don't have the analytical skills to do the work.

Lancaster Centre for ForecastingThe summer research projects offered by the Lancaster Centre for Forecasting, offer a cost-effective way to get yourself some answers.

Project Ideas

If you haven't done these things already, here are a few of my personal favorite projects to get started:

  • Compare your last year of forecasting performance to a naïve model.

This is the start of any forecasting improvement endeavor -- find out how you are doing today. Don't compare your performance to industry benchmarks, those are irrelevant. Find out whether your process performs at least as well as a simple method, such as a random walk or moving average forecast. (And don't be surprised to learn you are forecasting worse!)

  • Evaluate the volatility of demand for your products or services.

The Coefficient of Variation is a crude and imperfect, yet still useful indicator of the "forecastability" of your demand patterns. Low CV implies that you should be able to forecast fairly accurately with simple methods. High CV implies that you probably can't expect to forecast as accurately -- although some high CV patterns (e.g. something with lots of seasonality but stable, repeating patterns) can be forecast well.

  • Create the "comet chart" relating volatility to forecast accuracy.

Get a visual summary of your forecasting challenges by seeing how volatility and forecast accuracy are related. Use this as motivation to find ways to reduce the volatility of demand patterns.

Map out your forecasting process, and review the last year of forecasts at each step of the process (e.g. statistical forecast, analyst override, consensus override, executive approved forecast). If, like many companies, you aren't recording the data at each step, then start doing so. Use FVA to determine which steps and participants in the forecasting process are tending to make it better. And eliminate those process steps that are just making it worse. (For more information, view the Foresight/SAS Webinar, "FVA: A Reality Check on Forecasting Practices."

  • Replicate Steve Morlidge's analyses of forecast quality.

In a series of articles published in Foresight, Morlidge defined the "avoidability" of forecast error, and illustrated the value of a RAE (relative absolute error) metric for evaluating performance. Read the Foresight articles, find discussion of Morlidge's methodology in  several earlier BFD posts (such as this one), and view his recording from the Foresight/SAS Webinar series, "Avoidability of Forecast Error".

Doing these will give you a good foundation on which to do further research...

Post a Comment

Do you have a forecasting research project?

Lancaster Centre for ForecastingThe Lancaster Centre for Forecasting is seeking Master Student Projects in Forecasting, Data Mining, or Analytics for summer of 2015.

Projects normally run from mid-May to mid-August, with reports issued a few weeks after. These projects are a cost efficient way for a company to carry out analytical work by Master of Science candidates who are formally trained in forecasting. Many of the students have additional skills in areas like marketing analytics, logistics and supply chain, operations research and optimization, and simulation.

Costs are GDP 2,900 (about $4,525 USD) for a single MSc student for four months, plus travel expenses. Discounts available for additional students, non-profits, and SMEs.

Students will normally work on-site at your organization, under the joint supervision of a project leader at your company, and a forecasting expert from the Forecasting Centre. For a well structured problem, the student may remain in Lancaster with no or only occasional site visits.

For more information, and to discuss potential topics, contact:

Dr. Sven Crone

Dr. Sven Crone

Dr. Sven F. Crone

Assistant Professor in Management Science (Lecturer) & Director, Lancaster Research Centre for Forecasting

Lancaster University Management School, Room A53a, Lancaster, LA1 4YX  | T: +44 (0)1524 5-92991 | F: +44 (0)1524 844885 | M: +49 (0)171 4910100 | W: www.lums.lancs.ac.uk/forecasting | E: s.crone@lancaster.ac.uk

Post a Comment

Alfred Hitchcock and a classic forecasting scam

The Forecasting Savant

Suppose you received an email from a self-proclaimed forecasting savant, advising you of a big upset in the upcoming mayoral election...and it turns out to be correct.

You then get an email picking the underdog in the next championship boxing match...which is right again.

Over the course of a few weeks you receive four more such emails, predicting outcomes that turn out to be true.

After six correct picks in a row, this savant has proven his ability, wouldn't you think? And he certainly deserves a cut of earnings on the wagers you've started placing on these forecasts, as well as a healthy gratuity for the forthcoming 7th prediction, a stock pick which is promised to make you rich.

But has the savant established his skills as a forecaster, and does he deserve a big advance for his 7th prediction? Does it make sense to take a large position in the recommended stock?

What Would Alfred Hitchcock Do?

The above scenario (substituting mail for email) plays out in Season 3, Episode 2 of the 1950's television show, "Alfred Hitchcock Presents." Entitled "Mail Order Prophet," E.G. Marshall is the gullible (but so far money winning) recipient of the prophet's prophecies, while Jack Klugman is the skeptical office mate.

Klugman warns Marshall early on, "Don't be an idiot! Nobody can predict the future -- it's impossible." But after failing to act on the first two letters, Marshall is ahead over $1000 placing wagers on the last four, winning them all. He seeks out Klugman's advice on the latest letter, in which the supposed savant requests remuneration for his correct predictions, and promises his next tip is for a stock that will grow 10-fold.

After commenting on Marshall's natty new suit ("Seersucker no doubt? For every seer there's gotta be a sucker") Klugman continues his rant against forecasting, "Predicting the future is a scientific impossibility!" and warns that this is some kind of scam. Yet Marshall remains convinced of his savant's infallible prescience, and "borrows" $15,000 of company funds to buy $30,000 worth of Athabasca Mines on margin.

How You Can Become a Forecasting Savant

Have you figured it out (or seen this scam before)? It goes like this:

1) Send out thousands of emails projecting one outcome to half the recipients, the opposite outcome to the other half. (E.g., incumbent wins or challenger wins.)

2) To the half who received the correct prediction, send another email, again projecting one outcome to half the recipients, and the opposite outcome to the other half.

3) Repeat...each time you'll shrink your pool of "suckers" in half, but they'll be all the more convinced of your future-seeing powers by being correct every time.

4) Monetize the process, requesting a gratuity for all the correct predictions you've made so far, with the offer of an additional forecast that will make them huge money.

5) Pocket your gratuities and flee.

Jack Klugman's Critique of Forecasting

While I'm sympathetic to Klugman's critique ("Any intelligent man knows you can't predict the future"), it needs to be tempered just a bit. As forecasters we cannot expect to predict the future with 100% accuracy. But we can still deliver value to our organizations by assessing the uncertainty of the future, and providing guidance that can lead to improved decision making.

Paul Goodwin's recent Foresight/SAS Webinar provides many good ways of expressing forecast uncertainty, including ranges, prediction intervals, fan charts, and probability density charts. The recorded webinar "Getting Real About Uncertainty," and all previous webinars in the Foresight/SAS Webinar Series, are available for on-demand review.


Post a Comment

SAS forecasting and econometrics "tip of the week"

SAS Support Communities provide a forum in which to engage and share with your fellow SAS experts, now in over 20 topic areas including Forecasting and Econometrics. This is the go-to website for your hard core modeling questions, such as "Holdouts in PROC ARIMA," "2-stage Heckman (1979) procedure," and the perennially popular "Moving average forecast."

Ken Sanford

Ken Sanford explaining something complicated

My colleague Ken Sanford just announced a new "tip of the week" series for SAS forecasting and econometrics users on the Communities page. Ken provided a tip on controlling your output in the SAS Output Delivery System (ODS).

This week Bobby Gutierrez of the SAS/ETS development team provided a tip on Fixed vs. Random Effects in Panel Data.

If you get the idea that SAS software can help you build just about any kind of model you want, with data as big as you can get, you are correct.

But if you don't have the time, inclination, or skill set to create your own custom models, let the software do it for you. SAS Forecast Server provides large-scale automatic forecasting, and can combine with SAS Grid Manager to spread the workload and handle the biggest forecast challenges. (See how Wyndham Exchange & Rentals utilizes grid computing to generate 1.4 billion forecasts.)

For businesses operating on a more modest scale, SAS Forecasting for Desktop provides a cost-effective entry point to automatic forecasting.

Here is a 5-minute demonstration of the user interface shared by Forecast Server and Forecasting for Desktop.

If you are already a SAS forecasting or econometrics user, you are encouraged to join the community, post your questions, and share your expertise. And if you aren't already a SAS user, come see what you've been missing.

Post a Comment

Foresight/SAS webinar December 3: Getting real about uncertainty

Paul Goodwin

Paul Goodwin

Just a reminder that on Wednesday, December 3, 10:00am ET, Paul Goodwin will be delivering the next Foresight/SAS Webinar on "Getting Real About Uncertainty." From the description:

Many forecasts only tell us about things like probable events, expected sales and prospective demand. Often they tell us nothing about the level of uncertainty we face.

This webinar will discuss recent research suggesting that people make better decisions when forecasts contain information on uncertainty. It will demonstrate how to:

  • Estimate uncertainty.
  • Use forecasting uncertainty to your advantage.
  • Present forecasts in ways that are credible, understandable and useful.

Register for the live webinar, or check back later this week to view the on-demand recording.

For more discussion of Paul's topic, see this BFD post from last month.

Post a Comment

IBF blog series on Forecast Value Added

Calling All Forecasters

Have you tried Forecast Value Added analysis? What did you find out? Are you willing to share your learnings (at least those that can be revealed publicly)?Would you like to be featured in a new blog series on FVA, published by the Institute of Business Forecasting?

The IBF was an early advocate of FVA analysis, from including "FVA" in their Glossary of forecasting terms, to providing a platform for me and other industry forecasters to speak on the subject at IBF events. The blog series will include interviews with forecasting practitioners and provide case studies on the application of the FVA mindset at their organizations.

FVA Stairste Report

FVA Stairstep Report

In the forthcoming December interview, Jonathon Karelse (co-founder of NorthFind Partners) will discuss his use of FVA starting at Yokohama Tire Canada, and now with his consulting clientele.

How to Participate

Interested in sharing your story? Please email me (mike.gilliland@sas.com) to discuss.

For anyone who is publicity shy (due to company PR restrictions, outstanding arrest warrants, debt collectors, etc.), I'm happy to keep your identity (and your company) confidential.

More News from IBF: 2014 Recognition Award Recipients

While visiting the IBF blog site, don't miss announcement of IBF's 2014 Recognition Awards, to Mark Covas (Lifetime Achievement) and John Gallucci (Forecasting & Planning Excellence).

Mark is Group Director of the Planning Center of Excellence at Coca Cola, and a longtime friend of The BFD.* He was recognized for 20 years of contributions to the IBF as a speaker, journal contributor, and advisory board member. As a practitioner, he has led demand planning initiatives across the globe at Levi-Strauss, Johnson & Johnson, Starbucks, Gillette, Proctor & Gamble, and Georgia-Pacific.

John is Senior Director of Product Planning at Pinnacle foods, and was recognized for his use of lean techniques to improve forecasting process efficiency and effectiveness, and for managing ground up implementations of demand planning and S&OP processes. He is also a frequent speaker at IBF events, and contributor to the Journal of Business Forecasting.

Congratulations to this year's recipients!


*I have personally made tens of dollars in royalties from copies of The Business Forecasting Deal Mark purchased for his staff.



Post a Comment

Lancaster Centre for Forecasting survey

LancasterCan you spare a few minutes to assist researchers at the Lancaster Centre for Forecasting?

We have been commissioned by the European Journal of Operational Research to write a review article on Supply Chain Forecasting. In undertaking this task, we would like to ensure that the topics covered reflect the priorities of the forecasting community as a whole. To that end, we have prepared an (anonymous) questionnaire that aims to identify the most important areas that should be addressed in our paper, both from academic and practitioner perspectives.

To participate in this study, visit the Supply Chain Questionnaire. It will take no more than 5-10 minutes of your time. Please indicate if you are an academic or a practitioner in the last section of the questionnaire. We are working to very tight deadlines, so your timely help and participation is appreciated. Should you have any questions or comments on this work please contact Aris Syntetos. Thank you for your time and co-operation,

A.A. Syntetos, M.Z. Babai, J.E. Boylan, S. Kolassa & K. Nikolopoulos

You will recognize lead author Aris Syntetos, who delivered the October Foresight/SAS Webinar on "Forecasting by Temporal Aggregation." (The recorded webinar is now available for on-demand review.) And Stefan Kolassa co-authored a clever Foresight article "Percentage Errors Can Ruin Your Day" which was discussed in the BFD post Tumbling Dice in 2011.

Post a Comment

Gaming the forecast

Business forecasting is a highly politicized process, subject to the biases and personal agendas of all forecasting process participants. This is why many -- perhaps most -- human adjustments to the forecast fail to make it better. And this is why relative metrics, such as FVA, are so helpful in properly evaluating process performance.

The Games People Play

John Mello

John Mello

In his article "The Impact of Sales Forecast Game Playing on Supply Chains" (Foresight, Spring 2009), John Mello provides a useful taxonomy of seven such games that bias the forecasting process and degrade forecast accuracy.

Enforcing: Maintaining a higher forecast than actually anticipated sales, in order to keep forecasts in line with the organization's sales or financial goals.

When the voice of the marketplace is screaming that you aren't going to hit your sales targets, this should be considered a blessing! Perhaps now, knowing about this in advance, you can do something to address the demand shortfall. Unfortunately, many organizations ignore the screaming, and simply adjust their forecast to meet the objective. It is easy to understand why this happens, because as a practical matter, as long as your forecast still meets your target, your boss won't scream at you!

Filtering: Changing forecasts to reflect the amount of product actually available for sale in a given period.

When you are capacity constrained or otherwise experiencing demand greater than supply, a great way to show 100% forecast accuracy is to forecast sales equal to the expected supply.

Hedging: Overestimating sales in order to secure additional product or production capacity.

No sales person wants to explain unfilled orders to his or her customers. How to avoid this? Simply forecast a lot more than you really expect to need. (Just make sure your heightened forecasts go only to the supply planning folks, not to your own sales management. See next.)

Sandbagging: Underestimating sales in order to set expectations lower than actually anticipated demand.

For your sales management, send in low forecasts. That'll help keep your quotas within reach, and win you that award trip to Hawaii.

Second Guessing: Changing forecasts based on instinct or intuition about future sales.

Sometimes there is no data (new products), or the data may be otherwise inappropriate or insufficient. Sometimes you have to make adjustments based on experience and knowledge of the business. Just make sure to track the performance of such adjustments to make sure they are actually improving accuracy and reducing bias.

Spinning: Manipulating forecasts to obtain the most favorable reaction from individuals or departments in the organization.

Control the data, and control the means of measurement, and you can control the message.

Withholding: Refusing to share current sales information with other members of the organization.

Who wants to get yelled at twice? Why tell your boss you aren't going to hit this quarter's numbers (and get yelled at now), when you can just keep forecasting the hockey stick and only get yelled at quarter end after the sales don't materialize.

How many games are going on at your organization?

Post a Comment

Getting real about uncertainty

Paul Goodwin

Paul Goodwin

In his Spring 2014 article in Foresight, Paul Goodwin addressed the important issue of point vs. probabilistic forecasts.

A point forecast is a single number (e.g., the forecast for item XYZ in December is 635 units). We are all familiar with point forecasts, as these are what's commonly produced (either by the software, or judgment) in our forecasting processes.

The problem is that point forecasts provide no indication of the uncertainty in the number, and uncertainty is an important consideration in business planning. Knowing that the forecasting is 635 +/- 50 units may lead to dramatically different decisions than if the forecast were 635 +/- 500 units.

There are a number of ways to provide a probabilistic view of the forecast. With prediction intervals, the forecast is presented as a range of values with an associated probability. For example, "the 90% prediction interval for item XYZ in December is 635 +/- 500 units." This would indicate much less certainty in the forecast than if the 90% prediction interval were 635 +/- 50 units.

A fan chart provides a visual expansion of a prediction interval over multiple time periods. In Goodwin's example, the darkest band represents the 50% prediction interval, while the wider ranges show the 75% and 95% intervals.

Fan Chart

Fan Chart

A probability density chart can provide even more granular detail on the forecast for a single time period. In this example, the most probably sales are around 500 units, but will almost certainly be between 200 and 1200.

Density Forecast

Density Forecast

Goodwin reviews recent literature on the value of communicating uncertainty, and suggests that it can lead to improved decisions. However, research also showed that if interval forecasts are too wide, they were judged to be uninformative and less credible. So even if 635 +/- 500 is the appropriate width of the 90% prediction interval, decision makers may simply ignore the forecast and doubt the competence of the forecaster who produced it!

Estimating the level of uncertainty may be non-trivial, particularly when forecasts are based on human judgment. Research has repeatedly shown that people produce intervals that are far too narrow. Goodwin cites a 2013 study of financial executives providing 80% prediction intervals for one-year-ahead stock prices. Actual returns fell within the 80% intervals only 36% of the time!

A simple way to address inappropriately narrow intervals was suggested by Makridakis, Hogarth, & Gaba in one of my favorite forecasting-related books, Dance With Chance. They suggest taking your estimated prediction interval and doubling it. (I love a quick and dirty solution to a complex problem.)

Learn more in the next Foresight / SAS Webinar

On December 3, 10:00am ET, Paul Goodwin will present his finding in the next installment of the Foresight / SAS Webinar Series.

This webinar will discuss recent research suggesting that people make better decisions when forecasts contain information on uncertainty. It will demonstrate how to:

  • Estimate uncertainty.
  • Use forecasting uncertainty to your advantage.
  • Present forecasts in ways that are credible, understandable and useful.

Register for "Getting Real About Uncertainty in Forecasting" and watch the 30-second video preview.


Post a Comment
  • About this blog

    Michael Gilliland is a longtime business forecasting practitioner and currently Product Marketing Manager for SAS Forecasting. He initiated The Business Forecasting Deal to help expose the seamy underbelly of the forecasting practice, and to provide practical solutions to its most vexing problems.
  • Subscribe to this blog

    Enter your email address:

    Other subscription options

  • Archives