The Business Forecasting Deal - Exposing bad practices in business forecasting... and providing practical solutions

I’m going to put “An Operational Definition of ‘Demand’ – Part 3” on hold for a moment, to announce a new favorite article on forecasting, “Living in a world of low levels of predictability,” by Spyros Makridakis and Nassim Taleb (International Journal of Forecasting 25 (2009) 840-844. IJF is a publication of the International Institute of Forecasters, and if not already a subscriber you can purchase the article from ScienceDirect.)

Many of you already know Makridakis as co-author of the standard forecasting text Forecasting: Methods and Applications, and Taleb for his Fooled by Randomness and The Black Swan. Taleb, in particular, has drawn attention to the issue of the un-forecastability of complex systems, and the sometimes disastrous consequences of our “illusion of control, pretending that accurate forecasting was possible” (p.841).

While referring to the (mostly unforeseen) global financial collapse of 2008 as a “prime example of the serious limits of predictability” (p.240), this brief and non-technical article summarizes the empirical findings for why accurate forecasting is often not possible, and provides several practical approaches for dealing with this uncertainty.

So why am I, a vendor of forecasting software, so excited by an article telling us the world is largely unforecastable? Because Makridakis and Taleb are correct – we should not have high expectations for forecast accuracy, and should not expend heroic efforts trying to achieve unrealistic levels of accuracy. Instead, by accepting the reality that forecast accuracy is ultimately limited by the nature of what we are trying to forecast, we can instead focus on the efficiency of our forecasting processes, and seek alternative (non-forecasting) solutions to the business problem. Methods I have touted, such as Forecast Value Added analysis, can be used to identify and eliminate forecasting process activities that do not improve the forecast (or may even make it worse). Large-scale automated software, such as SAS Forecast Server, can deliver forecasts about as accurate and unbiased as anyone can reasonably be expected – and do this without elaborate processes or significant management intervention. For business forecasting, the objective should be:

To generate forecasts as accurate and unbiased as can reasonably be expected – and to do this as efficiently as possible.

The goal is not perfect forecasts – that is wildly impossible. The goal is to try to get your forecast in the ballpark, so you can plan and manage your business effectively, and not waste a lot of company resources doing it. And when, because of the nature of demand or other behavior, you cannot forecast with the degree of accuracy needed for effective planning, then seek alternative approaches to address the underlying business problem. In the past I’ve suggested things like demand smoothing (to make the demand forecastable), or supply chain re-engineering (to minimize your reliance on accurate forecasts). You can find more discussion of these in a 2001 article I co-authored with Drew Prince of NCR, “New Approaches to Unforecastable Demand” (Journal of Business Forecasting, Summer 2001, pp. 9-12), available for download from the Institute of Business Forecasting.

In the last post I argued that we don’t have a sure way to measure true (i.e. “unconstrained”) demand. While demand is commonly defined as “what the customer wants, and when they want it,” it is actually a nebulous concept. For a manufacturer, what a customer orders is not the same as true demand (for various reasons described in the prior blog post), nor is what actually ships. At a retailer, what is actually sold off the shelves is not the same as true demand, either. For example, the customer may not be able to find what they want in the store (due to out-of-stocks, or poor merchandise presentation), so there is true demand but no recorded sale. Determining true demand for a service can be equally vexing. I may have a taste for a Royale with Cheese at McDonald’s, but go to Wendy’s instead if the drive-thru line is too long. Or I may call the cable company to complain about my tv reception, only to hang-up in frustration trying to wade through their voice menu system.

While we may know true demand under certain special circumstances, we don’t have a general operational definition that will work in all situations. This means two things:

• We will be unable to construct a history of true demand to feed into our statistical forecasting models (except under certain special circumstances)
• We will be unable to assess the accuracy of our forecast of true demand (i.e. our “unconstrained forecast”) (except under certain special circumstances)

Conceptually these are important points. In most circumstance we don’t know what true demand really is, so we don’t know for sure how accurately we are forecasting it. I argued, therefore, that forecasting performance should be evaluated against the “constrained” forecast. The constrained forecast represents our best guess at what is “really going to happen” -- i.e. actual shipments or sales or services provided. We can measure what actually happens.

As a practical matter, maybe all of this isn’t so important. While we can’t know exactly what true demand really is under most circumstances, we can often get close enough to make the concept useful in forecasting. It is true that demand ≠ orders, yet if an organization does a good job at filling orders (say 98%+), then “orders” and “true demand” are virtually the same (within a few percentage points). When we forecast, our errors are often 25%, 50%, or more. The fact that the demand history upon which we build our forecasts is not perfect (but may be off by a few percentage points from true demand) is inconsequential compared to the magnitude of the forecast error. (Even if we were able to capture perfect history of true demand, it might only make our forecasts a few percentage points better at best – and that isn’t going to rock anyone’s world.)

There is probably more to say on this topic, so expect a Part 3.

Sorry about not getting a post out last week, but I spent a good part of it cowering under my desk in fear. The SAS Security office issued a warning that there were wild coyotes roaming the campus, and I was having post-traumatic flashbacks to a painful encounter I once had with a cougar during my late teens. While coyotes can be wily and ill-tempered, I now realize they aren’t nearly as aggressive as cougars, so there was little to fear. (Fortunately I don’t have to fear cougars anymore either, because at my age they no longer consider me an appetizing target.)

That brings us to today’s topic, creating an operational definition of “demand.” Those of us in forecasting use the word demand every day, and we don’t think much about it – it seems pretty straightforward. Demand is commonly characterized as “what the customers want, and when they want it,” sometimes with the added proviso “at a price they are willing to pay, along with any other products they want at that time.” Sounds good to me.

When we refer to demand, we really mean “unconstrained” or “true” demand, because we take no consideration of our ability to fulfill it. We use the phrase “constrained demand” to describe what’s left after incorporating any limitations on our ability to provide the product or service demanded. Thus, constrained demand ≤ demand. So far so good -- do I really need to devote a blog entry to something so self-evident?

This definition of “demand” is not problematic until we try to operationalize it, that is, when we start to describe the specific, systematic way to measure it. This is kind of important if we ever expect to measure the accuracy of our “demand forecast.” We need to know what actual demand really was!

Let’s work through an example I originally wrote about in the Summer 2003 issue of Journal of Business Forecasting, dealing with this issue at a manufacturer.

If customers place orders to express their “demand,” and if the company services its customers perfectly by filling all orders in full and on time, then we have our operational definition. In this case, demand = orders = shipments. If both order and shipment data are readily available in the company’s system, then we have the historical demand data, which we can use it to feed our statistical forecasting models and measure our forecasting performance.

Unfortunately, few organizations service their customers perfectly. As such, orders are not a perfect reflection of true demand. This is because when customer service is less than perfect, orders are subject to all kinds of gamesmanship. Here are a few examples:

1. An unfilled order may be rolled ahead (carried over) to a future time bucket.
2. If shortages are anticipated, customers may artificially inflate their orders to capture a larger share of an allocation.
3. If shortages are anticipated, customers may withhold orders, or direct their demand to alternative products or suppliers.

In the first example, demand (the order) appears in a time bucket later than when it was really wanted by the customer. Rolling unfilled orders causes demand to be overstated -- the orders appear in the original time bucket, and again in future buckets until the demand is filled or the order is cancelled.

In the second example, the savvy customer (or sales rep) has advanced knowledge that product is scarce and will be allocated. If the allocation is based on some criterion such as “fill all orders at X%,” the customer simply over-orders and ultimately may receive what it really wanted in the first place.

The third example not only contaminates the use of orders to reflect true demand, but it can also cause significant financial harm to your business. If you are in a situation of chronic supply shortages (due to either supply problems or much higher than anticipated demand), customers may simply go elsewhere. Customers may truly want your product (so there is real demand), but it won’t be reflected in your historical data because no orders were placed. While orders are often perceived as “equal to or greater than” true demand, this third example shows that what is ordered may also be less than true demand.

As with orders, the use of shipments to represent demand has a number of potential problems. Shipments are often perceived as “equal to or less than” true demand. Thus, shipments and orders are thought to represent true demand’s lower and upper bounds. But, as we see in example 3, orders can be lower than the true demand. Furthermore, by example 1, shipments can actually be greater than true demand. (This would occur when an unfilled order is rolled ahead into a future time bucket and then filled. In this situation the shipment occurs later than the true demand, and inflates demand in the time bucket in which it is finally shipped.)

The planning process starts with a compilation of unconstrained or “true” demand history. We feed that into our statistical software and generate an “unconstrained forecast.” In order to measure our forecasting performance, we need to know the actual demand that really occurred. Herein lies the problem. There may be no way of knowing what true demand really was. The manufacturer knew what orders it received, and knew what shipments it made. But neither orders, nor shipments, nor any other readily available data element is the same as true demand. Therefore, we cannot measure the accuracy of our unconstrained forecast.

The lesson is to always report forecasting performance against the constrained forecast. The constrained forecast represents what the manufacturer actually expects to ship, what the retailer actually expects to sell, and what the service organization actually expects to provide. The constrained forecast is our best guess at what is really going to happen, and we can measure what truly does happen. While we can measure the shipments, the sales, or the services provided, we cannot with certainty measure what was truly demanded.

Last week I was a guest of Gaurav Verma on the SAS Applying Business Analytics Web Series, and presented “What Management Must Know About Forecasting.” One of the most important things you can bring to management’s attention is the benefit of making your demand forecastable.

In forecasting we tend to treat demand patterns as given, as if there were nothing we could do about them. This “passive” attitude makes our performance contingent on the forecastability of the demand patterns. If the patterns are easy to forecast then we should do fine, but if they are not, we risk failing to meet our forecast accuracy objectives.

In my July 1 posting on The BFD, I used this scatterplot to illustrate the usual relationship between demand volatility (along the horizontal axis), and forecast accuracy (the vertical axis). I argued that while imperfect, the volatility of a demand pattern (as measured by its coefficient of variation) is a good indicator of how accurately we can expect to forecast that demand. As a practical first step to improving your forecasting performance, it is worthwhile to construct this scatterplot with your own data…and here’s how:



















1) Determine at what level you want to do the analysis. This is typically the level at which you focus your forecasting and planning efforts. For example, a manufacturer might forecast the number of units demanded for each Item, or at each Item / Distribution Center combination. A retailer might forecast unit sales by Item / Store. An insurance company might forecast claims by Region or Postal Code. (In the above scatterplot, this manufacturer sold roughly 500 Items through 10 DCs, and forecasted at the Item / DC level. There are, therefore, roughly 5000 (= 500 x 10) points in the plot.) Suppose for this example that we are doing the analysis by Item.

2) You will need to gather the last year of data for each Item, in whatever time bucket you use (typically week or month). (I prefer using weekly data if it is available, but many organizations plan in monthly buckets, so we’ll use monthly for this example.) For each item you will need to capture the monthly Sales, and whatever the Forecast was for that month. So you will create a dataset with four variables . The four columns on the left show what this dataset might look like with the 12 monthly observations for Item XXX.



3) You next compute the volatility of sales over the past year. Volatility is measured by the coefficient of variation (CV) of the monthly sales, which is defined as the standard deviation divided by the mean. For item XXX over the past year, it had mean monthly sales of 102.5 units, and a standard deviation of 26.8. Therefore, CV = 26.8 / 102.5 = 26.2%.

4) You finally have to compute Forecast Accuracy for each Item. This requires two new computations shown in the two rightmost columns. The Absolute Error in forecasted sales each month (i.e. |Forecast – Sales|) and the Maximum of Forecast or Sales in each month. Forecast Accuracy is computed as 1 – (Sum of Absolute Errors) / (Sum of Maximums) = 1 – (150/1320) = 88.6%.

5) Create the scatterplot by plotting the Forecast Accuracy and Volatility coordinates for each Item. The Forecast Accuracy metric will always be from 0 to 100%, so the vertical axis can be scaled 0 – 100. The Volatility, however, can become indefinitely large. For clarity and usefulness of the presentation, you may want to ignore really extreme values of CV (unless there are a lot of them), and only run the horizontal axis from 0 to 150% or 200%, and footnote how many Items are not being shown. The point is to illustrate the overall volatility of your sales patterns, and to show the relationship with your ability to forecast.

As long as you have the past year of sales data, along with the last year of forecasts, it is very easy to create this scatterplot. It is a simple way to visualize the extent of volatility issues with your sales, and the likely impact on your forecasting performance.

There are some things about forecasting that you may only discover by being a forecaster. Simply managing a forecasting process, or being a downstream consumer of the forecast, isn't always enough. If you have something to say to your management about forecasting, but would rather avoid the confrontation, maybe we can help.

Please bring along your management to join me and my SAS colleague, Business Analytics Marketing Manager Gaurav Verma, for Part 8 of the Applying Business Analytics Webinar Series (Wednesday October 21, 1:00 p.m. ET): What Management Must Know About Forecasting. We'll be covering some of the fundamental issues and worst practices that may not be so apparent to those managers and executives not directly involved in the forecasting process.

Click here for more information and free registration.

For those unable to attend live, this webinar will be recorded and made available for free on-demand viewing.

Before we begin, I want to thank all of you readers who supported my righteous appeal for “THE BFD” license plate from the state of North Carolina. I am sad to report, however, that our efforts were in vain and the appeal was denied by Censorship Board of the Division of Motor Vehicles. Along with the denial, these arbiters of good taste were kind enough to send me an application to re-submit another idea – any suggestions? (Note: I’ve already given one new suggestion to the DMV, but they responded back that it was anatomically impossible.)

Now back to the business of forecasting…

Just how confident can we be about the future? This is an important consideration that plays into the decisions we make based on our forecasts. Makridakis, et al, discuss this in Forecasting Methods and Applications:

“It is usually desirable to provide not only forecast values but accompanying uncertainty statements, usually in the form of prediction intervals. This is useful because it provides the user of the forecasts with “worst” or “best” case estimates and with a sense of how dependable the forecast is, and because it protects the forecaster from the criticism that the forecasts are “wrong.” Forecasts cannot be expected to be perfect and intervals emphasize this.” (p. 52)

Not all forecasting software provides prediction intervals, but SAS High-Performance Forecasting (the “engine” inside SAS Forecast Server software) does. The width of the prediction interval (sometimes referred to as the confidence limits or confidence bounds) indicates how close the future is likely to be to the forecasted value. (Confidence bounds are shown by the shaded blue area on the far right of this plot.)

When the confidence bounds are wide, forecasters sometimes whine and complain that they are useless – what good is a forecast in which we can have no confidence? But what is there to do in this situation? My colleague Bob Lucas provides some thoughts on this matter.

Guest Blogger: Bob Lucas, SAS Education & Training

I have taught forecasting for 12 years here at SAS and many a student is often disappointed in the widths of the forecast confidence bounds. They have said, “What good are the confidence bounds that are so wide?” Here is my typical response:

What affects the widths of the confidence bounds?

1. The amount of data you have. (You may or may not have control over that.)
2. The model that you are using. (You do have control over that, however, with limited data you may not have too many choices.)
3. The natural variability in the data. (You have no control over that!!!)

If your data has a lot of variability, guess what – you’re going to get wide confidence bounds. Some customers have unrealistic expectations for the accuracy of their forecast.

Let’s examine the one aspect that you do have control over – the model that you select to produce the forecast. One of the standard premises of forecasting (actually prediction in general) is simpler models do better – all else being the same. While a more complex model (with more parameters) will be able to “fit the history” better than a simpler model, it will likely do no better at forecasting the future.

If you do not use a holdout sample, fits statistics that ignore the number of parameters will favor models with more parameters. Consequently, I favor using SBC to choose my forecast models when not using holdout data. (The Schwarz Bayesian Criterion, also called the Bayesian Information Criterion (BIC), is a criterion for selecting a model from among a class of models with different numbers of parameters.) I still report MAPE to business user, however.

One way to restrict forecast and confidence bounds is to fit the log of the series. SAS Forecast Server will transform back to the original scale for you. This disadvantage is that now the parameter estimates are on the log scale and complicate interpretation if desired.

Now my answer to the question “What use are the confidence bounds that are so wide?”

Don’t you think you would make a different business decision of your forecast was 100 +/- 10 versus 100 +/- 50? How does this relate to your situation? The inventory cost for a fixed service level will be much higher for a series with high volatility than for one with low volatility. The realistic way to handle this is to have different service levels for different products based on the volatility of the series.

As Bob is pointing out, when you have highly volatile demand for a product, you probably aren’t going to be able to forecast it as accurately, and will have wider confidence bounds around the forecast. As a result, you are going to need correspondingly (and perhaps prohibitively expensive) amounts of inventory available to meet a fixed service level target. What an organization needs to do, rather than set a fixed service level (such as “98% order fill rate”) across all products, is to set lower service level objectives for the more difficult to forecast products. If a certain service level (like 98%) is absolutely necessary, such as due to contractual obligations with the customer, you need to address the problem in an alternative manner, such as by finding ways to reduce the volatility (and “unforecastability”) of the demand patterns.

[For further discussion on ways to reduce the variability of demand, tune in to my upcoming webinar “What Management Must Know About Forecasting” (October 21, 1:00pm ET). This is part of the SAS Applying Business Analytics Webinar Series hosted by my colleague Gaurav Verma.]

In summary, if your software does not produce confidence bounds along with its forecasts, you are missing out on some very useful information. There is uncertainty in any forecast, but having an understanding of this uncertainty and an appreciation of the risk helps your organization avoid the really bad business decisions.

In case you haven't heard, the state of North Carolina needs money. Roads are falling apart, parks are left unkempt, prisoners are being released, and public school cafeterias can no longer afford to put anything surprising in their Chef's Surprise.

In order to help assuage the crisis, and in my most altruistic act since filling a rental car with Premium*, I gave the NC Division of Motor Vehicles an extra $30 to get myself a personalized plate (shown here from the actual DMV website):

The way I had it figured, what better way could there be to promote this blog and spread the word about all that is wrong in forecasting. The DMV website took my application, my credit card was charged, and all I had to do was wait for the friendly gentlemen in the NC state prison system to stamp out my plate and put it in the mail.

Apparently I figured wrong. Either they have released all the prisoners who know how to run the license plate machines, or the folks at the DMV are spending way too much time reading urbandictionary.com. Instead of getting my new plate, I received this:

DEAR MR. GILLILAND:
PERSONALIZED LICENSE PLATE THE BFD WAS ORDERED FOR YOU IN ERROR. THIS PLATE MAY BE CONSIDERED IN POOR TASTE AND THEREFORE CANNOT BE ISSUED.

WTF? (Which, by the way, was issued on 9,999 license plates across the state in 2008.) I don't know what they could have possibly found in poor taste about THE BFD. I immediately appealed, even providing a link to this blog, The Business Forecasting Deal, to show them what an important public service is being provided.

Anyway, if you'd like to help the just cause of THE BFD, please write in support of my appeal to:

Kay Hatcher
North Carolina Division of Motor Vehicles - Special License Unit
3155 Mail Service Center Division
Raleigh, NC 27966-3155

or email to: khatcher@ncdot.gov

Would you buy something that doesn’t have demonstrable Return on Investment? Of course you would! Whether you realize it or not, you do this every time you buy software. There is no proven ROI.

Maybe this is just a pet peeve of mine. Maybe I care too much about cause and effect. Maybe I care too much about logic and scientific method. Maybe I care too much about what it means to prove something. Or maybe I’m just another egomaniacal jerk with a blog that thinks he is entitled to raise a stir about nothing. I am, therefore I will.

Every software vendor touts the ROI you will achieve by buying their products. 10%, 100%, 1000% ROI – the claims are everywhere, and part of every sales cycle. And it isn’t just the vendors claiming ROI. It is also de rigueur for the customer’s internal project sponsor or champion to make claims of anticipated ROI -- in order to secure the funding to buy the software.

ROI is a sham??? Wow!

Think about it – how would you ever prove that implementing software X caused effect Y (where Y is increased revenue, reduced costs, higher profits, etc.). Just because one thing happens along with something else, does not prove there is any relationship between the two whatsoever – let alone a “causal” relationship. The problem is that things change, they vary over time. (Perhaps this is why they are called variables?) The interrelationships among variables can be subtle and mysterious, and we don’t really know for sure what mechanism (if any) is driving things.

We often (and wrongly) attribute causality in these sorts of situations. The geniuses in the financial news media, for example, can provide brilliant explanations for the cause of every market fluctuation. Nassim Nicholas Taleb has a great example of this in his book The Black Swan:

"One day in December 2003, when Saddam Hussein was captured, Bloomberg News flashed the following headline at 13:01: U.S. TREASURIES RISE; HUSSEIN CAPTURE MAY NOT CURB TERRORISM." Bond prices then fell. "At 13:31 they issued the next bulletin: U.S. TREASURIES FALL; HUSSEIN CAPTURE BOOSTS ALLURE OF RISKY ASSETS."

So, per Bloomberg, the capture of Saddam caused bond prices to both rise and fall? (I guess if Saddam were not captured, bond prices would both fall and rise?) Foolish me, I thought only Jim Cramer could come up with gold like this!

In some situations purported causality has been reasonably well established through controlled experimentation. For example, I accept that downing 400mg of ibuprofen will cause my headache to go away. But claims of causality from forecasting software are not subject to such rigorous oversight. There is not yet a Federal Forecasting Administration to vet our ROI claims.

When the economy is good and profits are rising, a positive ROI can be attributed to just about anything an organization does. (True Story: Within weeks after interviewing and accepting the position of forecasting manager at Iomega in 1996, its stock price rose from $28 to $110. Wall Street apparently heard I was coming!) Any vendor is sure to jump on this bandwagon, and attribute profit gains to implementation of its software. This makes a good story, but fails to consider what else has changed. Perhaps your company introduced a hot new product that generated tremendous revenues and profits. Or perhaps your customers were just spending more freely in the booming economy and you would have increased revenue and profits without the new software. There needs to be much better evidence to demonstrate the cause and effect.

If a vendor uses this argument in good faith, and accepts the “logic” that its software caused these profit gains at its customers, then what happens next? What about its customers that implemented the software in late 2007 or the first half of 2008? Is the vendor willing to apply the same “logic” and admit that its software “caused” these customers to have falling revenues, huge losses, and perhaps even go out of business in the greatest financial collapse since the Great Depression? Who would ever admit that?

This is my gripe – we are fast to apply causality when the “effect” is good, but can always come up with a creative explanation when the “effect” is bad. Yet without a rigorous (and probably impractical) proof of cause and effect, we are just talking a bunch of nonsense.

Don’t talk to me about ROI.

Another time I'll discuss more appropriate ways to evaluate the value of forecasting software.

I’m back from a week of vacation in Michigan and bursting with new topics, but will have to get to those next week (still playing catch-up in my day job). I do, however, have the pleasure to announce the publication of a new book by my friend and colleague Charlie Chase, Demand-Driven Forecasting: A Structured Approach to Forecasting. Charlie is currently Business Enablement Manager for the SAS Manufacturing and Supply Chain Global Practice, and has been a longtime contributor to the forecasting profession through his publications and involvement with the Institute of Business Forecasting. Charlie also preceded me as marketing manager for SAS forecasting software, and led the launch of our flagship SAS Forecast Server in 2005.

In his column in the Summer 2009 issue of Journal of Business Forecasting, Larry Lapide had this to say about Charlie and the book:

“I believe he captures a lot of what industry forecasters (like yourself) have been doing over time to improve forecasting by understanding the impact of demand-shaping activities. Much as a practitioner, Bob Brown described history-based forecasting over 50 years ago, Charlie, another industry practitioner, has done the same for today’s demand-driven forecasting.”

This book is part of the Wiley & SAS Business Series, and you can order it now.

The SAS internal discussion boards are always full of fascinating topics, some of which are even decipherable to a non-Ph.D. in statistics like me. A recent topic involved how to calculate the benefits of good forecasting software, and my colleague Robin Way offered an interesting perspective that he allowed me to share:

Guest Blogger: Robin Way, Analytics Consulting Manager at SAS

Perhaps the primary benefit of good forecasting software is reduced uncertainty around the most likely forecast, as well as potential improvement in accuracy. Most of the time, I have found that lay business people who are not that familiar with forecasting and the business payoff of better forecasts immediately think that a better forecasting engine will lead to improved accuracy. However, most seasoned forecasters accept that nobody can really tell the future and every forecast is inherently wrong. The point of better forecasting processes is that you now have an enhanced ability to know how far wrong from the point forecast you are likely to end up, and what factors influence that degree of wrong-ness.

I know that all sounds pretty soft; so consider this business case made for a call center capacity planning group. The main driver of business performance in managing capacity for a call center is the cost of call handling to deliver a specified level of service to customers. (Note that call handling costs include personnel costs (salaries and benefits), scheduling, availability of current resources, likely attrition of existing staff, cost and timeframe to bring on new trained staff, and other factors.) Put too few call center agents on the phones, and your service level suffers; put too many agents on the phones, and cost per call suffers.

Improved accuracy around demand for inbound calls will certainly help the capacity planners target the most appropriate number of agents to schedule for any period of time, but everybody knows there’s no way to know for sure exactly how many calls will come in. This is the element of forecast error. You don’t want to staff for the most likely number—really you want to staff for the upper range of likely call volume, because it’s typically worse to deliver a low level of service than to have too many low-utilization agents on the phones. (You can always divert the slack resource to outbound calls or post-call cleanup for busy agents.)

So how to estimate this upper range? That’s where having a really good forecasting engine comes in. Tools like SAS Forecast Server help you reduce the confidence limits around the most likely forecast by reducing unexplained variance—that’s what you get by having a better model, via a better engine. You can identify what drives the uncertainty so that you can advise management whether the sources of uncertainty in the forecast are under their control (marketing promotions sponsored by the firm, price changes, new product introductions, contractual changes with existing customers), or outside their control (market and competitor forces, seasonal shifts, market events). And you can develop a more robust upper bound on the likely forecast in order to intelligently set your service levels for upcoming call handling periods.

If you then compare the upper bound on the forecast from a relatively naïve forecast model with one produced by a superior model, presumably generated with Forecast Server, you can show the difference in the upper bound (in this case, in call demand) as a reduced operational risk to the firm. All that on top of what is possibly a more accurate forecast to begin with, though I advise against making your case solely on the point of accuracy, because often times the process generating the actual results can vary relative to the forecast for reasons outside the control of the forecaster.


Robin delivers an important reminder. Your forecast should be more than just a point estimate representing your best guess at what is really going to happen. To drive better decisions and more profitable operations, you need to take into consideration the likely error in your point forecast. Forecasting for new products gives a prime illustration.

New products, especially those requiring new technology or manufacturing processes that require a lot of capital investment, can be a big gamble. If we fail to take into consideration the uncertainty (risk) in our point forecasts, we may take unnecessarily dangerous gambles. I’m not arguing against taking gambles with your business investments – every decision and action involves some risk. However, it is important to recognize the scale of the risk, and act accordingly.

Personally, I don’t get Twitter. I have an account (mvgilliland) for anyone interested in not hearing any tweets from me. I follow a few people and have a few followers (including some that aren't porn bots) -- but what is the point? Does anyone really care that I’m out hanging floss on the line to dry, or that I’m stuck in the waiting room of my urologist with a prostate the size of a grapefruit?

The fact is, if someone is that interested in what I’m doing right now, it makes me kind of nervous. Do I really want people “following” me? Aren’t anti-stalking laws enacted for good reason?

Call me old school, a luddite, a 21st century puritan, even a techno-prude. Or perhaps I’m just blind to the great new opportunities for data (not just dating) that social networking provides. I have actually been swayed a little bit in this direction by my colleague in Australia, Evan Stubbs. Leveraging some code from SAS software developer Zach Marshall, Evan put together a neat little demo for customers, illustrating how SAS Text Miner can be used to identify patterns that could be used in forecasting. From Evan:

Guest Blogger: Evan Stubbs, Solution Manager for SAS Analytics

Rightly or wrongly, we seem to love telling the world what we’re doing, often even if no-one’s listening! Morgan Stanley recently published a report by a 15 year-old intern that for many, seemed to state the obvious: “On the other hand, teenagers do not use twitter … they realise that no one is viewing their profile, so their ‘tweets’ are pointless”.

Ignoring the bizarre implications that ‘only oldies use Twitter’, to me, this misses the point; it’s not about who’s listening right now, it’s about who might be listening. One of the points of talking publicly about a particular topic is the hope that other people who are also interested in that topic might just join in. For me, it’s about the chance of finding like-minded people with similar interests (whether they agree with me or not!). It’s about connecting with new people, people I may never have met otherwise. It doesn’t matter whether it’s about my passion for analytics, my fascination with my latest gadget, or my displeasure with my latest billing experience; with the growth of the Internet, there’s bound to be people out there thinking and debating about similar things.

And, that’s the clincher - the scale of these social networks can’t be underestimated. A back-of-the-napkin poll I recently did to see how big some of the sites I knew about were stunned me; out of approximately 20 sites myself or my colleagues are a part of, only two had membership levels below 22 million. That may seem like an arbitrary number, but it has quite a bit of significance for me – it’s the population of Australia, the place I live. Sites like Facebook and MySpace have over ten times the population of Australia; These aren’t just social networks, they’re almost countries in their own right!

With that level of membership, it’s not surprising that there’s a wealth of information available within them, information that changes as rapidly as the discussion does. Google Trends and Twitter Trending Topics are great to help see what people are talking about overall, but they’re not personal – they don’t always relate to what I’m interested in. And, trying to trim down the torrent of information is almost an exercise in frustration – applications like TweetDeck help targeted searching and monitoring, but they don’t solve the real problem around pattern extraction and trend analysis.

So, based on the excellent work done by Zach Marshall, one of the geniuses behind our Web Services development, I thought it’d be rather fun to use SAS to create a personalized Twitter search process that takes into account geographic information, language-based searches, and then use Enterprise Guide’s Stored Process capabilities to package it up into an installable process usable by anyone. For me, the exciting thing was how much of SAS’s functionality I was easily able to use in what amounts to such a small effort:

• SAS 9.2’s Web Services capabilities, to connect to Twitter and create the query
• SAS’s Regular Expressions parsing, to cleanse the XML documents and structure them correctly
• SAS’s XML parsing and data handling capabilities, to extract and structure data
• SAS’s Stored Process capabilities, to turn it into a reusable process that’ll deliver the results to anything (a SAS dataset, Excel, Internet Explorer …)
• SAS’s Text Mining capabilities, to extract trends and patterns of particular discussion

I spend a lot of time on planes, so one of the first things I searched for was what people were saying about some of our major airlines over the last seven days, centered approximately 100 kilometers around Sydney (where I live). The breakup was fascinating – for one, the discussion was focused around:

• 4% of all discussions: TV related discussion, namely the Australian anti-censorship video being screened on the airline and various television awards programs
• 41% of all discussions: Discussion about the airline’s lounge, posts of people in-transit and waiting for the flights / going home
• 22% of all discussions: Frequent flier points, the airline’s club, a new joint loyalty reward program
• 23% of all discussions: Work-related discussion and industry issues (e.g. A380, working at the airline)
• 8% of all discussions: Cargo price fixing

The level of interest around their newly launched joint loyalty program must be pretty gratifying for them; it’s pretty clearly a hot topic on Twitter!

For me, it’s a brilliant way to extend my network, monitor the pulse of discussion, and spend more time thinking and debating and less time clicking. For organisations who care about their customers, it might be a way to create a personal, two-way dialogue with all of their customers. Or, it might be a way to help them solve their customers’ issues as they experience and Tweet about them. Or, it might simply be a way of keeping track of what’s hot at the moment, quickly, easily, and dynamically.

In any case, I find it tremendously empowering. It’s not just that I’ve got another way of taming the deluge of information I’m increasing hit with every day; it’s also that I know that if I say something, the odds of someone hearing it who cares about similar things to me increases every day. And, SAS is right there, helping make it easier.

A great thing about working at SAS is that I’m surrounded by smart and creative colleagues like Evan and Zach (and over 10,000 others from across the globe). If you aren’t familiar with SAS, here is a recent write-up in Investor’s Business Daily. May I never have to leave SAS, or ever again have to work at a public company.

Can the results from text mining tweets be of use in forecasting? Like the use of Google Trends data in forecasting (discussed in this blog on July 10), this is an area of active research. While it is exciting to have all these new data sources, it is still to-be-determined whether they can actually improve the accuracy of our forecasts. Are you doing research in this area? If so, I invite you to share your results in a guest blogger posting on The BFD.

For more information on using SAS to analyze Twitter data, and for a sample of Evan’s code, you can contact him directly at evan.stubbs@sas.com.

The Summer 2009 issue of Foresight is now available, and features a section on “Rethinking the Ways We Forecast.” Here is Editor Len Tashman’s preview:

Are traditional forecasting tools suitable for predicting complex systems like the economy and the global climate? Basically, no, argue David Orrell and Patrick McSharry: such tools are based on equations that model a system’s components but ignore its emergent properties, the global effects arising from those components. They call this the reductionist approach. All models, they assert, make simplifying assumptions, but the reductionist
approach makes the wrong assumptions.

David and Patrick then describe key elements for more effective modeling of complex systems, including agent-based models, network analysis, nonlinear dynamics, and scenarios. These models shift the emphasis from the point forecasts most often demanded by business decision makers to the assessment of risks in what the future may bring.

The section continues with two commentaries. Roy Batchelor illustrates the difference between a simple macroeconomic-forecasting model (representing the reductionist approach) and a complex-system model, and compares the virtues of the two viewpoints. Roy’s concern is that specific complex-systems models for the economy may be unwieldy and unstable, complexifying without improving forecasting.

Robert Fildes and Paul Goodwin note that the complex-systems models are untried in the arena of economics and that we need to explore whether these models are better applied individually or together. They reinforce the Orrell-McSharry thesis that point forecasts are overemphasized and misapplied but argue that they are unavoidable in the business world. Robert and Paul conclude with their own scenario on the future of forecast modeling.

This special feature wraps up with David and Patrick’s responses to the commentaries. They emphasize that complex-systems models need not be complex and that many biological applications of these models are relatively simple, indeed simpler than many current models in use.

In your editor’s view, the section reveals essential agreement among forecasters that there is much room for rethinking and refining our current approaches to forecasting, placing greater emphasis on risk assessment and preparation for uncertain futures.

On a Monday July 20 segment of consumer advocate Clark Howard’s radio show, Clark discussed the common practice of hidden payments to influential bloggers. Apparently these high-tech shills pocket the payola, and then make favorable postings about particular products or services. According to Clark, there are new rules to prevent this kind nefarious behavior, requiring bloggers to disclose receipt of such compensation.

All this made me realize that some of my readers might not be aware that the world’s largest privately held software company, SAS Institute, the undisputed leader in advanced analytics software, pays me to write this blog. I know it may be surprising to those individuals who, aware of my vast independent wealth and altruistic instincts, were thinking I’ve been serving as a product marketing manager at SAS for free these past five years. So for anyone who has been confused, let me state it again unambiguously: SAS Institute of Cary, NC, USA, pays me to write this blog.

Aren’t the internets wonderful? Just today I was trying to find the antonym of “naïve” and came across several terrific choices (sophisticated, worldly, well-informed, and intelligent) and one that didn’t make any sense (svelte???). However, upon further review at Merriam-Webster.com, I discovered that in addition to slender, lithe, and sleek (the definitions I expected), svelte could also mean urbane or suave. So a person could actually be svelte and obese at the same time. I never would have known that – thank you Al Gore!

In real life, it is probably a good thing to be informed, skeptical, and difficult to be taken advantage of. In short, it is good to be svelte. This applies to your (hopefully limited) encounters with strange men at highway rest stops, as much as it does to your (hopefully even more limited) encounters with forecasting software vendors.

Despite my plea that you remain svelte in real life, I implore you to be naïve in business forecasting – and use a naïve forecasting model early and often. A naïve forecasting model is the most important model you will ever use in business forecasting. It should also be the worst forecasting model you will ever use – but probably won’t be. Let me explain…

Per the standard forecasting text, naïve forecasts are “Forecasts obtained with the minimal amount of effort and data manipulation and based solely on the most recent information available.” An important characteristic of a naïve forecasting model is that it can be easily automated and produced at virtually no cost -- without the need for forecasters or forecasting software. This is important because it sets a baseline for performance. If you can achieve X% error using a naïve model, then you sure as heck better be able to achieve less than X% error with whatever people and process and technology you are using to forecast. This is the fundamental idea behind Forecast Value Added Analysis, where you compare all forecasting process activities to “doing nothing” and eliminate those activities that aren’t making the forecast any better.

Purists may argue that the only true naïve forecast is the “no-change” forecast, meaning either a random walk (forecast = last known actual) or a seasonal random walk (e.g. forecast = actual from corresponding period last year). These are referred to as NF1 and NF2 in the Makridakis text (where NF = Naïve Forecast). In our 2006 SAS webseries Finding Flaws in Forecasting, an attendee asked “What about using a simple time series forecast with no intervention as the naïve forecast?” Is that allowed?

Our purpose is to determine whether all our elaborate forecasting systems and processes are adding value by making the forecast better. For this objective, it is perfectly acceptable to use something more sophisticated than a random walk as another point of comparison in the FVA analysis. A thorough FVA analysis evaluates the performance of every step and participant in the forecasting process. If you have forecasting software that will automatically generate forecasts for you (essentially for “free” once you have licensed and installed the software), it is important to know whether that system generated forecast is any better than NF1 or NF2. The key is comparing costly and heroic forecasting efforts to forecasts created by doing the minimum amount of work. Does the extra cost and effort make a meaningful improvement in the forecast? If not, then the cost and effort probably aren’t worth it.

I personally won’t report you to the forecasting police if you do use something a bit more sophisticated than NF1 or NF2 as your naïve model. A moving average or simple exponential smoothing are suitable choices. However, I will report you for failing to do the appropriate comparisons – and you know what happens then.

In April 2009, Google published a draft research paper “Predicting the Present with Google Trends,” by Google’s Chief Economist Hal Varian and Decision Support Engineering Analyst Hyungyoung Choi. The paper is available for download in an April 2 posting by Varian and Choy on the Google Research Blog that has stirred a lot of commentary.

The paper describes using search query volumes to predict economic activity – such as Ford car sales. The authors contend that by incorporating relevant search query volume in their models, considerable accuracy improvements are obtained over standard auto-regressive models. Note that the authors are focused on very short-term economic prediction.

If this approach really works, the search data (publicly available through Google Insight) could be a boon to near-term forecasting. But does it work? My colleague Udo Sglavo, Solution Architect in the SAS Technology Global Practice, attempted to reproduce the paper’s results, and found some problems with the authors’ claims. Here are Udo’s findings:

Guest Blogger: Udo Sglavo, SAS Technology Global Practice

1. The authors write: “We are not claiming that Google Trends data help predict the future. Rather we are claiming that Google Trends may help in predicting the present. For example, the volume of queries on a particular brand of automobile during the second week in June may be helpful in predicting the June sales report for that brand, when it is released in July.”

• To me it does not matter how they call it – it is still a forecast I think, as it is about predicting the future. In the first example (predicting Ford sales) the sales data is monthly, while the Google Trend data is weekly. What is illustrated in the example is that one could use the Google’s trend data (in fact the first week of a given month) to increase accuracy of the monthly prediction. With other words – the trend data is used as a leading indicator for sales. Even if they are measured on different frequencies (monthly vs. weekly) they are still using a future week to predict the upcoming month. In their example: use trend information of first week in September 2008 to predict September 2008 sales.



• I have tried to replicate the Ford Sales example however, when using SAS Forecast Server I don’t seem to manage to outperform the ESM – compared on in-sample performance, which is what the authors are suggesting to do. Unfortunately I don’t seem to be able to draw the same conclusion as the authors that the transfer model (using an input) is superior to a smoothing model (without input). Even when flagging July 2005 as an outlier (as suggested by the authors). I’d be interested if somebody else is able to replicate their findings.

2. As we it has been stated by Mike earlier there are lots of misconceptions about forecasting accuracy – and sometime even researchers are not fully aware of them – for example:

•The authors draw a lot of the conclusions on statistics based on in-sample data – not out-of-sample. Not recommended by Armstrong’s Standard's & Practices for Forecasting (13.26).

• In chapter 2.1 the authors claim: “Note that the R-squared moves from 0.6206 (Model 0) to 0.7852 (Model 1) to 0.7696 (Model 2).” However, R-squared should not be used to compare accuracy of forecasting models (Standards & Practices for Forecasting (13.28), or see the Forecasting Principles website).

• The author uses standard regression diagnostics plots to make his points – this seems not correct to me.

Our SAS colleague Terry Woodfield, Statistical Services Specialist, also weighed in, arguing there is a fundamental flaw in the behavioral model underlying the Google approach:

Guest Blogger: Terry Woodfield, SAS Education and Training

The premise is appealing, but if true, could we not make millions in the stock market? The behavioral model suggests that a person visits the Web because of interest in making a purchase, learns neat stuff, and then goes out to make the purchase. The negative side is not mentioned. Consider….

Before the Internet, I go to an automobile dealership because I am interested in making a purchase. I succumb to high pressure sales tactics, and I make a purchase.

In the Internet age, I first go to the Internet when I am interested in making a purchase. From the Internet, I learn that the vehicle I am interested in: (1) harms the environment, (2) has a terrible safety record, (3) has a terrible frequency of repair record, (4) costs more than anticipated, and (5) angers God. I choose not to go to the dealership, thus preventing an opportunity to be persuaded by the evil, devious salesperson. Because of the Internet, I DO NOT make a purchase.

Consider also that while I am browsing porn sites on the Internet, I am not watching slick television ads that might convince me to buy a light duty truck. (Hypothetically browsing porn sites for the purpose of discussion, of course.)

If the Internet has a positive effect on sales, I benefit from Google data. If the Internet has a negative effect on sales, I benefit from Google data. If the Internet has both positive and negative effects that cancel each other out, I do not benefit from Google data.

Where does this leave us? If we grant that adding the Google Trends data improves the authors’ seasonal autoregressive model, we are left with Udo’s finding that in this case the exponential smoothing model (that doesn’t use the Google data) is simpler and performs better! Of course, we can’t draw any broad brush conclusions based on just the few examples exhibited in this paper. However, this reminds us that when it comes to forecasting (and not just fitting models to history), simpler is often better -- and simpler is always preferred.