The Business Forecasting Deal - Exposing bad practices in business forecasting... and providing practical solutions

In case you missed the business news last week, SAS (the company that pays me to write this blog, and the leader in business analytics) was named #1 on the FORTUNE 100 Best Companies to Work For 2010 list. Having been a user (and huge fan) of SAS software since 1985, I am delighted to see the growing recognition of something I’ve known for 25 years: SAS has great products, and SAS has great people. (I only regret I wasn’t considered photogenic enough to make it into the FORTUNE video. They did tell me, however, that I have the perfect face for radio – or blogging.)

I must also point out that for 2009, SAS achieved its 34th consecutive year of revenue growth and profitability, up 2.2% to US$2.31 billion, despite poor economic conditions worldwide. Said CEO Jim Goodnight, “Our continued growth in 2009 is further proof that investing in long-term relationships with customers and employees and maintaining a deep commitment to R&D pays.” Let’s hope other business executives can take a few lessons from the SAS approach to management. It isn’t just charming colleagues like me that make SAS such a great place to work.

If you need a good excuse to visit SAS in person, please join forecasting professionals from industry, government and academics, and from around the globe, at the F2010 Business Forecasting Conference here at SAS world headquarters in Cary, NC, June 7-8.

If you can’t wait until June to get your forecasting fix, please join me and Emily Rodriguez, Program Manager at Intel, for a ½ day workshop “A Primer for Management: Fundamentals of Business Forecasting and Conducting Forecast Value Added Analysis,” at the IBF’s Supply Chain Forecasting Conference in Phoenix, February 21-23. A webinar previewing the workshop is available for free on-demand review.

Today my colleague Alison Bolen, Editor of sascom magazine, sent me this link to an interesting piece on NPR: "Can Economic Forecasting Predict The Future?" In a somewhat lighthearted take on the inability of our economists to predict the future -- or even precisely report the past for that matter -- we hear about the modeling efforts of "one of the leading economic forecasters in America." These efforts revolve around 447 pages of math equations. Do I hear "**** DANGER WILL ROBINSON DANGER ****" ringing in my ears?

One sure warning sign that you aren't going to be able to forecast something worth a darn is when it takes 447 pages of equations to model it. Makridakis, Hogarth, and Gaba state it nicely in their recent International Journal of Forecasting article, "Decision making and planning under low levels of predictability":

…a huge body of empirical evidence has led to the following conclusions.
• The future is never exactly like the past. This means that the extrapolation of past patterns or relationships cannot provide accurate predictions.
• Statistically sophisticated, or complex, models fit past data well but do not necessarily predict the future accurately.
• “Simple” models do not necessarily fit past data well but predict the future better than complex or sophisticated models.

Happy New Year.

Although I would like to flatter myself and believe that my readers rely solely on The BFD for all their business forecasting news and information, I realize this is not the case. While other sources may not be as honest, useful, or delightfully entertaining to young and old alike, they do provide additional material I'm unable (or unwilling) to cover, and they provide alternative viewpoints. Here are some sources that may be of interest:

International Institute of Forecasters (http://www.forecasters.org/)
o Academic and practitioner membership
o Publications:
 International Journal of Forecasting – academic journal
 Foresight – practitioner journal
 Oracle of the IIF – online newsletter
o International Symposium on Forecasting held every June at sites worldwide
o IIF members get free online access to the journals.

Institute of Business Forecasting & Planning (www.ibf.org)
o Predominantly forecasting practitioners
o Publications:
 Journal of Business Forecasting – practitioner journal
 Supply Chain Forecasting Digest – online newsletter
o Offer webinars, training courses, tutorials, and several conferences per year in the US and abroad
o IBF members get free online access to all past JBF articles (over 25 years worth).

Websites & Discussion Groups:
• Forecasting Principles (http://www.forecastingprinciples.com/) – Sponsored by the IIF, providing (per the website) a summarization of “all useful knowledge about forecasting.” (And I thought The BFD was run by an egomaniac.)
• IBF Forecasting Discussion Group (http://www.ibf.org/forums/) – There is also an IBF discussion group on LinkedIn that is quite active.
• SAS Forecasting User Forum (http://support.sas.com/forums/forum.jspa?forumID=28) – Generally more technical discussion of forecasting issues and SAS forecasting software.
• Applied Forecasting (http://www.appliedforecasting.com/) – A frequently updated compilation of links to forecasting articles, blogs, and events.
• Forecast Focus (http://forecastingexcellence.ning.com/) – Social networking site for forecasting professionals, with links, blogs, and a discussion forum. (If eHarmony has rejected you, and Craigslist personals isn't working out, I think you can still get a date here.)

Constance Korol oversees the Institute of Business Forecasting & Planning group on LinkedIn. (No, she isn’t the meaner one I will be referring to, but she can swing a nasty rolling pin if you get out of line.) This week Constance posted a Wall Street Journal article “Follow the Tweets,” and solicited feedback from the group. The article described a method of monitoring comments on Twitter to predict where next week’s sales are heading.

While I was initially excited to hear about a new method to improve forecasting, it turned out to be just another tease. As often happens (even for some forecasting consultants and software vendors), the article’s authors had confused forecasting with fitting a model to history. They never actually did any forecasting with the proposed method to test how well it worked, and provided no convincing reasons to believe it would be of any use. Being the kind of unpleasant person I was raised to be, I left a relatively uncomplimentary review of the article for the IBF group:

The authors state:

"Thus, by observing the number of positive tweets after opening weekend, we could have predicted that the movie would continue to generate revenue in the coming weeks."

To claim that they "could have" predicted something assumes that their model works. But they fail to give us any reason to believe the model works, since they don't appear to have actually done any forecasting with it. (Forecasting is predicting future behavior, not fitting a model to past behavior.)

Would like to see these folks actually forecast something with their model, and then let us know whether the model is useful. So far, sounds like they have fit a model to the historical behavior of 3 movie releases. Fitting a model to history is not the challenge -- the challenge is to create good (or at least useable) forecasts. I would be very happy to see them demonstrate the ability to create good forecasts using Twitter data -- we all need ways to improve our forecasts -- but so far they have not shown evidence of that capability.

This caused me an (ever so slight) twinge of remorse for being so harsh.

Fast forward to one Deb Di Gregorio of Camarès Communications. Deb commented on the article via a wonderfully fiendish blog post “Biz Flash: Tree Falls in Forest! No One Tweets!” which begins thusly:

A most remarkably huge section of the Wall Street Journal was wasted today on Follow The Tweets an article by three professors who set out to attempt to predict which movie would do best at the box office based on Tweets.

It doesn’t get any nicer after that, as she states later on:

This only proved one thing: there are enough Tweets about movies to give three professors enough data to write something utterly inane and enough ignorant editors at the WSJ to approve the article.

Leona Helmsley and Lisa Lampanelli step aside, we have a new Queen of Mean! (I say this in the most complimentary fashion possible.)

The purpose of The BFD is to encourage new ideas, creative thinking, and alternative approaches to the problems of business forecasting. But this does not entail the blind promotion of each new forecasting method that is developed. On the contrary, The BFD is a forum for exposing practices that are either unproven, or demonstrably wrong. Like the value of Google search statistics (which was discussed here on July 10), the value of Twitter data to aid in forecasting remains to be proven.

And I thought I was mean…

I’m going to put “An Operational Definition of ‘Demand’ – Part 3” on hold for a moment, to announce a new favorite article on forecasting, “Living in a world of low levels of predictability,” by Spyros Makridakis and Nassim Taleb (International Journal of Forecasting 25 (2009) 840-844. IJF is a publication of the International Institute of Forecasters, and if not already a subscriber you can purchase the article from ScienceDirect.)

Many of you already know Makridakis as co-author of the standard forecasting text Forecasting: Methods and Applications, and Taleb for his Fooled by Randomness and The Black Swan. Taleb, in particular, has drawn attention to the issue of the un-forecastability of complex systems, and the sometimes disastrous consequences of our “illusion of control, pretending that accurate forecasting was possible” (p.841).

While referring to the (mostly unforeseen) global financial collapse of 2008 as a “prime example of the serious limits of predictability” (p.240), this brief and non-technical article summarizes the empirical findings for why accurate forecasting is often not possible, and provides several practical approaches for dealing with this uncertainty.

So why am I, a vendor of forecasting software, so excited by an article telling us the world is largely unforecastable? Because Makridakis and Taleb are correct – we should not have high expectations for forecast accuracy, and should not expend heroic efforts trying to achieve unrealistic levels of accuracy. Instead, by accepting the reality that forecast accuracy is ultimately limited by the nature of what we are trying to forecast, we can instead focus on the efficiency of our forecasting processes, and seek alternative (non-forecasting) solutions to the business problem. Methods I have touted, such as Forecast Value Added analysis, can be used to identify and eliminate forecasting process activities that do not improve the forecast (or may even make it worse). Large-scale automated software, such as SAS Forecast Server, can deliver forecasts about as accurate and unbiased as anyone can reasonably be expected – and do this without elaborate processes or significant management intervention. For business forecasting, the objective should be:

To generate forecasts as accurate and unbiased as can reasonably be expected – and to do this as efficiently as possible.

The goal is not perfect forecasts – that is wildly impossible. The goal is to try to get your forecast in the ballpark, so you can plan and manage your business effectively, and not waste a lot of company resources doing it. And when, because of the nature of demand or other behavior, you cannot forecast with the degree of accuracy needed for effective planning, then seek alternative approaches to address the underlying business problem. In the past I’ve suggested things like demand smoothing (to make the demand forecastable), or supply chain re-engineering (to minimize your reliance on accurate forecasts). You can find more discussion of these in a 2001 article I co-authored with Drew Prince of NCR, “New Approaches to Unforecastable Demand” (Journal of Business Forecasting, Summer 2001, pp. 9-12), available for download from the Institute of Business Forecasting.

In the last post I argued that we don’t have a sure way to measure true (i.e. “unconstrained”) demand. While demand is commonly defined as “what the customer wants, and when they want it,” it is actually a nebulous concept. For a manufacturer, what a customer orders is not the same as true demand (for various reasons described in the prior blog post), nor is what actually ships. At a retailer, what is actually sold off the shelves is not the same as true demand, either. For example, the customer may not be able to find what they want in the store (due to out-of-stocks, or poor merchandise presentation), so there is true demand but no recorded sale. Determining true demand for a service can be equally vexing. I may have a taste for a Royale with Cheese at McDonald’s, but go to Wendy’s instead if the drive-thru line is too long. Or I may call the cable company to complain about my tv reception, only to hang-up in frustration trying to wade through their voice menu system.

While we may know true demand under certain special circumstances, we don’t have a general operational definition that will work in all situations. This means two things:

• We will be unable to construct a history of true demand to feed into our statistical forecasting models (except under certain special circumstances)
• We will be unable to assess the accuracy of our forecast of true demand (i.e. our “unconstrained forecast”) (except under certain special circumstances)

Conceptually these are important points. In most circumstance we don’t know what true demand really is, so we don’t know for sure how accurately we are forecasting it. I argued, therefore, that forecasting performance should be evaluated against the “constrained” forecast. The constrained forecast represents our best guess at what is “really going to happen” -- i.e. actual shipments or sales or services provided. We can measure what actually happens.

As a practical matter, maybe all of this isn’t so important. While we can’t know exactly what true demand really is under most circumstances, we can often get close enough to make the concept useful in forecasting. It is true that demand ≠ orders, yet if an organization does a good job at filling orders (say 98%+), then “orders” and “true demand” are virtually the same (within a few percentage points). When we forecast, our errors are often 25%, 50%, or more. The fact that the demand history upon which we build our forecasts is not perfect (but may be off by a few percentage points from true demand) is inconsequential compared to the magnitude of the forecast error. (Even if we were able to capture perfect history of true demand, it might only make our forecasts a few percentage points better at best – and that isn’t going to rock anyone’s world.)

There is probably more to say on this topic, so expect a Part 3.

Sorry about not getting a post out last week, but I spent a good part of it cowering under my desk in fear. The SAS Security office issued a warning that there were wild coyotes roaming the campus, and I was having post-traumatic flashbacks to a painful encounter I once had with a cougar during my late teens. While coyotes can be wily and ill-tempered, I now realize they aren’t nearly as aggressive as cougars, so there was little to fear. (Fortunately I don’t have to fear cougars anymore either, because at my age they no longer consider me an appetizing target.)

That brings us to today’s topic, creating an operational definition of “demand.” Those of us in forecasting use the word demand every day, and we don’t think much about it – it seems pretty straightforward. Demand is commonly characterized as “what the customers want, and when they want it,” sometimes with the added proviso “at a price they are willing to pay, along with any other products they want at that time.” Sounds good to me.

When we refer to demand, we really mean “unconstrained” or “true” demand, because we take no consideration of our ability to fulfill it. We use the phrase “constrained demand” to describe what’s left after incorporating any limitations on our ability to provide the product or service demanded. Thus, constrained demand ≤ demand. So far so good -- do I really need to devote a blog entry to something so self-evident?

This definition of “demand” is not problematic until we try to operationalize it, that is, when we start to describe the specific, systematic way to measure it. This is kind of important if we ever expect to measure the accuracy of our “demand forecast.” We need to know what actual demand really was!

Let’s work through an example I originally wrote about in the Summer 2003 issue of Journal of Business Forecasting, dealing with this issue at a manufacturer.

If customers place orders to express their “demand,” and if the company services its customers perfectly by filling all orders in full and on time, then we have our operational definition. In this case, demand = orders = shipments. If both order and shipment data are readily available in the company’s system, then we have the historical demand data, which we can use it to feed our statistical forecasting models and measure our forecasting performance.

Unfortunately, few organizations service their customers perfectly. As such, orders are not a perfect reflection of true demand. This is because when customer service is less than perfect, orders are subject to all kinds of gamesmanship. Here are a few examples:

1. An unfilled order may be rolled ahead (carried over) to a future time bucket.
2. If shortages are anticipated, customers may artificially inflate their orders to capture a larger share of an allocation.
3. If shortages are anticipated, customers may withhold orders, or direct their demand to alternative products or suppliers.

In the first example, demand (the order) appears in a time bucket later than when it was really wanted by the customer. Rolling unfilled orders causes demand to be overstated -- the orders appear in the original time bucket, and again in future buckets until the demand is filled or the order is cancelled.

In the second example, the savvy customer (or sales rep) has advanced knowledge that product is scarce and will be allocated. If the allocation is based on some criterion such as “fill all orders at X%,” the customer simply over-orders and ultimately may receive what it really wanted in the first place.

The third example not only contaminates the use of orders to reflect true demand, but it can also cause significant financial harm to your business. If you are in a situation of chronic supply shortages (due to either supply problems or much higher than anticipated demand), customers may simply go elsewhere. Customers may truly want your product (so there is real demand), but it won’t be reflected in your historical data because no orders were placed. While orders are often perceived as “equal to or greater than” true demand, this third example shows that what is ordered may also be less than true demand.

As with orders, the use of shipments to represent demand has a number of potential problems. Shipments are often perceived as “equal to or less than” true demand. Thus, shipments and orders are thought to represent true demand’s lower and upper bounds. But, as we see in example 3, orders can be lower than the true demand. Furthermore, by example 1, shipments can actually be greater than true demand. (This would occur when an unfilled order is rolled ahead into a future time bucket and then filled. In this situation the shipment occurs later than the true demand, and inflates demand in the time bucket in which it is finally shipped.)

The planning process starts with a compilation of unconstrained or “true” demand history. We feed that into our statistical software and generate an “unconstrained forecast.” In order to measure our forecasting performance, we need to know the actual demand that really occurred. Herein lies the problem. There may be no way of knowing what true demand really was. The manufacturer knew what orders it received, and knew what shipments it made. But neither orders, nor shipments, nor any other readily available data element is the same as true demand. Therefore, we cannot measure the accuracy of our unconstrained forecast.

The lesson is to always report forecasting performance against the constrained forecast. The constrained forecast represents what the manufacturer actually expects to ship, what the retailer actually expects to sell, and what the service organization actually expects to provide. The constrained forecast is our best guess at what is really going to happen, and we can measure what truly does happen. While we can measure the shipments, the sales, or the services provided, we cannot with certainty measure what was truly demanded.

Last week I was a guest of Gaurav Verma on the SAS Applying Business Analytics Web Series, and presented “What Management Must Know About Forecasting.” One of the most important things you can bring to management’s attention is the benefit of making your demand forecastable.

In forecasting we tend to treat demand patterns as given, as if there were nothing we could do about them. This “passive” attitude makes our performance contingent on the forecastability of the demand patterns. If the patterns are easy to forecast then we should do fine, but if they are not, we risk failing to meet our forecast accuracy objectives.

In my July 1 posting on The BFD, I used this scatterplot to illustrate the usual relationship between demand volatility (along the horizontal axis), and forecast accuracy (the vertical axis). I argued that while imperfect, the volatility of a demand pattern (as measured by its coefficient of variation) is a good indicator of how accurately we can expect to forecast that demand. As a practical first step to improving your forecasting performance, it is worthwhile to construct this scatterplot with your own data…and here’s how:



















1) Determine at what level you want to do the analysis. This is typically the level at which you focus your forecasting and planning efforts. For example, a manufacturer might forecast the number of units demanded for each Item, or at each Item / Distribution Center combination. A retailer might forecast unit sales by Item / Store. An insurance company might forecast claims by Region or Postal Code. (In the above scatterplot, this manufacturer sold roughly 500 Items through 10 DCs, and forecasted at the Item / DC level. There are, therefore, roughly 5000 (= 500 x 10) points in the plot.) Suppose for this example that we are doing the analysis by Item.

2) You will need to gather the last year of data for each Item, in whatever time bucket you use (typically week or month). (I prefer using weekly data if it is available, but many organizations plan in monthly buckets, so we’ll use monthly for this example.) For each item you will need to capture the monthly Sales, and whatever the Forecast was for that month. So you will create a dataset with four variables . The four columns on the left show what this dataset might look like with the 12 monthly observations for Item XXX.



3) You next compute the volatility of sales over the past year. Volatility is measured by the coefficient of variation (CV) of the monthly sales, which is defined as the standard deviation divided by the mean. For item XXX over the past year, it had mean monthly sales of 102.5 units, and a standard deviation of 26.8. Therefore, CV = 26.8 / 102.5 = 26.2%.

4) You finally have to compute Forecast Accuracy for each Item. This requires two new computations shown in the two rightmost columns. The Absolute Error in forecasted sales each month (i.e. |Forecast – Sales|) and the Maximum of Forecast or Sales in each month. Forecast Accuracy is computed as 1 – (Sum of Absolute Errors) / (Sum of Maximums) = 1 – (150/1320) = 88.6%.

5) Create the scatterplot by plotting the Forecast Accuracy and Volatility coordinates for each Item. The Forecast Accuracy metric will always be from 0 to 100%, so the vertical axis can be scaled 0 – 100. The Volatility, however, can become indefinitely large. For clarity and usefulness of the presentation, you may want to ignore really extreme values of CV (unless there are a lot of them), and only run the horizontal axis from 0 to 150% or 200%, and footnote how many Items are not being shown. The point is to illustrate the overall volatility of your sales patterns, and to show the relationship with your ability to forecast.

As long as you have the past year of sales data, along with the last year of forecasts, it is very easy to create this scatterplot. It is a simple way to visualize the extent of volatility issues with your sales, and the likely impact on your forecasting performance.

There are some things about forecasting that you may only discover by being a forecaster. Simply managing a forecasting process, or being a downstream consumer of the forecast, isn't always enough. If you have something to say to your management about forecasting, but would rather avoid the confrontation, maybe we can help.

Please bring along your management to join me and my SAS colleague, Business Analytics Marketing Manager Gaurav Verma, for Part 8 of the Applying Business Analytics Webinar Series (Wednesday October 21, 1:00 p.m. ET): What Management Must Know About Forecasting. We'll be covering some of the fundamental issues and worst practices that may not be so apparent to those managers and executives not directly involved in the forecasting process.

Click here for more information and free registration.

For those unable to attend live, this webinar will be recorded and made available for free on-demand viewing.

Before we begin, I want to thank all of you readers who supported my righteous appeal for “THE BFD” license plate from the state of North Carolina. I am sad to report, however, that our efforts were in vain and the appeal was denied by Censorship Board of the Division of Motor Vehicles. Along with the denial, these arbiters of good taste were kind enough to send me an application to re-submit another idea – any suggestions? (Note: I’ve already given one new suggestion to the DMV, but they responded back that it was anatomically impossible.)

Now back to the business of forecasting…

Just how confident can we be about the future? This is an important consideration that plays into the decisions we make based on our forecasts. Makridakis, et al, discuss this in Forecasting Methods and Applications:

“It is usually desirable to provide not only forecast values but accompanying uncertainty statements, usually in the form of prediction intervals. This is useful because it provides the user of the forecasts with “worst” or “best” case estimates and with a sense of how dependable the forecast is, and because it protects the forecaster from the criticism that the forecasts are “wrong.” Forecasts cannot be expected to be perfect and intervals emphasize this.” (p. 52)

Not all forecasting software provides prediction intervals, but SAS High-Performance Forecasting (the “engine” inside SAS Forecast Server software) does. The width of the prediction interval (sometimes referred to as the confidence limits or confidence bounds) indicates how close the future is likely to be to the forecasted value. (Confidence bounds are shown by the shaded blue area on the far right of this plot.)

When the confidence bounds are wide, forecasters sometimes whine and complain that they are useless – what good is a forecast in which we can have no confidence? But what is there to do in this situation? My colleague Bob Lucas provides some thoughts on this matter.

Guest Blogger: Bob Lucas, SAS Education & Training

I have taught forecasting for 12 years here at SAS and many a student is often disappointed in the widths of the forecast confidence bounds. They have said, “What good are the confidence bounds that are so wide?” Here is my typical response:

What affects the widths of the confidence bounds?

1. The amount of data you have. (You may or may not have control over that.)
2. The model that you are using. (You do have control over that, however, with limited data you may not have too many choices.)
3. The natural variability in the data. (You have no control over that!!!)

If your data has a lot of variability, guess what – you’re going to get wide confidence bounds. Some customers have unrealistic expectations for the accuracy of their forecast.

Let’s examine the one aspect that you do have control over – the model that you select to produce the forecast. One of the standard premises of forecasting (actually prediction in general) is simpler models do better – all else being the same. While a more complex model (with more parameters) will be able to “fit the history” better than a simpler model, it will likely do no better at forecasting the future.

If you do not use a holdout sample, fits statistics that ignore the number of parameters will favor models with more parameters. Consequently, I favor using SBC to choose my forecast models when not using holdout data. (The Schwarz Bayesian Criterion, also called the Bayesian Information Criterion (BIC), is a criterion for selecting a model from among a class of models with different numbers of parameters.) I still report MAPE to business user, however.

One way to restrict forecast and confidence bounds is to fit the log of the series. SAS Forecast Server will transform back to the original scale for you. This disadvantage is that now the parameter estimates are on the log scale and complicate interpretation if desired.

Now my answer to the question “What use are the confidence bounds that are so wide?”

Don’t you think you would make a different business decision of your forecast was 100 +/- 10 versus 100 +/- 50? How does this relate to your situation? The inventory cost for a fixed service level will be much higher for a series with high volatility than for one with low volatility. The realistic way to handle this is to have different service levels for different products based on the volatility of the series.

As Bob is pointing out, when you have highly volatile demand for a product, you probably aren’t going to be able to forecast it as accurately, and will have wider confidence bounds around the forecast. As a result, you are going to need correspondingly (and perhaps prohibitively expensive) amounts of inventory available to meet a fixed service level target. What an organization needs to do, rather than set a fixed service level (such as “98% order fill rate”) across all products, is to set lower service level objectives for the more difficult to forecast products. If a certain service level (like 98%) is absolutely necessary, such as due to contractual obligations with the customer, you need to address the problem in an alternative manner, such as by finding ways to reduce the volatility (and “unforecastability”) of the demand patterns.

[For further discussion on ways to reduce the variability of demand, tune in to my upcoming webinar “What Management Must Know About Forecasting” (October 21, 1:00pm ET). This is part of the SAS Applying Business Analytics Webinar Series hosted by my colleague Gaurav Verma.]

In summary, if your software does not produce confidence bounds along with its forecasts, you are missing out on some very useful information. There is uncertainty in any forecast, but having an understanding of this uncertainty and an appreciation of the risk helps your organization avoid the really bad business decisions.

In case you haven't heard, the state of North Carolina needs money. Roads are falling apart, parks are left unkempt, prisoners are being released, and public school cafeterias can no longer afford to put anything surprising in their Chef's Surprise.

In order to help assuage the crisis, and in my most altruistic act since filling a rental car with Premium*, I gave the NC Division of Motor Vehicles an extra $30 to get myself a personalized plate (shown here from the actual DMV website):

The way I had it figured, what better way could there be to promote this blog and spread the word about all that is wrong in forecasting. The DMV website took my application, my credit card was charged, and all I had to do was wait for the friendly gentlemen in the NC state prison system to stamp out my plate and put it in the mail.

Apparently I figured wrong. Either they have released all the prisoners who know how to run the license plate machines, or the folks at the DMV are spending way too much time reading urbandictionary.com. Instead of getting my new plate, I received this:

DEAR MR. GILLILAND:
PERSONALIZED LICENSE PLATE THE BFD WAS ORDERED FOR YOU IN ERROR. THIS PLATE MAY BE CONSIDERED IN POOR TASTE AND THEREFORE CANNOT BE ISSUED.

WTF? (Which, by the way, was issued on 9,999 license plates across the state in 2008.) I don't know what they could have possibly found in poor taste about THE BFD. I immediately appealed, even providing a link to this blog, The Business Forecasting Deal, to show them what an important public service is being provided.

Anyway, if you'd like to help the just cause of THE BFD, please write in support of my appeal to:

Kay Hatcher
North Carolina Division of Motor Vehicles - Special License Unit
3155 Mail Service Center Division
Raleigh, NC 27966-3155

or email to: khatcher@ncdot.gov

Would you buy something that doesn’t have demonstrable Return on Investment? Of course you would! Whether you realize it or not, you do this every time you buy software. There is no proven ROI.

Maybe this is just a pet peeve of mine. Maybe I care too much about cause and effect. Maybe I care too much about logic and scientific method. Maybe I care too much about what it means to prove something. Or maybe I’m just another egomaniacal jerk with a blog that thinks he is entitled to raise a stir about nothing. I am, therefore I will.

Every software vendor touts the ROI you will achieve by buying their products. 10%, 100%, 1000% ROI – the claims are everywhere, and part of every sales cycle. And it isn’t just the vendors claiming ROI. It is also de rigueur for the customer’s internal project sponsor or champion to make claims of anticipated ROI -- in order to secure the funding to buy the software.

ROI is a sham??? Wow!

Think about it – how would you ever prove that implementing software X caused effect Y (where Y is increased revenue, reduced costs, higher profits, etc.). Just because one thing happens along with something else, does not prove there is any relationship between the two whatsoever – let alone a “causal” relationship. The problem is that things change, they vary over time. (Perhaps this is why they are called variables?) The interrelationships among variables can be subtle and mysterious, and we don’t really know for sure what mechanism (if any) is driving things.

We often (and wrongly) attribute causality in these sorts of situations. The geniuses in the financial news media, for example, can provide brilliant explanations for the cause of every market fluctuation. Nassim Nicholas Taleb has a great example of this in his book The Black Swan:

"One day in December 2003, when Saddam Hussein was captured, Bloomberg News flashed the following headline at 13:01: U.S. TREASURIES RISE; HUSSEIN CAPTURE MAY NOT CURB TERRORISM." Bond prices then fell. "At 13:31 they issued the next bulletin: U.S. TREASURIES FALL; HUSSEIN CAPTURE BOOSTS ALLURE OF RISKY ASSETS."

So, per Bloomberg, the capture of Saddam caused bond prices to both rise and fall? (I guess if Saddam were not captured, bond prices would both fall and rise?) Foolish me, I thought only Jim Cramer could come up with gold like this!

In some situations purported causality has been reasonably well established through controlled experimentation. For example, I accept that downing 400mg of ibuprofen will cause my headache to go away. But claims of causality from forecasting software are not subject to such rigorous oversight. There is not yet a Federal Forecasting Administration to vet our ROI claims.

When the economy is good and profits are rising, a positive ROI can be attributed to just about anything an organization does. (True Story: Within weeks after interviewing and accepting the position of forecasting manager at Iomega in 1996, its stock price rose from $28 to $110. Wall Street apparently heard I was coming!) Any vendor is sure to jump on this bandwagon, and attribute profit gains to implementation of its software. This makes a good story, but fails to consider what else has changed. Perhaps your company introduced a hot new product that generated tremendous revenues and profits. Or perhaps your customers were just spending more freely in the booming economy and you would have increased revenue and profits without the new software. There needs to be much better evidence to demonstrate the cause and effect.

If a vendor uses this argument in good faith, and accepts the “logic” that its software caused these profit gains at its customers, then what happens next? What about its customers that implemented the software in late 2007 or the first half of 2008? Is the vendor willing to apply the same “logic” and admit that its software “caused” these customers to have falling revenues, huge losses, and perhaps even go out of business in the greatest financial collapse since the Great Depression? Who would ever admit that?

This is my gripe – we are fast to apply causality when the “effect” is good, but can always come up with a creative explanation when the “effect” is bad. Yet without a rigorous (and probably impractical) proof of cause and effect, we are just talking a bunch of nonsense.

Don’t talk to me about ROI.

Another time I'll discuss more appropriate ways to evaluate the value of forecasting software.

I’m back from a week of vacation in Michigan and bursting with new topics, but will have to get to those next week (still playing catch-up in my day job). I do, however, have the pleasure to announce the publication of a new book by my friend and colleague Charlie Chase, Demand-Driven Forecasting: A Structured Approach to Forecasting. Charlie is currently Business Enablement Manager for the SAS Manufacturing and Supply Chain Global Practice, and has been a longtime contributor to the forecasting profession through his publications and involvement with the Institute of Business Forecasting. Charlie also preceded me as marketing manager for SAS forecasting software, and led the launch of our flagship SAS Forecast Server in 2005.

In his column in the Summer 2009 issue of Journal of Business Forecasting, Larry Lapide had this to say about Charlie and the book:

“I believe he captures a lot of what industry forecasters (like yourself) have been doing over time to improve forecasting by understanding the impact of demand-shaping activities. Much as a practitioner, Bob Brown described history-based forecasting over 50 years ago, Charlie, another industry practitioner, has done the same for today’s demand-driven forecasting.”

This book is part of the Wiley & SAS Business Series, and you can order it now.

The SAS internal discussion boards are always full of fascinating topics, some of which are even decipherable to a non-Ph.D. in statistics like me. A recent topic involved how to calculate the benefits of good forecasting software, and my colleague Robin Way offered an interesting perspective that he allowed me to share:

Guest Blogger: Robin Way, Analytics Consulting Manager at SAS

Perhaps the primary benefit of good forecasting software is reduced uncertainty around the most likely forecast, as well as potential improvement in accuracy. Most of the time, I have found that lay business people who are not that familiar with forecasting and the business payoff of better forecasts immediately think that a better forecasting engine will lead to improved accuracy. However, most seasoned forecasters accept that nobody can really tell the future and every forecast is inherently wrong. The point of better forecasting processes is that you now have an enhanced ability to know how far wrong from the point forecast you are likely to end up, and what factors influence that degree of wrong-ness.

I know that all sounds pretty soft; so consider this business case made for a call center capacity planning group. The main driver of business performance in managing capacity for a call center is the cost of call handling to deliver a specified level of service to customers. (Note that call handling costs include personnel costs (salaries and benefits), scheduling, availability of current resources, likely attrition of existing staff, cost and timeframe to bring on new trained staff, and other factors.) Put too few call center agents on the phones, and your service level suffers; put too many agents on the phones, and cost per call suffers.

Improved accuracy around demand for inbound calls will certainly help the capacity planners target the most appropriate number of agents to schedule for any period of time, but everybody knows there’s no way to know for sure exactly how many calls will come in. This is the element of forecast error. You don’t want to staff for the most likely number—really you want to staff for the upper range of likely call volume, because it’s typically worse to deliver a low level of service than to have too many low-utilization agents on the phones. (You can always divert the slack resource to outbound calls or post-call cleanup for busy agents.)

So how to estimate this upper range? That’s where having a really good forecasting engine comes in. Tools like SAS Forecast Server help you reduce the confidence limits around the most likely forecast by reducing unexplained variance—that’s what you get by having a better model, via a better engine. You can identify what drives the uncertainty so that you can advise management whether the sources of uncertainty in the forecast are under their control (marketing promotions sponsored by the firm, price changes, new product introductions, contractual changes with existing customers), or outside their control (market and competitor forces, seasonal shifts, market events). And you can develop a more robust upper bound on the likely forecast in order to intelligently set your service levels for upcoming call handling periods.

If you then compare the upper bound on the forecast from a relatively naïve forecast model with one produced by a superior model, presumably generated with Forecast Server, you can show the difference in the upper bound (in this case, in call demand) as a reduced operational risk to the firm. All that on top of what is possibly a more accurate forecast to begin with, though I advise against making your case solely on the point of accuracy, because often times the process generating the actual results can vary relative to the forecast for reasons outside the control of the forecaster.


Robin delivers an important reminder. Your forecast should be more than just a point estimate representing your best guess at what is really going to happen. To drive better decisions and more profitable operations, you need to take into consideration the likely error in your point forecast. Forecasting for new products gives a prime illustration.

New products, especially those requiring new technology or manufacturing processes that require a lot of capital investment, can be a big gamble. If we fail to take into consideration the uncertainty (risk) in our point forecasts, we may take unnecessarily dangerous gambles. I’m not arguing against taking gambles with your business investments – every decision and action involves some risk. However, it is important to recognize the scale of the risk, and act accordingly.

Personally, I don’t get Twitter. I have an account (mvgilliland) for anyone interested in not hearing any tweets from me. I follow a few people and have a few followers (including some that aren't porn bots) -- but what is the point? Does anyone really care that I’m out hanging floss on the line to dry, or that I’m stuck in the waiting room of my urologist with a prostate the size of a grapefruit?

The fact is, if someone is that interested in what I’m doing right now, it makes me kind of nervous. Do I really want people “following” me? Aren’t anti-stalking laws enacted for good reason?

Call me old school, a luddite, a 21st century puritan, even a techno-prude. Or perhaps I’m just blind to the great new opportunities for data (not just dating) that social networking provides. I have actually been swayed a little bit in this direction by my colleague in Australia, Evan Stubbs. Leveraging some code from SAS software developer Zach Marshall, Evan put together a neat little demo for customers, illustrating how SAS Text Miner can be used to identify patterns that could be used in forecasting. From Evan:

Guest Blogger: Evan Stubbs, Solution Manager for SAS Analytics

Rightly or wrongly, we seem to love telling the world what we’re doing, often even if no-one’s listening! Morgan Stanley recently published a report by a 15 year-old intern that for many, seemed to state the obvious: “On the other hand, teenagers do not use twitter … they realise that no one is viewing their profile, so their ‘tweets’ are pointless”.

Ignoring the bizarre implications that ‘only oldies use Twitter’, to me, this misses the point; it’s not about who’s listening right now, it’s about who might be listening. One of the points of talking publicly about a particular topic is the hope that other people who are also interested in that topic might just join in. For me, it’s about the chance of finding like-minded people with similar interests (whether they agree with me or not!). It’s about connecting with new people, people I may never have met otherwise. It doesn’t matter whether it’s about my passion for analytics, my fascination with my latest gadget, or my displeasure with my latest billing experience; with the growth of the Internet, there’s bound to be people out there thinking and debating about similar things.

And, that’s the clincher - the scale of these social networks can’t be underestimated. A back-of-the-napkin poll I recently did to see how big some of the sites I knew about were stunned me; out of approximately 20 sites myself or my colleagues are a part of, only two had membership levels below 22 million. That may seem like an arbitrary number, but it has quite a bit of significance for me – it’s the population of Australia, the place I live. Sites like Facebook and MySpace have over ten times the population of Australia; These aren’t just social networks, they’re almost countries in their own right!

With that level of membership, it’s not surprising that there’s a wealth of information available within them, information that changes as rapidly as the discussion does. Google Trends and Twitter Trending Topics are great to help see what people are talking about overall, but they’re not personal – they don’t always relate to what I’m interested in. And, trying to trim down the torrent of information is almost an exercise in frustration – applications like TweetDeck help targeted searching and monitoring, but they don’t solve the real problem around pattern extraction and trend analysis.

So, based on the excellent work done by Zach Marshall, one of the geniuses behind our Web Services development, I thought it’d be rather fun to use SAS to create a personalized Twitter search process that takes into account geographic information, language-based searches, and then use Enterprise Guide’s Stored Process capabilities to package it up into an installable process usable by anyone. For me, the exciting thing was how much of SAS’s functionality I was easily able to use in what amounts to such a small effort:

• SAS 9.2’s Web Services capabilities, to connect to Twitter and create the query
• SAS’s Regular Expressions parsing, to cleanse the XML documents and structure them correctly
• SAS’s XML parsing and data handling capabilities, to extract and structure data
• SAS’s Stored Process capabilities, to turn it into a reusable process that’ll deliver the results to anything (a SAS dataset, Excel, Internet Explorer …)
• SAS’s Text Mining capabilities, to extract trends and patterns of particular discussion

I spend a lot of time on planes, so one of the first things I searched for was what people were saying about some of our major airlines over the last seven days, centered approximately 100 kilometers around Sydney (where I live). The breakup was fascinating – for one, the discussion was focused around:

• 4% of all discussions: TV related discussion, namely the Australian anti-censorship video being screened on the airline and various television awards programs
• 41% of all discussions: Discussion about the airline’s lounge, posts of people in-transit and waiting for the flights / going home
• 22% of all discussions: Frequent flier points, the airline’s club, a new joint loyalty reward program
• 23% of all discussions: Work-related discussion and industry issues (e.g. A380, working at the airline)
• 8% of all discussions: Cargo price fixing

The level of interest around their newly launched joint loyalty program must be pretty gratifying for them; it’s pretty clearly a hot topic on Twitter!

For me, it’s a brilliant way to extend my network, monitor the pulse of discussion, and spend more time thinking and debating and less time clicking. For organisations who care about their customers, it might be a way to create a personal, two-way dialogue with all of their customers. Or, it might be a way to help them solve their customers’ issues as they experience and Tweet about them. Or, it might simply be a way of keeping track of what’s hot at the moment, quickly, easily, and dynamically.

In any case, I find it tremendously empowering. It’s not just that I’ve got another way of taming the deluge of information I’m increasing hit with every day; it’s also that I know that if I say something, the odds of someone hearing it who cares about similar things to me increases every day. And, SAS is right there, helping make it easier.

A great thing about working at SAS is that I’m surrounded by smart and creative colleagues like Evan and Zach (and over 10,000 others from across the globe). If you aren’t familiar with SAS, here is a recent write-up in Investor’s Business Daily. May I never have to leave SAS, or ever again have to work at a public company.

Can the results from text mining tweets be of use in forecasting? Like the use of Google Trends data in forecasting (discussed in this blog on July 10), this is an area of active research. While it is exciting to have all these new data sources, it is still to-be-determined whether they can actually improve the accuracy of our forecasts. Are you doing research in this area? If so, I invite you to share your results in a guest blogger posting on The BFD.

For more information on using SAS to analyze Twitter data, and for a sample of Evan’s code, you can contact him directly at evan.stubbs@sas.com.