Before we begin, I want to thank all of you readers who supported my righteous appeal for “THE BFD” license plate from the state of North Carolina. I am sad to report, however, that our efforts were in vain and the appeal was denied by Censorship Board of the Division of Motor Vehicles. Along with the denial, these arbiters of good taste were kind enough to send me an application to re-submit another idea – any suggestions? (Note: I’ve already given one new suggestion to the DMV, but they responded back that it was anatomically impossible.)

Now back to the business of forecasting…

Just how confident can we be about the future? This is an important consideration that plays into the decisions we make based on our forecasts. Makridakis, et al, discuss this in Forecasting Methods and Applications:

“It is usually desirable to provide not only forecast values but accompanying uncertainty statements, usually in the form of prediction intervals. This is useful because it provides the user of the forecasts with “worst” or “best” case estimates and with a sense of how dependable the forecast is, and because it protects the forecaster from the criticism that the forecasts are “wrong.” Forecasts cannot be expected to be perfect and intervals emphasize this.” (p. 52)

Not all forecasting software provides prediction intervals, but SAS High-Performance Forecasting (the “engine” inside SAS Forecast Server software) does. The width of the prediction interval (sometimes referred to as the confidence limits or confidence bounds) indicates how close the future is likely to be to the forecasted value. (Confidence bounds are shown by the shaded blue area on the far right of this plot.)

When the confidence bounds are wide, forecasters sometimes whine and complain that they are useless – what good is a forecast in which we can have no confidence? But what is there to do in this situation? My colleague Bob Lucas provides some thoughts on this matter.

Guest Blogger: Bob Lucas, SAS Education & Training

I have taught forecasting for 12 years here at SAS and many a student is often disappointed in the widths of the forecast confidence bounds. They have said, “What good are the confidence bounds that are so wide?” Here is my typical response:

What affects the widths of the confidence bounds?

1. The amount of data you have. (You may or may not have control over that.)
2. The model that you are using. (You do have control over that, however, with limited data you may not have too many choices.)
3. The natural variability in the data. (You have no control over that!!!)

If your data has a lot of variability, guess what – you’re going to get wide confidence bounds. Some customers have unrealistic expectations for the accuracy of their forecast.

Let’s examine the one aspect that you do have control over – the model that you select to produce the forecast. One of the standard premises of forecasting (actually prediction in general) is simpler models do better – all else being the same. While a more complex model (with more parameters) will be able to “fit the history” better than a simpler model, it will likely do no better at forecasting the future.

If you do not use a holdout sample, fits statistics that ignore the number of parameters will favor models with more parameters. Consequently, I favor using SBC to choose my forecast models when not using holdout data. (The Schwarz Bayesian Criterion, also called the Bayesian Information Criterion (BIC), is a criterion for selecting a model from among a class of models with different numbers of parameters.) I still report MAPE to business user, however.

One way to restrict forecast and confidence bounds is to fit the log of the series. SAS Forecast Server will transform back to the original scale for you. This disadvantage is that now the parameter estimates are on the log scale and complicate interpretation if desired.

Now my answer to the question “What use are the confidence bounds that are so wide?”

Don’t you think you would make a different business decision of your forecast was 100 +/- 10 versus 100 +/- 50? How does this relate to your situation? The inventory cost for a fixed service level will be much higher for a series with high volatility than for one with low volatility. The realistic way to handle this is to have different service levels for different products based on the volatility of the series.

As Bob is pointing out, when you have highly volatile demand for a product, you probably aren’t going to be able to forecast it as accurately, and will have wider confidence bounds around the forecast. As a result, you are going to need correspondingly (and perhaps prohibitively expensive) amounts of inventory available to meet a fixed service level target. What an organization needs to do, rather than set a fixed service level (such as “98% order fill rate”) across all products, is to set lower service level objectives for the more difficult to forecast products. If a certain service level (like 98%) is absolutely necessary, such as due to contractual obligations with the customer, you need to address the problem in an alternative manner, such as by finding ways to reduce the volatility (and “unforecastability”) of the demand patterns.

[For further discussion on ways to reduce the variability of demand, tune in to my upcoming webinar “What Management Must Know About Forecasting” (October 21, 1:00pm ET). This is part of the SAS Applying Business Analytics Webinar Series hosted by my colleague Gaurav Verma.]

In summary, if your software does not produce confidence bounds along with its forecasts, you are missing out on some very useful information. There is uncertainty in any forecast, but having an understanding of this uncertainty and an appreciation of the risk helps your organization avoid the really bad business decisions.