Last week I blogged about how to construct a smoother for a time series for the temperature in Albany, NY from 1995 to March, 2012. I smoothed the data by "folding" the time series into a single "year" that contains repeated measurements for each day of the year. Experts in
The birthday matching problem is a classic problem in probability theory. The part of it that people tend to remember is that in a room of 23 people, there is greater than 50% chance that two people in the room share a birthday. But the birthday matching problem is also
The first line of this week's SAS tip grabs your attention, "PROC SQL provides a helpful (though potentially dangerous) tool in the form of the DESCRIBE TABLE statement." SAS author, consultant, and member of the SAS-L Hall of Fame Howard Schreier included this intriguing statement in his book PROC SQL by Example: Using
The INFORMS Conference on Business Analytics and Operations Research kicks off April 15 in Huntington Beach, CA. I had a chance to preview a presentation by Glenn Bailey, Sr. Director of Operations Research at Manheim (the $3B wholesaler auto auctioneer). Glenn's talk is on "The Need for Speed: Responsive Predictive Analytics,"
In yesterday's post, I discussed a "quick and dirty" method to smooth periodic data. However, after I smoothed the data I remarked that the smoother itself was not exactly periodic. At the end points of the periodic interval, the smoother did not have equal slopes and the method does not
Over at the SAS and R blog, Ken Kleinman discussed using polar coordinates to plot time series data for multiple years. The time series plot was reproduced in SAS by my colleague Robert Allison. The idea of plotting periodic data on a circle is not new. In fact it goes
Over at the SAS Discussion Forums, someone asked how to use SAS to fit a Poisson distribution to data. The questioner asked how to fit the distribution but also how to overlay the fitted density on the data and to create a quantile-quantile (Q-Q) plot. The questioner mentioned that the
In September 2010, I questioned whether you should care about native 64-bit client applications (or the lack thereof). At the time, SAS did not have a 64-bit version of SAS Enterprise Guide or SAS Add-In for Microsoft Office. A skeptical reader might assume that I was just trying to make
Locating missing values is important in statistical data analysis. I've previously written about how to count the number of missing values for each variable in a data set. In Base SAS, I showed how to use the MEANS or FREQ procedures to count missing values. In the SAS/IML language, I
Today, SAS announces a new addition to its global certification program: you can now become "SAS Versions"-certified. Over the past 36 years, SAS has delivered hundreds of software products to the business analytics marketplace. Each of those products has a version number. As new revisions are released, the combination of
A 23-year Harvard and Columbia University study was recently published shedding new light on the long-term impacts of teachers with both high and low value-added estimates. Researchers Chetty, Friedman, and Rockoff tracked math and reading assessment data on over 2.5 million students from 1989-2009. They then incorporated 90% of these
Let's end 2012-Q1 with a graphic editorial comment: Forecast Accuracy vs. Effort Using a naïve model will achieve a certain level of forecast accuracy. That accuracy may be high if the demand is smooth and stable, or low if the demand is erratic. But you achieve this level of accuracy with virtually no
In a previous post I showed how to implement Stewart's (1980) algorithm for generating random orthogonal matrices in SAS/IML software. By using the algorithm, it is easy to generate a random matrix that contains a specified set of eigenvalues. If D = diag(λ1, ..., λp) is a diagonal matrix and
The March 28 edition of APICS extra features an article by Fred Tolbert on "The Seven Deadly Sins of Sales Forecasting." Although I have some objection to his Deadly Sin #1: Using Shipment History (and will discuss the objection in a forthcoming guest-post on the Institute of Business Forecasting blog),
Even Rod Serling recognized that sometimes we can't forecast worth a darn. "The Rip Van Winkle Caper" is an episode from Season 2 of the television series, The Twilight Zone, and first aired in 1961. It involves four train robbers who steal a million dollars worth of gold bars, hide
Because I am writing a new book about simulating data in SAS, I have been doing a lot of reading and research about how to simulate various quantities. Random integers? Check! Random univariate samples? Check! Random multivariate samples? Check! Recently I've been researching how to generate random matrices. I've blogged
With the publication of JSL Companion: Applications of the JMP Scripting Language, by Theresa Utlaut, Georgia Morgan, and Kevin Anderson, novice scripters now have a resource that helps them go beyond the basics of the JMP Scripting Language (JSL). Why JSL? The authors have the answers: 1. Easy to start
SAS Global Forum 2012 is right around the corner. If you will be in Orlando, too, be sure to say hello! If you have ideas for improving SAS/IML software or you would like to discuss my blog, please visit me during my hours at the SAS/IML booth in the Demo
The fundamental units in the SAS/IML language are matrices and vectors. Consequently, you might wonder about conditional expression such as if v>0 then.... What does this expression mean when v contains more than a single element? Evaluating vector expressions When you test a vector for some condition, expressions like v>0
Did that set off a trigger for you? It did for my SAS SQL 1: Essentials class, packed with SQL and SAS programmers alike. To clarify matters I pulled up some examples to help get the differences quickly. Set operators and Joins are similar in that they both combine multiple
Earlier this week I described a common programming pattern in the SAS macro language. The pattern sets up a loop for processing each distinct value of a classification variable. The program uses the PROC SQL SELECT INTO feature to populate SAS macro variables. The effect: you can roll your own
As public safety officials leaf through their favorite criminal justice periodical they are greeted with pages and pages of analytics advertisements. These ads are laden with promises of robust and scalable solutions, improved efficiencies and, yes, the promise of prediction. While reading the advertisements, the mental conversation may go something
After my post on detecting outliers in multivariate data in SAS by using the MCD method, Peter Flom commented "when there are a bunch of dimensions, every data point is an outlier" and remarked on the curse of dimensionality. What he meant is that most points in a high-dimensional cloud
Covariance, correlation, and distance matrices are a few examples of symmetric matrices that are frequently encountered in statistics. When you create a symmetric matrix, you only need to specify the lower triangular portion of the matrix. The VECH and SQRVECH functions, which were introduced in SAS/IML 9.3, are two functions
How to write a SAS macro program to repeat your SAS processing for each value of a BY grouping variable.
Art Carpenter’s newest book, Carpenter’s Guide to Innovative SAS Techniques, offers advanced SAS programmers an all-in-one programming reference that includes advanced topics not easily found outside the depths of SAS documentation or more advanced training classes. No matter how you approach the use of SAS software, the techniques provided in
The SAS/IML language supports both row vectors and column vectors. This is useful for performing linear algebra, but it can cause headaches when you are writing a SAS/IML module. I want my modules to be able to handle both row vectors and column vectors. I don't want the user to
When the data is classified by multiple class variables, you can certainly create graphs using BY variables. This results in separate graphs, one for each level of the BY variable crossings. Each graph is scaled by its own data subset, and comparisons across BY levels is harder. When comparisons need to be
When you are constantly taking the data tables and completing joins to begin working on your reports or analysis it might be time to consider creating permanent views. Then you can just add the view to the Enterprise Guide project rather than dealing with the joins in a Query Builder
A recent discussion on the SAS-L discussion forum concerned how to implement linear interpolation in SAS. Some people suggested using PROC EXPAND in SAS/ETS software, whereas others proposed a DATA step solution. For me, the SAS/IML language provides a natural programming environment to implement an interpolation scheme. It also provides
