Rick Wicklin, PhD, is a distinguished researcher in computational statistics at SAS and is a principal developer of SAS/IML software. His areas of expertise include computational statistics, simulation, statistical graphics, and modern methods in statistical data analysis. Rick is author of the books Statistical Programming with SAS/IML Software and Simulating Data with SAS.
Last week Chris Hemedinger posted an article about spam that is sent to SAS blogs and discussed how anti-spam software helps to block spam. No algorithm can be 100% accurate at distinguishing spam from valid comments because of the inherent trade-off between specificity and sensitivity in any statistical test. Therefore,
SAS programmers are probably familiar with how SAS stores a character variable in a data set, but how is a character vector stored in the SAS/IML language? Recall that a character variable is stored by using a fixed-width storage structure. In the SAS DATA step, the maximum number of characters
While at a conference recently, I was asked whether it was possible to use SAS to simulate data from an inverse gamma distribution. The SAS customer had looked at the documentation for the RAND function and did not see "inverse gamma" listed among the possible choices. The answer is "yes."
Dear Rick, I am trying to create a numerical matrix with 100,000 rows and columns in PROC IML. I get the following error: (execution) Unable to allocate sufficient memory. Can IML allocate a matrix of this size? What is wrong? Several times a month I see a variation of this
Just one last short article about properties of the Hilbert matrix. I've already blogged about how to construct a Hilbert matrix in the SAS/IML language and how to compute a formula for the determinant. One reason that the Hilbert matrix is a famous (some would say infamous!) example in numerical
I have previously written about the scope of local and global variables in the SAS/IML language. You might wonder whether SAS/IML modules can also have local scope. The answer is no. All SAS/IML modules are known globally and can be called by any other modules. Some object-oriented programming languages support
Last week I described the Hilbert matrix of size n, which is a famous square matrix in numerical linear algebra. It is famous partially because its inverse and its determinant have explicit formulas (that is, we know them exactly), but mainly because the matrix is ill-conditioned for moderate values of
I enjoy blogging about new functionality in the SAS/IML language because I can go into more depth and provide more complicated examples than the SAS/IML documentation. Today's article is a summary of all of my posts about features that were added to SAS/IML 12.1, which shipped in August 2012 as
Yesterday I blogged about the Hilbert matrix. The (i,j)th element of the Hilbert matrix has the value 1 / (i+j-1), which is the reciprocal of an integer. However, the printed Hilbert matrix did not look exactly like the formula because the elements print as finite-precision decimals. For example, the last
The Hilbert matrix is the most famous ill-conditioned matrix in numerical linear algebra. It is often used in matrix computations to illustrate problems that arise when you compute with ill-conditioned matrices. The Hilbert matrix is symmetric and positive definite, properties that are often associated with "nice" and "tame" matrices. The
I enjoy reading the Graphically Speaking blog because it teaches me a lot about ODS statistical graphics, especially features of the SGPLOT procedure and the Graph Template Language (GTL). Yesterday Sanjay blogged about how to construct a stacked bar chart of percentages so that each bar represents 100%. His chart
Did you know that SAS/IML 12.1 provides built-in functions that compute the norm of a vector or matrix? A vector norm enables you to compute the length of a vector or the distance between two vectors in SAS. Matrix norms are used in numerical linear algebra to estimate the condition
While at SAS Global Forum 2014 I attended a talk by Jorge G. Morel on the analysis of data with overdispersion. (His slides are available, along with a video of his presentation.) The Wikipedia defines overdispersion as "greater variability than expected from a simple model." For count data, the "simple
When spontaneous applause broke out during Dr. Jim Goodnight's presentation at the opening session of SAS Global Forum 2014, I was one of the people cheering the loudest. The SAS CEO had just announced free software for students and professors at universities around the world. The SAS University Edition will
SAS Global Forum 2014, included a meetup of SAS users who are active in various online communities. During the meetup I was struck by the tremendous opportunities that these communities provide. All year long, the online communities demonstrate the conference theme: "the potential of one, the power of all." This
When I visualize three-dimensional data, I prefer to use interactive graphics. For example, I often use the rotating plot in SAS/IML Studio (shown at the left) to create a three-dimensional scatter plot. The interactive plot enables me to rotate the cloud of points and to use a pointer to select
SAS/IML 13.1 shipped a few months ago. I was asked to produce a video that highlights some of the new features in SAS/IML 13.1. In this video I describe several changes to the language before introducing the new built-in subroutines that create ODS statistical graphs. If your browser does not
Last month I blogged about defining SAS/IML functions that have default parameter values. This language feature, which was introduced in SAS/IML 12.1, enables you to skip arguments when you call a user-defined function. The same technique enables you to define optional parameters. Inside the function, you can determine whether the
Once again I'll be at SAS Global Forum this year. The 2014 location is Washington, D. C., so I am looking forward to greeting many friends in the government and consulting sectors. I always enjoy talking with SAS customers about statistics, simulations, matrix computations, and the SAS/IML product, so here's
The SAS/IML language has several functions for finding the unions, intersections, and differences between sets. In fact, two of my favorite utility functions are the UNIQUE function, which returns the unique elements in a matrix, and the SETDIF function, which returns the elements that are in one vector and not
Many geeky mathematical people celebrate "pi day" on March 14, because the date is written 3/14 in the US, which is evocative of the decimal representation of π = 3.14.... Most people are familiar with the decimal representation of π. The media occasionally reports on a new computational tour-de-force that
Last week I showed how to find parameters that maximize the integral of a certain probability density function (PDF). Because the function was a PDF, I could evaluate the integral by calling the CDF function in SAS. (Recall that the cumulative distribution function (CDF) is the integral of a PDF.)
On most Mondays I blog about a function, programming technique, or resource that is useful for programmers who are getting started with SAS software. Recently I learned that my colleagues in the SAS education division have been hard at work developing a series of short videos that explain basic tasks
SAS programmers use the SAS/IML language for many different tasks. One important task is computing an integral. Another is optimizing functions, such as maximizing a likelihood function to find parameters that best fit a set of data. Last week I saw an interesting problem that combines these two important tasks.
A colleague sent me an interesting question: What is the best way to abort a SAS/IML program? For example, you might want to abort a program if the data is singular or does not contain a sufficient number of observations or variables. As a first attempt would be to try
My last blog post described three ways to add a smoothing spline to a scatter plot in SAS. I ended the post with a cautionary note: From a statistical point of view, the smoothing spline is less than ideal because the smoothing parameter must be chosen manually by the user.
Like many SAS programmers, I use the Statistical Graphics (SG) procedures to graph my data in SAS. To me, the SGPLOT and SGRENDER procedures are powerful, easy to use, and produce fabulous ODS graphics. I was therefore surprised when a SAS customer told me that he continues to use the
My previous post described how to use the "missing response trick" to score a regression model. As I said in that article, there are other ways to score a regression model. This article describes using the SCORE procedure, a SCORE statement, the relatively new PLM procedure, and the CODE statement.
A fundamental operation in statistical data analysis is to fit a statistical regression model on one set of data and then evaluate the model on another set of data. The act of evaluating the model on the second set of data is called scoring. One of first "tricks" that I
Although I currently work as a statistician, my original training was in mathematics. In many mathematical fields there is a result that is so profound that it earns the name "The Fundamental Theorem of [Topic Area]." A fundamental theorem is a deep (often surprising) result that connects two or more
