Rick Wicklin, PhD, is a distinguished researcher in computational statistics at SAS and is a principal developer of SAS/IML software. His areas of expertise include computational statistics, simulation, statistical graphics, and modern methods in statistical data analysis. Rick is author of the books Statistical Programming with SAS/IML Software and Simulating Data with SAS.
Many people have an intuitive feel for residuals in least square models and know that the sum of squared residuals is a goodness-of-fit measure. Generalized linear regression models use a different but related idea, called deviance residuals. What are deviance residuals, and how can you compute them? Deviance residuals (and
A previous article describes how to use SAS to find the inflection points of a 1-D function that you can evaluate at any point. The function must be given by a formula (or by an algorithm) because the root-finding algorithm needs to evaluate the function at arbitrary locations. However, sometimes
A SAS programmer asked if it is possible to numerically find an inflection point for a univariate function, f(x). Yes! This can be solved as a variation of a classic numerical root-finding problem. Recall that an inflection point is a value (call it x0) in the domain where the graph
I previously wrote an article about the Lambert W function. The Lambert W function is the inverse of the function g(x) = x exp(x). This means that you can use it to find the value of x such that g(x)=c for any value of c in the range of g, which
A SAS programmer had many polynomials for which he wanted to compute the real roots. By the Fundamental Theorem of Algebra, every polynomial of degree d has d complex roots. You can find these complex roots by using the POLYROOT function in SAS IML. The programmer only wanted to output
Here's a SAS tip for you. Most SAS programmers know that SAS provides syntax that makes it easy to specify a list of variables. For example, you can use the hyphen and colon operators to specify lists of variables on many SAS statements: You can use the hyphen operator (-)
A colleague asked me an interesting question: Suppose you have a structured correlation matrix, such as a matrix that has a compound symmetric, banded, or an AR1(ρ) structure. If you generate a random correlation matrix that has the same eigenvalues as the structured matrix, does the random matrix have the
In a previous article, I presented some of the most popular blog posts from The DO Loop in 2024. In general, popular articles deal with elementary topics that have broad appeal. However, I also write technical articles about advanced topics, which typically do not make it onto a Top 10
A previous article discusses covariance matrices that have the same set of eigenvalues. The set of eigenvalues is called the spectrum of the matrix. For symmetric matrices, the spectrum contains real numbers. For covariance matrices, which are positive semidefinite, the eigenvalues are nonnegative. It turns out that two symmetric matrices
In 2024, I wrote about 80 articles for The DO Loop blog. My most popular articles were about SAS programming, data visualization, and statistics. If you missed any of these articles, here is the "Reader's Choice Awards" for some of the most popular articles from 2024! SAS Programming The following
A previous article discusses how to generate a random covariance matrix with a specified set of (positive) eigenvalues. A SAS programmer asked whether it is possible to produce a correlation matrix that has a specified set of eigenvalues. After discussing the problem with a friend, I am happy to report
O Christmas tree, O Christmas tree, How lovely are your branches! SAS programmers have a long history of creating yuletide-themed graphics. Christmas trees are a popular image because of their simplicity. I admit that I have indulged more than once in this holiday tradition: An old-school ASCII art image A
While researching the topic of Latin hypercube sampling (LHS), I read an article by Emily Gao (2019) that shows how to use PROC IML in SAS to perform the algorithm. It is possible to simplify Gao's implementation of Latin hypercube sampling in SAS while also making the computation more efficient.
Decades ago, it was a challenge to generate (pseudo-) random numbers that had good statistical properties. The proliferation of desktop computers in the 1980s and '90s led to many advances in computational mathematics, including better ways to generate pseudorandom variates from a wide range of probability distributions. (For brevity, I
The article "Order two-dimensional vectors by using angles" shows how to re-order a set of 2-D vectors by their angles. Because angles are on a circle, which has no beginning and no end, you must specify which vector will appear first in the list. The previous article finds the largest
Order matters. The order of variables in tables and rows of a correlation matrix can make a big difference in how easy it is to observed correlations between variables or groups of variables. There are many ways to order the variables, but this article shows how to display the variables
In a correlation analysis, it is common to consider the correlations between all pairs of numerical variables. That is, if there are k numerical variables, most people examine the complete k x k matrix of correlations. This matrix is symmetric and has 1s on the diagonal, so more than half of the
A previous article discusses the MakeString function, which you can use to convert an IML character vector into a string. This can be very useful. When I originally wrote the MakeString function, I was disappointed that I could not vectorize the computation. Recently, I learned about the COMBL function in
When the SAS Global Forum 2020 conference was cancelled by the global COVID-19 pandemic, I felt sorry for the customers and colleagues who had spent months preparing their presentations. One presentation I especially wanted to attend was by Bucky Ransdell and Randy Tobias: "Introducing PROC SIMSYSTEM for Systematic Nonnormal Simulation".
A previous article shows a simulation of two different models of a foraging animal. The first model is a random walk, which assumes that the animal chooses a random direction, then takes a step that is distributed according to a Gaussian random variable. In the second model, the animal again
In SAS, range attribute maps enable you to specify the range of values that determine the colors used for graphical elements. There are various examples that use the GTL to define a range attribute map, but fewer examples that show how to use a range attribute map with PROC SGPLOT.
A common way to visualize the sample correlations between many numeric variables is to display a heat map that shows the Pearson correlation for each pair of variables, as shown in the image to the right. The correlation is a number in the range [-1, 1], where -1 indicated perfect
The INPUT function and PUT function in SAS are used to apply informats and formats (respectively) to data. For both functions, you must know in advance which informat or format you want to apply. For brevity, let's consider only applying a format. To use the PUT function, you must know
In SAS, the INPUT and PUT functions are powerful functions that enable you to convert data from character type to numeric type and vice versa. They work by applying SAS formats or informats to data. You cannot fully understand the INPUT and PUT functions without understanding formats and informats in
SAS software supports two kinds of procedures: interactive and non-interactive. Most SAS procedures are non-interactive. They begin with a PROC statement, include one or more additional statements, and end with a RUN statement. When SAS encounters the RUN statement, the procedure executes all statements, then exits. On the other hand,
A remarkable result in probability theory is the "three-sigma rule," which is a generic name for theorems that bound the probability that a univariate random variable will appear near the center of its distribution. This article discusses the familiar three-sigma rule for the normal distribution, a less-familiar rule for unimodal
In practice, there is no need to remember textbook formulas for the ANOVA test because all modern statistical software will perform the test for you. In SAS, the ANOVA procedure is designed to handle balanced designs (the same number of observations in each group) whereas the GLM procedure can handle
A previous article about how to display missing values in SAS prompted a comment about special missing values in ODS tables in SAS. Did you know that statistical tables in SAS include special missing values to represent certain situations in statistical analyses? This article explains how to interpret four special
In statistical tables in SAS, a dot (.) represents a numerical missing value. Although a dot is the default symbol in SAS, other languages use other symbols. The R language prints the symbol NA, which stands for "not available." The MATLAB language uses NaN ("Not a Number"). In Python, many
Modern software for statistical graphics automatically handles many details and graph defaults, such as the range of the axes and the placement of tick marks. In the days of yore, these details required tedious manual calculations. Think about what is required to place ticks on a scatter plot. On the
