Rick Wicklin, PhD, is a distinguished researcher in computational statistics at SAS and is a principal developer of SAS/IML software. His areas of expertise include computational statistics, simulation, statistical graphics, and modern methods in statistical data analysis. Rick is author of the books Statistical Programming with SAS/IML Software and Simulating Data with SAS.
A recent article describes the main features of simulation by using the Synthetic Minority Over-sampling Technique (SMOTE). SMOTE was created to oversample from a set of rare events prior to running a machine learning classification algorithm. However, at its heart, the SMOTE algorithm (Chawla et al., 2002) provides a way
The Synthetic Minority Over-sampling Technique (SMOTE) was created to address class-imbalance problems in machine learning algorithms. The idea is to oversample from the rare events prior to running a machine learning classification algorithm. However, at its heart, the SMOTE algorithm (Chawla et al., 2002) is essentially a way to simulate
SAS programmers love to brag that the SAS will still run a program they wrote twenty or forty years. This is both a blessing and a curse. It's a blessing because it frees the statistical programmer from needing to revisit and rewrite code that was written long ago. It's a
Isaac Newton had many amazing scientific and mathematical accomplishments. His law of universal gravitation and his creation of calculus are at the top of the list! But in the field of numerical analysis, "Newton's Method" was a groundbreaking advancement for solving for a root of a nonlinear smooth function. The
Newton's method was in the news this week. Not the well-known linear method for finding roots, but a more complicated method for finding minima, sometimes called the method of successive parabolic approximations. Newton's parabolic method was recently improved by modern researchers who extended the method to use higher-dimensional polynomials. The
Nearly every statistician has heard the aphorism, "All models are wrong, but some are useful." The quote is attributed to George Box, an early and influential thinker about statistics. Did George Box actually say this quote? Yes, he did. The first part of the quote ("All models are wrong") appeared
After giving a talk about how to create effective statistical graphics in SAS, I was asked a question: "When do you suggest using the graph template language (GTL) to build graphs?" I replied that I turn to the GTL when I cannot create the graph I want by using PROC
A common task in statistics is to model data by using a parametric probability distribution, such as the normal, lognormal, beta, or gamma distributions. There are many ways to assess how well the model fits the data, including graphical methods such as a Q-Q plot and formal statistical tests such
Every programmer makes errors. Therefore, learning to debug a program is an important part of learning to program. Another skill is learning to decipher cryptic error messages, which can be as hard to interpret as hieroglyphs. One helpful skill is learning to navigate a "traceback" error. A traceback error message
Happy Pi Day! Every year on March 14th (written 3/14 in the US), people in the mathematical sciences celebrate "all things pi-related" because 3.14 is the three-decimal approximation to π ≈ 3.14159265358979.... The purpose of this day is to have fun, celebrate the importance of mathematics, and maybe learn a
The first-order autoregressive (AR(1)) correlation structure is important for applications in time series modeling and for repeated measures analysis. The AR(1) model provides a simple situations where measurements (on the same subject) that are closer in time are correlated more strongly than measurements recorded far apart. The AR(1) model uses
In a binomial regression model, the response variable is the proportion of successes for a given number of trials. In SAS regression procedures, you specify a binomial model by using the EVENTS/TRIALS syntax on the MODEL statement. Many analysts use the LOGISTIC or GENMOD procedures to fit binomial models. Visualizing
Many people have an intuitive feel for residuals in least square models and know that the sum of squared residuals is a goodness-of-fit measure. Generalized linear regression models use a different but related idea, called deviance residuals. What are deviance residuals, and how can you compute them? Deviance residuals (and
A previous article describes how to use SAS to find the inflection points of a 1-D function that you can evaluate at any point. The function must be given by a formula (or by an algorithm) because the root-finding algorithm needs to evaluate the function at arbitrary locations. However, sometimes
A SAS programmer asked if it is possible to numerically find an inflection point for a univariate function, f(x). Yes! This can be solved as a variation of a classic numerical root-finding problem. Recall that an inflection point is a value (call it x0) in the domain where the graph
I previously wrote an article about the Lambert W function. The Lambert W function is the inverse of the function g(x) = x exp(x). This means that you can use it to find the value of x such that g(x)=c for any value of c in the range of g, which
A SAS programmer had many polynomials for which he wanted to compute the real roots. By the Fundamental Theorem of Algebra, every polynomial of degree d has d complex roots. You can find these complex roots by using the POLYROOT function in SAS IML. The programmer only wanted to output
Here's a SAS tip for you. Most SAS programmers know that SAS provides syntax that makes it easy to specify a list of variables. For example, you can use the hyphen and colon operators to specify lists of variables on many SAS statements: You can use the hyphen operator (-)
A colleague asked me an interesting question: Suppose you have a structured correlation matrix, such as a matrix that has a compound symmetric, banded, or an AR1(ρ) structure. If you generate a random correlation matrix that has the same eigenvalues as the structured matrix, does the random matrix have the
In a previous article, I presented some of the most popular blog posts from The DO Loop in 2024. In general, popular articles deal with elementary topics that have broad appeal. However, I also write technical articles about advanced topics, which typically do not make it onto a Top 10
A previous article discusses covariance matrices that have the same set of eigenvalues. The set of eigenvalues is called the spectrum of the matrix. For symmetric matrices, the spectrum contains real numbers. For covariance matrices, which are positive semidefinite, the eigenvalues are nonnegative. It turns out that two symmetric matrices
In 2024, I wrote about 80 articles for The DO Loop blog. My most popular articles were about SAS programming, data visualization, and statistics. If you missed any of these articles, here is the "Reader's Choice Awards" for some of the most popular articles from 2024! SAS Programming The following
A previous article discusses how to generate a random covariance matrix with a specified set of (positive) eigenvalues. A SAS programmer asked whether it is possible to produce a correlation matrix that has a specified set of eigenvalues. After discussing the problem with a friend, I am happy to report
O Christmas tree, O Christmas tree, How lovely are your branches! SAS programmers have a long history of creating yuletide-themed graphics. Christmas trees are a popular image because of their simplicity. I admit that I have indulged more than once in this holiday tradition: An old-school ASCII art image A
While researching the topic of Latin hypercube sampling (LHS), I read an article by Emily Gao (2019) that shows how to use PROC IML in SAS to perform the algorithm. It is possible to simplify Gao's implementation of Latin hypercube sampling in SAS while also making the computation more efficient.
Decades ago, it was a challenge to generate (pseudo-) random numbers that had good statistical properties. The proliferation of desktop computers in the 1980s and '90s led to many advances in computational mathematics, including better ways to generate pseudorandom variates from a wide range of probability distributions. (For brevity, I
The article "Order two-dimensional vectors by using angles" shows how to re-order a set of 2-D vectors by their angles. Because angles are on a circle, which has no beginning and no end, you must specify which vector will appear first in the list. The previous article finds the largest
Order matters. The order of variables in tables and rows of a correlation matrix can make a big difference in how easy it is to observed correlations between variables or groups of variables. There are many ways to order the variables, but this article shows how to display the variables
In a correlation analysis, it is common to consider the correlations between all pairs of numerical variables. That is, if there are k numerical variables, most people examine the complete k x k matrix of correlations. This matrix is symmetric and has 1s on the diagonal, so more than half of the
A previous article discusses the MakeString function, which you can use to convert an IML character vector into a string. This can be very useful. When I originally wrote the MakeString function, I was disappointed that I could not vectorize the computation. Recently, I learned about the COMBL function in
