In univariate data analysis, the median is often used as an alternative to the mean because the mean is sensitive to outliers in the data, whereas the median is a robust statistic. For higher-dimensional data, the mean and the centroid are both used to represent the "center" of a cloud
In some applications, it is useful to permute the rows or the columns of a matrix. A previous article discusses how random permutation of columns (within each row) are useful in constructing permutation tests. This article shows a simpler situation: Permuting the rows of a matrix to change their order.
It is difficult to evaluate high-dimensional integrals. One numerical technique that can be useful is quasi-Monte Carlo integration. In this article, I show how you can generate quasirandom points in SAS and use them to evaluate a definite integral on a compact region. For simplicity, the example in this article
A previous article shows how to convert a positive integer from base 10 to any other arbitrary base. For example, 15 (base 10) = 120 (base 3) because 15 = 1*32 + 2*31 + 0*30. Representing integers is probably familiar to many readers. But did you know that you can
Many data analysts are familiar with logistic regression, where the response variable, Y, has two observed values, often represented as Y=0 and Y=1. The case Y=0 encodes that an event did not happen. For example, a patient did not experience some disease or did not die. The opposite case (Y=1)
A previous article discusses various ways to overlay a density curve on a histogram in SAS. SAS provides several procedures that handle this task for common univariate probability distributions such as normal, lognormal, and gamma. If you define and use a less common distribution, you can write a GTL template
SAS has several procedures that can fit a probability distribution to data, plot a histogram, and overlay one or more density estimates: PROC UNIVARIATE in Base SAS enables you to overlay parametric density curves from about 20 common continuous probability distributions, such as normal, lognormal, and gamma. It also enables
SAS supports more than 25 common probability distributions for the PDF, CDF, QUANTILE, and RAND functions. If you need a less-common distribution, you can implement new distributions by using Base SAS (specifically, PROC FCMP) or the SAS/IML language. On the SAS Support Communities, a SAS programmer asked how to implement
Suppose you measure data weekly. According to the ISO standard, weeks are measured in the range 1-53, where most years have 52 weeks, but occasionally there is a "leap week." (The WEEK function in SAS implements the ISO standard to find the week-of-the-year for any date.) The heat map to
While researching a topic on effect sizes, I learned about a SAS function that is related to noncentrality parameters. I previously wrote an article about the noncentral t distribution, which is one of several well-known distributions that contains an optional noncentrality parameter. I mentioned that the PDF, CDF, and QUANTILE
I have previously written about the moment-ratio diagram as a graphical tool for modeling univariate distributions and also as a tool for examining the distribution of the skewness and kurtosis statistics for distributions. The simple moment-ratio (M-R) diagram from my book (Wicklin, 2013, Simulating Data with SAS, Chapter 16) is
This article shows how to classify a set of high-dimensional data into orthants. An orthant is the d-dimensional generalization of a quadrant. For 2-D Euclidean space, there are four quadrants, often labeled by Roman numerals I-IV. The quadrants are open sets that are defined by the signs of each coordinate
An article by David Corliss in Amstat News (Corliss D. (2025) "Quantifying Diversity: Calculating the Gini-Simpson Diversity Index") discusses a new statistical measure of diversity that was adopted by the US Census Bureau. The statistic is called the Gini-Simpson diversity index. The Census Bureau has published an article about how
A recent article describes the main features of simulation by using the Synthetic Minority Over-sampling Technique (SMOTE). SMOTE was created to oversample from a set of rare events prior to running a machine learning classification algorithm. However, at its heart, the SMOTE algorithm (Chawla et al., 2002) provides a way
In 2024, I wrote about 80 articles for The DO Loop blog. My most popular articles were about SAS programming, data visualization, and statistics. If you missed any of these articles, here is the "Reader's Choice Awards" for some of the most popular articles from 2024! SAS Programming The following
A previous article discusses how to generate a random covariance matrix with a specified set of (positive) eigenvalues. A SAS programmer asked whether it is possible to produce a correlation matrix that has a specified set of eigenvalues. After discussing the problem with a friend, I am happy to report
While researching the topic of Latin hypercube sampling (LHS), I read an article by Emily Gao (2019) that shows how to use PROC IML in SAS to perform the algorithm. It is possible to simplify Gao's implementation of Latin hypercube sampling in SAS while also making the computation more efficient.
The article "Order two-dimensional vectors by using angles" shows how to re-order a set of 2-D vectors by their angles. Because angles are on a circle, which has no beginning and no end, you must specify which vector will appear first in the list. The previous article finds the largest
Order matters. The order of variables in tables and rows of a correlation matrix can make a big difference in how easy it is to observed correlations between variables or groups of variables. There are many ways to order the variables, but this article shows how to display the variables
A previous article discusses the MakeString function, which you can use to convert an IML character vector into a string. This can be very useful. When I originally wrote the MakeString function, I was disappointed that I could not vectorize the computation. Recently, I learned about the COMBL function in
A common way to visualize the sample correlations between many numeric variables is to display a heat map that shows the Pearson correlation for each pair of variables, as shown in the image to the right. The correlation is a number in the range [-1, 1], where -1 indicated perfect
In SAS, DATA step programmers use the IN operator to determine whether a value is contained in a set of target values. Did you know that there is a similar functionality in the SAS IML language? The ELEMENT function in the SAS IML language is similar to the IN operator
A previous article shows how to implement recursive formulas in SAS. The article points out that you can often avoid recursion by using an iterative algorithm, which is more efficient. An example is the Fibonacci sequence, which is usually defined recursively as F(n) = F(n-1) + F(n-2) for n
Many well-known distributions become more and more "normal looking" for large values of a parameter. Famously, the binomial distribution, Binom(p, N), can be approximated by a normal distribution when N (the sample size) is large. Similarly, the Poisson(λ) distribution is well approximated by the normal distribution when λ is large.
There are two programming tools that I rarely use: the SAS macro language and recursion. The SAS macro language is a tool that enables you to generate SAS statements. I rarely use the SAS macro language because the SAS IML language supports all the functionality required to write complex programs,
In SAS, the easiest way to draw random sampling from data is to use PROC SURVEYSELECT or the SAMPLE function in SAS IML software. I have previously written about how to implement four common sampling schemes by using PROC SURVEYSELECT and the SAMPLE function. The DATA step in SAS is
A previous article shows that you can run a simple (one-variable) isotonic regression by using a quadratic programming (QP) formulation. While I was reading a book about computational geometry, I learned that there is a connection between isotonic regression and the convex hull of a certain set of points. Whaaaaat?
Since the pandemic began in 2020, the SAS IML developers have added about 50 new functions and enhancements to the SAS IML language in SAS Viya. Among these functions are new modern methods for optimization that have a simplified syntax as compared to the older 'NLP' functions that are available
One of the most exciting features of SAS Viya Workbench is that the code editor includes a generative AI component called SAS Viya Copilot. This feature was announced and demonstrated at SAS Innovate 2024. With the Copilot, you can specify a text prompt that generates SAS code. For example, you
In a recent article, I graphed the PDF of a few Beta distributions that had a variety of skewness and kurtosis values. I thought that I had chosen the parameter values to represent a wide variety of Beta shapes. However, I was surprised to see that the distributions were all
