Advanced Analytics

Rick WicklinAugust 28, 2017 0

The singular value decomposition: A fundamental technique in multivariate data analysis

The singular value decomposition (SVD) could be called the "billion-dollar algorithm" since it provides the mathematical basis for many modern algorithms in data science, including text mining, recommender systems (think Netflix and Amazon), image processing, and classification problems. Although the SVD was mathematically discovered in the late 1800s, computers have

Read More

Programming Tips

Rick WicklinAugust 23, 2017 0

The arithmetic-geometric mean

All statisticians are familiar with the classical arithmetic mean. Some statisticians are also familiar with the geometric mean. Whereas the arithmetic mean of n numbers is the sum divided by n, the geometric mean of n nonnegative numbers is the n_th root of the product of the numbers. The geometric

Read More

English

Programming Tips

Rick WicklinAugust 21, 2017 0

6 tips for timing the performance of algorithms

When you implement a statistical algorithm in a vector-matrix language such as SAS/IML, R, or MATLAB, you should measure the performance of your implementation, which means that you should time how long a program takes to analyze data of varying sizes and characteristics. There are some general tips that can

Read More

English

Analytics | Data Visualization

Rick WicklinAugust 16, 2017 0

Use a bar chart to visualize pairwise correlations

Visualizing the correlations between variables often provides insight into the relationships between variables. I've previously written about how to use a heat map to visualize a correlation matrix in SAS/IML, and Chris Hemedinger showed how to use Base SAS to visualize correlations between variables. Recently a SAS programmer asked how

Read More

English

Analytics | Learn SAS

Rick WicklinAugust 14, 2017 0

What is rank correlation?

When someone refers to the correlation between two variables, they are probably referring to the Pearson correlation, which is the standard statistic that is taught in elementary statistics courses. Elementary courses do not usually mention that there are other measures of correlation. Why would anyone want a different estimate of

Read More

English

Advanced Analytics

Rick WicklinAugust 9, 2017 0

Robust principal component analysis in SAS

Recently, I was asked whether SAS can perform a principal component analysis (PCA) that is robust to the presence of outliers in the data. A PCA requires a data matrix, an estimate for the center of the data, and an estimate for the variance/covariance of the variables. Classically, these estimates

Read More

English

Programming Tips

Rick WicklinAugust 7, 2017 0

The curse of non-unique eigenvectors

A SAS customer asked, "I computed the eigenvectors of a matrix in SAS and in another software package. I got different answers? How do I know which answer is correct?" I've been asked variations of this question dozens of times. The answer is usually "both answers are correct." The mathematical

Read More

English

Programming Tips

Rick WicklinAugust 2, 2017 0

Dimension reduction: Guidelines for retaining principal components

Last week I blogged about the broken-stick problem in probability, which reminded me that the broken-stick model is one of the many techniques that have been proposed for choosing the number of principal components to retain during a principal component analysis. Recall that for a principal component analysis (PCA) of

Read More

English

Learn SAS | Programming Tips

Rick WicklinJuly 31, 2017 0

Flip it. Flip it good.

A SAS user needed to convert a program from MATLAB into the SAS/IML matrix language and asked whether there is a SAS/IML equivalent to the fliplr and flipud functions in MATLAB. These functions flip the columns or rows (respectively) of a matrix; "LR" stands for "left-right" and "UD" stands for

Read More

English

Programming Tips

Rick WicklinJuly 26, 2017 0

Random segments and broken sticks

A classical problem in elementary probability asks for the expected lengths of line segments that result from randomly selecting k points along a segment of unit length. It is both fun and instructive to simulate such problems. This article uses simulation in the SAS/IML language to estimate solutions to the

Read More

English

Programming Tips

Rick WicklinJuly 24, 2017 0

Difference operators as matrices

For a time series { y1, y2, ..., yN }, the difference operator computes the difference between two observations. The kth-order difference is the series { yk+1 - y1, ..., yN - yN-k }. In SAS, the DIF function in the DATA step computes differences between observations. The DIF function

Read More

English

Analytics

Rick WicklinJuly 19, 2017 0

A quantile definition for skewness

Skewness is a measure of the asymmetry of a univariate distribution. I have previously shown how to compute the skewness for data distributions in SAS. The previous article computes Pearson's definition of skewness, which is based on the standardized third central moment of the data. Moment-based statistics are sensitive to

Read More

English

Advanced Analytics | Data Visualization

Rick WicklinJuly 17, 2017 0

3 ways to visualize prediction regions for classification problems

An important problem in machine learning is the "classification problem." In this supervised learning problem, you build a statistical model that predicts a set of categorical outcomes (responses) based on a set of input features (explanatory variables). You do this by training the model on data for which the outcomes

Read More

English

Advanced Analytics | Programming Tips

Rick WicklinJuly 12, 2017 0

The bias-corrected and accelerated (BCa) bootstrap interval

I recently showed how to compute a bootstrap percentile confidence interval in SAS. The percentile interval is a simple "first-order" interval that is formed from quantiles of the bootstrap distribution. However, it has two limitations. First, it does not use the estimate for the original data; it is based only

Read More

English

Programming Tips

Rick WicklinJuly 10, 2017 0

Bootstrap estimates in SAS/IML

I previously wrote about how to compute a bootstrap confidence interval in Base SAS. As a reminder, the bootstrap method consists of the following steps: Compute the statistic of interest for the original data Resample B times from the data to form B bootstrap samples. B is usually a large

Read More

English

Analytics | Programming Tips

Rick WicklinJuly 5, 2017 0

Test for the equality of two proportions in SAS

A SAS customer asked how to use SAS to conduct a Z test for the equality of two proportions. He was directed to the SAS Usage Note "Testing the equality of two or more proportions from independent samples." The note says to "specify the CHISQ option in the TABLES statement

Read More

English

Programming Tips

Rick WicklinJuly 3, 2017 0

Summary statistics and t tests in SAS

Students in introductory statistics courses often use summary statistics (such as sample size, mean, and standard deviation) to test hypotheses and to compute confidence intervals. Did you know that you can provide summary statistics (rather than raw data) to PROC TTEST in SAS and obtain hypothesis tests and confidence intervals?

Read More

English

Programming Tips

Rick WicklinJune 28, 2017 0

The average bootstrap sample omits 36.8% of the data

Suppose you roll six identical six-sided dice. Chance are that you will see at least one repeated number. The probability that you will see six unique numbers is very small: only 6! / 6^6 ≈ 0.015. This example can be generalized. If you draw a random sample with replacement from

Read More

English

Programming Tips

Rick WicklinJune 26, 2017 0

Video: Create and use lists and tables in SAS/IML

My presentation at SAS Global Forum 2017 was "More Than Matrices: SAS/IML Software Supports New Data Structures." The paper was published in the conference proceedings several months ago, but I recently recorded a short video that gives an overview of using the new data structures in SAS/IML 14.2: If your

Read More

English

Advanced Analytics | Learn SAS | Programming Tips

Rick WicklinJune 21, 2017 0

Jackknife estimates in SAS

One way to assess the precision of a statistic (a point estimate) is to compute the standard error, which is the standard deviation of the statistic's sampling distribution. A relatively large standard error indicates that the point estimate should be viewed with skepticism, either because the sample size is small

Read More

English

Analytics

Rick WicklinJune 19, 2017 0

How to find a feasible point for a constrained optimization in SAS

Most numerical optimization routines require that the user provides an initial guess for the solution. I have previously described a method for choosing an initial guess for an optimization, which works well for low-dimensional optimization problems. Recently a SAS programmer asked how to find an initial guess when there are

Read More

English

Analytics | Programming Tips

Rick WicklinJune 14, 2017 0

Two ways to compute maximum likelihood estimates in SAS

In a previous article, I showed two ways to define a log-likelihood function in SAS. This article shows two ways to compute maximum likelihood estimates (MLEs) in SAS: the nonlinear optimization subroutines in SAS/IML and the NLMIXED procedure in SAS/STAT. To illustrate these methods, I will use the same data

Read More

English

Analytics | Programming Tips

Rick WicklinJune 12, 2017 0

Two simple ways to construct a log-likelihood function in SAS

Maximum likelihood estimation (MLE) is a powerful statistical technique that uses optimization techniques to fit parametric models. The technique finds the parameters that are "most likely" to have produced the observed data. SAS provides many tools for nonlinear optimization, so often the hardest part of maximum likelihood is writing down

Read More

English

Programming Tips

Rick WicklinJune 7, 2017 0

The IFN function versus the IF-THEN/ELSE statement in SAS

I have previously discussed how to define functions that safely evaluate their arguments and return a missing value if the argument is not in the domain of the function. The canonical example is the LOG function, which is defined only for positive arguments. For example, to evaluate the LOG function

Read More

English

Programming Tips

Rick WicklinJune 5, 2017 0

Runs in coin tosses; patterns in random seating

If you toss a coin 28 times, you would not be surprised to see three heads in a row, such as ...THHHTH.... But what about eight heads in a row? Would a sequence such as THHHHHHHHTH... be a rare event? This question popped into my head last weekend as I

Read More

English

Learn SAS | Programming Tips

Rick WicklinJune 1, 2017 0

How to choose a seed for generating random numbers in SAS

Last week I was asked a simple question: "How do I choose a seed for the random number functions in SAS?" The answer might surprise you: use any seed you like. Each seed of a well-designed random number generator is likely to give rise to a stream of random numbers,

Read More

English

Programming Tips

Rick WicklinMay 30, 2017 0

On the SMOOTHCONNECT option in the SERIES statement

By default, when you use the SERIES statement in PROC SGPLOT to create a line plot, the observations are connected (in order) by straight line segments. However, SAS 9.4m1 introduced the SMOOTHCONNECT option which, as the name implies, uses a smooth curve to connect the observations. In Sanjay Matange's blog,

Read More

English

Analytics | Data Visualization

Rick WicklinMay 24, 2017 0

Sample quantiles: A comparison of 9 definitions

According to Hyndman and Fan ("Sample Quantiles in Statistical Packages," TAS, 1996), there are nine definitions of sample quantiles that commonly appear in statistical software packages. Hyndman and Fan identify three definitions that are based on rounding and six methods that are based on linear interpolation. This blog post shows

Read More

English

Analytics | Learn SAS

Rick WicklinMay 22, 2017 0

Quantile definitions in SAS

In last week's article about the Flint water crisis, I computed the 90th percentile of a small data set. Although I didn't mention it, the value that I reported is different from the the 90th percentile that is reported in Significance magazine. That is not unusual. The data only had

Read More

English

Analytics | Data Visualization

Rick WicklinMay 17, 2017 0

Quantiles and the Flint water crisis

The April 2017 issue of Significance magazine features a cover story by Robert Langkjaer-Bain about the Flint (Michigan) water crisis. For those who don't know, the Flint water crisis started in 2014 when the impoverished city began using the Flint River as a source of city water. The water was

Read More

English