Last week I showed how to use simulation to estimate the power of a statistical test. I used the two-sample t test to illustrate the technique. In my example, the difference between the means of two groups was 1.2, and the simulation estimated a probability of 0.72 that the t test would detect the difference.
Of course, the value 1.2 was completely arbitrary. You could just as well ask for the probability of detecting a difference of size 0.5, 1.0, or 2.0. You can repeat the simulation for any value of the difference and thereby construct a power curve. That's what I do in this article, which is based on material in Chapter 5 of Simulating Data with SAS.
Here is the result of running the previous simulation for differences of size 0, 0.1, 0.2, ..., 2.0. The points along the curve are power estimates from the simulation, along with 95% confidence intervals. The size of the error bars reflects the number of simulations used for each point estimate, which is 5,000 for this simulation. The underlying curve is the result of running PROC POWER, which can solve this problem exactly. For more complicated statistical tests, an exact power curve is not available and simulation is the only way to estimate power.
The simulation takes about 30 seconds to run on a desktop PC from 2010. You can download the complete SAS program that runs the simulation. The program is virtually identical to the program I discussed last week, except that there is now a loop over the parameter that controls the mean difference (do Delta = 0 to 2 by 0.1), and I have included the Delta parameter in two BY statements.
A few comments about the power curve:
- In this simulation, the two populations have unit standard deviations. The graph shows that when the true difference between the means is 0.5, there is a 20% chance of detecting the difference with such a small sample size (n1=n2=10). The chance rises to 56% when the difference is 1. To have an 80% chance, the means must differ by about 1.35.
- When there is no difference between the population means, the graph shows that there is a 5% chance that the t test will reject the null hypothesis (which is that the means are equal). This is consistent with the fact that the test is performed at the α = 5% significance level.
- Notice that the confidence intervals for the power are smaller when the curve is flatter and larger when the curve is steeper. You can exploit this to make your simulations run faster by using fewer simulation trials in certain parameter ranges. See the Chapter 6, "Strategies for Efficient and Effective Simulation," of Simulating Data with SAS.
Download the program and experiment with changing the parameters. Change NumSamples to 1000 to make your experiments run faster. What does the curve look like when (n1=n2=20)?