Did you know that the FREQ procedure in SAS can compute exact p-values for more than 20 statistical tests and statistics that are associated with contingency table? Mamma mia! That's a veritable smorgasbord of options!

Some of the tests are specifically for one-way tables or 2 x 2 tables, but many apply to two-way tables of any dimension. You can specify exact p-values by using the EXACT statement in PROC FREQ.

The "Details" section of the PROC FREQ documentation lists all of the tests and statistics that support exact p-values or exact confidence intervals. With so many tests and options, it can be challenging to determine which tests are available for which kinds of tables. To better understand the choices, I classified the options according to the dimensions of table: one-way tables, 2 x 2 tables, general two-way tables, and stratified 2 x 2 tables. In the following lists, each description mentions the name of the option on the EXACT statement that specifies the computation.

One-way tables

For one-way tables, PROC FREQ provides exact p-values for the following tests:

• Binomial proportion tests (BINOMIAL). The same option also provides exact (Clopper-Pearson) confidence limits for binomial proportions.
• Chi-square goodness-of-fit test (PCHI and LRCHI options). You can tests a null hypothesis of equal proportions, or you can specify the proportions.

Two-way tables

For two-way tables, PROC FREQ provides exact p-values for the following tests:

• Chi-square tests, including the Pearson chi-square test (PCHI), likelihood ratio chi-square test (LRCHI), and Mantel-Haenszel chi-square test (MHCHI), or use the CHISQ option to get all three.
• Fisher’s exact test (FISHER).
• Jonckheere-Terpstra test (JT), which assumes that a column represents an ordinal response variables.
• Cochran-Armitage test for trend (TREND).

There are also exact p-values for the following statistics:

• Kendall’s tau-b and Stuart’s tau-c statistics (TAUB and TAUC).
• Somers’s D statistics (SMDCR and SMDRC).
• Pearson and Spearman correlation coefficients (PCORR and SCORR).
• Simple and weighted kappa coefficients (KAPPA and WTKAPPA).

2 x 2 tables

In addition to the tests for general two-way tables, for 2 x 2 tables PROC FREQ provides exact p-values for the following tests:

• McNemar’s exact test for matched pairs (MCNEM).
• Exact confidence limits for the odds ratio (OR).
• Unconditional exact test for the proportion difference (BARNARD).
• Exact unconditional confidence limits for the proportion difference (RISKDIFF) and for the relative risk (RELRISK).

Stratified 2 x 2 tables

For stratified 2 x 2 tables, PROC FREQ provides the following:

• Zelen’s exact test for equal odds ratios (EQOR).
• An exact test for the common odds ratio, which also provides exact confidence limits for the common odds ratio (COMOR).

When were these tests first available in SAS?

The previous sections contain a lot of information. Because not everyone is running SAS 9.4, I was curious as to when some of these tests were introduced. I was able to compile the following (non-exhaustive) list by trolling through various "What's New" documents:

• SAS Version 6: Fisher's exact test was introduced.
• SAS Version 8: The EXACT statement was introduced in SAS version 8. It contained exact p-values for binomial proportions in one-way tables, the many chi-square tests, and Fisher's exact test.
• SAS 9.1: Exact confidence limits for the common odds ratio and related tests.
• SAS 9.2: Exact unconditional confidence limits for the proportion (risk) difference and Zelen’s exact test for equal odds ratios.
• SAS 9.3: Exact unconditional confidence limits for the relative risk.
• SAS 9.4: The MIDP option in the EXACT statement produces exact mid p-values for several tests.

How does PROC FREQ compute exact p-values?

I've discussed what exact computations are available and when some of them became available, but what about how the computation works?

Because tables are integer-valued, you can theoretically enumerate all tables that satisfy a particular null hypothesis. For example, in a recent article about 2 x 2 tables, I was able to write down the nine possible tables that satisfied a particular constraint and to assign probabilities for each distinct table. Consequently, if you compute a statistic for each table, you can exactly calculate the probability that the statistic is more extreme than a particular observed value. That probability is the exact p-value. (For a more precise definition, see the "Exact Statistics" section of the documentation.)

In practice, PROC FREQ does not directly enumerate all of the possible tables that satisfy the null hypothesis for each test. For some of the exact tests for a general two-way table, it uses a network algorithm of Mehta and Patel (1983), which is more efficient than direct enumeration. The algorithm enables the procedure to compute exact p-values for many small and moderately sized tables.