Data visualization, statistical discovery, design of experiments, predictive modeling and more

On Feb. 8, my colleague, Professor Douglas Montgomery of Arizona State University, and I presented a webinar for the American Statistical Association. Our first demonstration dealt with designing an experiment for six factors each having two levels in 24 runs. One natural way to construct such a design would be to choose six columns from the orthogonal array discovered by Plackett and Burman, which has 24 rows and 23 columns (see below).

+++++++++++++++++++++++
−++++−+−++−−++−−+−+−−−−
−−++++−+−++−−++−−+−+−−−
−−−++++−+−++−−++−−+−+−−
−−−−++++−+−++−−++−−+−+−
−−−−−++++−+−++−−++−−+−+
+−−−−−++++−+−++−−++−−+−
−+−−−−−++++−+−++−−++−−+
+−+−−−−−++++−+−++−−++−−
−+−+−−−−−++++−+−++−−++−
−−+−+−−−−−++++−+−++−−++
+−−+−+−−−−−++++−+−++−−+
++−−+−+−−−−−++++−+−++−−
−++−−+−+−−−−−++++−+−++−
−−++−−+−+−−−−−++++−+−++
+−−++−−+−+−−−−−++++−+−+
++−−++−−+−+−−−−−++++−+−
−++−−++−−+−+−−−−−++++−+
+−++−−++−−+−+−−−−−++++−
−+−++−−++−−+−+−−−−−++++
+−+−++−−++−−+−+−−−−−+++
++−+−++−−++−−+−+−−−−−++
+++−+−++−−++−−+−+−−−−−+
++++−+−++−−++−−+−+−−−−−

Hold on a sec. What is an orthogonal array?

An orthogonal array is a matrix of symbols, in our case “+” and “–“. In each pair of columns, there are only four possible pairs of symbols: “+ +”, “+ –“, ”– +” and “– –“. To be an orthogonal array, each pair of symbols has to occur equally often in every pair of columns. In the above array, each of the four pairs of symbols appears six times in every one of the 253 pairs of columns.

Orthogonal arrays have great historical significance in the field of experiment design. Until the advent of computers, virtually every experiment design used an orthogonal array. There are two main reasons why early practitioners focused on orthogonal arrays. First, it is easy to calculate the effect of changing any factor – you just average all the responses for trials run at the + level of the factor and subtract the average of all the response for trials run at the – level of the factor. Second, the estimated effect of any factor is statistically independent of the estimated effect of any other factor.

If orthogonal arrays are so great, why use anything else?

Orthogonal arrays are very useful, but they only exist for certain numbers of trials. For example, orthogonal arrays having two symbols in each column only exist when the number of trials is a multiple of four. If all the factors have two levels, an orthogonal array with 15 rows does not exist.

More important is the fact that although the main effects of each factor in an orthogonal array are independent, the two-factor interactions may not be. In the example that Doug and I presented, we wanted to be able to estimate both the six main effects as well as the 15 two-factor interactions. If you include the overall average, there are 22 unknown quantities that we want to estimate. Since there are 24 runs, it seems possible to fit this 22-term model using ordinary least squares.

Can you choose six columns from the Plackett-Burman design to fit this model?

It depends on which six columns you use. There are 100,947 ways you can pick six columns out of the 23. Of those, more than half are incapable of fitting the two-factor interactions model. Of course, there are 49,588 ways of choosing the six columns that do allow for fitting all the two-factor interactions. Depending on which group of six columns you choose, the average variance of the coefficient estimates can vary by a factor of >28. That is, for the six-column design, the least desirable choice estimates the coefficients 28 times worse than the most desirable choice. So, to construct your design by picking six columns from the above array, you have to be very careful.

Are there other orthogonal design choices that are better?

I asked my colleague, Dr. Eric Schoen of the University of Antwerp, for help here. Eric is a world-renowned researcher in orthogonal design. He has constructed a catalog of all the statistically different orthogonal arrays having six columns and 24 rows. It turns out that there are only 1,350 of them. So, most of the six column choices from the 23 columns of the Plackett-Burman design were non-unique. Also, they are not exhaustive. It turns out that 20 of Eric’s orthogonal arrays were better than any of the six column choices from the Plackett-Burman design. The best of these was about 8% better than the best orthogonal array I found previously. Only 447 of the 1,350 unique orthogonal arrays could fit the model.

Can you do better at estimating all the effects of interest?

The answer is yes but only if you do not use an orthogonal array! I constructed a D-optimal design using the Custom Designer in JMP. Table 1 shows the comparison of the variance inflation factors (VIF) for the D-optimal design compared to the best orthogonal array.

Table 1 – Variance inflation factors for D-optimal design compared to the best orthogonal array.

Note that the VIF for every main effect and two-factor interaction is lower (better) for the D-optimal design than for the best of the orthogonal arrays.

The bottom line – don’t limit yourself by only considering orthogonal arrays.

Many investigators only consider orthogonal arrays when planning their experiments. This restriction comes at a price as the example from the webinar shows.

Finding the best orthogonal array actually required more experiment design expertise and more computing than finding the D-optimal design, and the resulting design was statistically inferior.

Table 2 – Six-factor 24-run D-optimal design for the main effects and two-factor interactions model.

Table 3 – Six-factor 24-run best orthogonal array for the main effects and two-factor interactions model.

In my two previous posts, I introduced the correlation cell plot for design evaluation and then showed how to use the plot to compare designs. Here, I want to use the same plot to show why definitive screening designs are, well, definitive.

For a complete technical description of definitive screening designs, you can read "A Class of Three-Level Designs for Definitive Screening in the Presence of Second-Order Effects" -- an article I co-wrote with Chris Nachtsheim of the University of Minnesota. Chris and I were delighted to learn recently we had won the American Society for Quality's 2012 Brumbaugh Award for our paper. This award is presented to the author(s) of the paper that has made the largest single contribution to the development of industrial application of quality control. The paper was published in January 2011 in the Journal of Quality Technology, and you can read it via the JMP website.

What is a definitive screening design?

The most notable way that definitive screening designs are different from standard designs is that all the factors are numeric and are tested at three levels. A second distinctive feature of a definitive screening design is that it is a self-foldover. That is, the runs of the design come in pairs that “mirror” each other. Suppose we encode the low setting of a factor as “–“, the high setting as “+” and the middle setting as “0”. Then, if one run of a foldover pair has factor settings encoded “+ 0 – + – +”, the other run has factor settings encoded “– 0 + – + –”. Each pair of runs has one factor at its middle value and all the others at their high or low values. One run is at the center of the design region with all the factors at their middle setting. Table 1 shows a definitive screening design for eight factors. Notice that it has one more than twice as many runs as there are factors, that is, 17 runs.

Table 1 Definitive screening design for eight three-level factors.

So what makes this design so special?

To see why the design in Table 1 is fantastic, let us use the correlation cell plot in Figure 1. Our potential model terms are all the main effects, two-factor interactions and quadratic effects. Note that only the cells on the diagonal of the plot are pure red. That means that none of the model terms are confounded with each other.

Figure 1 Correlation plot for definitive screening design.

The last eight columns of the cell plot show the quadratic effect terms. These effects are only mildly correlated with each other (|r| = 0.19). Each quadratic effect is uncorrelated with a two-factor interaction involving its factor. That is, the quadratic effect of factor A is uncorrelated with the AB interaction. Other two-factor interactions have an absolute correlation of 0.37. It turns out that all eight quadratic effects are estimable with the definitive screening design. The main effects of the design are all orthogonal to each other and to all the second order terms (two-factor interactions and quadratic effects).

The two-factor interactions have pairwise correlations that can take one of three values. The pink cells represent absolute correlations of two-thirds. The light blue cells represent correlations of only one-sixth. The pure blue cells show uncorrelated interaction pairs.

Let us compare this design and plot to the standard screening design for eight factors. That design is the minimum aberration fractional factorial design. This design is in Table 2, which has one added center run to make both designs have 17 runs including one center run.

Table 2 Standard screening design with one center run.

Figure 2 shows the cell plot for the fractional factorial design. The most notable feature of this cell plot is that all the cells are either pure blue or red. That is, every pair of columns is either completely uncorrelated or completely confounded.

Figure 2 Correlation plot for the standard screening design.

Note the block of red cells in the lower right. These red cells indicate that all the quadratic effects are confounded with each other. With one added center run, the standard screening design has some ability to detect very strong nonlinearity in the factor/response relationship. However, there is no way to determine which factor is causing the nonlinearity. By contrast, the definitive screening design can separately estimate the nonlinear effect of each factor.

Each two-factor interaction in the fractional factorial design is confounded with three other two-factor interactions. This means that if any two-factor interaction is active, the analysis can only indicate that there are four possible interactions that could explain the observed effect. Narrowing down this field to one interaction requires further experimentation. By contrast, the definitive screening design can reliably resolve any two-factor interaction that is large compared to its standard error.

Why are definitive screening designs definitive?

The purpose of screening is to separate the vital few factors that have a substantial effect on the response from the trivial many that have negligible effects. If a factor’s effect is strongly curved, a traditional screening design may miss this effect and screen out that factor. If there is a two-factor interaction, standard screening designs having a similar number of runs to the definitive screening design with the same number of factors will require follow-up experimentation to resolve the ambiguity. The definitive screening design can reliably accomplish the task of screening even if there are a couple of second order effects.