Data visualization, statistical discovery, design of experiments, predictive modeling and more

In my two previous posts, I introduced the correlation cell plot for design evaluation and then showed how to use the plot to compare designs. Here, I want to use the same plot to show why definitive screening designs are, well, definitive.

For a complete technical description of definitive screening designs, you can read "A Class of Three-Level Designs for Definitive Screening in the Presence of Second-Order Effects" -- an article I co-wrote with Chris Nachtsheim of the University of Minnesota. Chris and I were delighted to learn recently we had won the American Society for Quality's 2012 Brumbaugh Award for our paper. This award is presented to the author(s) of the paper that has made the largest single contribution to the development of industrial application of quality control. The paper was published in January 2011 in the Journal of Quality Technology, and you can read it via the JMP website.

What is a definitive screening design?

The most notable way that definitive screening designs are different from standard designs is that all the factors are numeric and are tested at three levels. A second distinctive feature of a definitive screening design is that it is a self-foldover. That is, the runs of the design come in pairs that “mirror” each other. Suppose we encode the low setting of a factor as “–“, the high setting as “+” and the middle setting as “0”. Then, if one run of a foldover pair has factor settings encoded “+ 0 – + – +”, the other run has factor settings encoded “– 0 + – + –”. Each pair of runs has one factor at its middle value and all the others at their high or low values. One run is at the center of the design region with all the factors at their middle setting. Table 1 shows a definitive screening design for eight factors. Notice that it has one more than twice as many runs as there are factors, that is, 17 runs.

Table 1 Definitive screening design for eight three-level factors.

So what makes this design so special?

To see why the design in Table 1 is fantastic, let us use the correlation cell plot in Figure 1. Our potential model terms are all the main effects, two-factor interactions and quadratic effects. Note that only the cells on the diagonal of the plot are pure red. That means that none of the model terms are confounded with each other.

Figure 1 Correlation plot for definitive screening design.

The last eight columns of the cell plot show the quadratic effect terms. These effects are only mildly correlated with each other (|r| = 0.19). Each quadratic effect is uncorrelated with a two-factor interaction involving its factor. That is, the quadratic effect of factor A is uncorrelated with the AB interaction. Other two-factor interactions have an absolute correlation of 0.37. It turns out that all eight quadratic effects are estimable with the definitive screening design. The main effects of the design are all orthogonal to each other and to all the second order terms (two-factor interactions and quadratic effects).

The two-factor interactions have pairwise correlations that can take one of three values. The pink cells represent absolute correlations of two-thirds. The light blue cells represent correlations of only one-sixth. The pure blue cells show uncorrelated interaction pairs.

Let us compare this design and plot to the standard screening design for eight factors. That design is the minimum aberration fractional factorial design. This design is in Table 2, which has one added center run to make both designs have 17 runs including one center run.

Table 2 Standard screening design with one center run.

Figure 2 shows the cell plot for the fractional factorial design. The most notable feature of this cell plot is that all the cells are either pure blue or red. That is, every pair of columns is either completely uncorrelated or completely confounded.

Figure 2 Correlation plot for the standard screening design.

Note the block of red cells in the lower right. These red cells indicate that all the quadratic effects are confounded with each other. With one added center run, the standard screening design has some ability to detect very strong nonlinearity in the factor/response relationship. However, there is no way to determine which factor is causing the nonlinearity. By contrast, the definitive screening design can separately estimate the nonlinear effect of each factor.

Each two-factor interaction in the fractional factorial design is confounded with three other two-factor interactions. This means that if any two-factor interaction is active, the analysis can only indicate that there are four possible interactions that could explain the observed effect. Narrowing down this field to one interaction requires further experimentation. By contrast, the definitive screening design can reliably resolve any two-factor interaction that is large compared to its standard error.

Why are definitive screening designs definitive?

The purpose of screening is to separate the vital few factors that have a substantial effect on the response from the trivial many that have negligible effects. If a factor’s effect is strongly curved, a traditional screening design may miss this effect and screen out that factor. If there is a two-factor interaction, standard screening designs having a similar number of runs to the definitive screening design with the same number of factors will require follow-up experimentation to resolve the ambiguity. The definitive screening design can reliably accomplish the task of screening even if there are a couple of second order effects.

What is a correlation cell plot?

In my previous post, I proposed a new graphic diagnostic tool for evaluating designed experiments. The suggested graph is a cell plot showing the pairwise correlation between two model terms as a colored square. If, as in Figure 1, there are 45 terms in the model, then the cell plot created using JMP shows 45*45 = 2025 colored cells.

Figure 1 Correlation plot for a textbook design.

What does this plot tell me?

Absolute correlations go from zero (blue) to one (red). The diagonal line of cells is red for any design, but it is most desirable for all the off-diagonal cells to be blue or at least on the blue end of the color spectrum. Every cell in Figure 1 is either pure blue or red. Cells colored pure red indicate that the associated model terms are indistinguishable from one another. In such a case, we say that the two effects are confounded. Looking at the first row of cells, we can see that the main effect of factor A is confounded with the D*G two-factor interaction.

The design shown in Table 1 below and graphed in Figure 1 is a regular fractional factorial design for nine factors in 16 runs. One way to construct such a design is to start with a full factorial design – in this case having four factors and 16 runs. Then, factor E is generated by the element-wise multiplication of the columns for factors C and D. Similarly, F = BD, G = AD, H = ABC, and I = ABCD. The result of this procedure is a design that is orthogonal for the main effects but which confounds 12 of the 36 two-factor interactions with a main effect. Suppose you ran this experiment and observed large effects associated with factors A, D and G. You would like to say that these three factors are all active. However, an equally explanatory model is the one with the terms D, G and D*G. So, the confounding in the design produces ambiguous analytical results. The only data-driven way to resolve such ambiguity is to do further experimentation.

Is there some way to avoid confounding main effects and two-factor interactions?

Figure 2 shows our diagnostic plot for one of the 87 nonisomorphic orthogonal fractional factorial designs for nine factors in 16 runs (see Table 2). Unlike Figure 1, the upper right section of Figure 2 has no red squares. Thus, no main effect is confounded with any two-factor interaction. Moreover, the largest absolute correlation of any main effect with any two-factor interaction is only one-half. On the negative side, this design has more interactions correlated with every main effect even though the correlations are not as large.

Figure 2 Another orthogonal 16 run fraction al factorial.

Let us go back to the scenario we considered before. Can we distinguish between the model having the main effects A, D and G and the alternative model having the main-effects D and G plus the D*G two-factor interaction? Absolutely! This is true as long as the observed effects are sufficiently larger than their standard errors.

What is the bottom line?

Comparing Figures 1 and 2 side by side provides a quick way of assessing the relative merits of the two designs. I prefer designs where the diagnostic plot contains fewer pure red squares.

Design Tables

Table 1 Minimum aberration fractional factorial design.

Table 2 Alternative orthogonal two-level fractional factorial design.

One concern I often hear about the use of software for optimal design of experiments is that the algorithm producing the design is a “black box.” To use the design, an investigator has to trust the black box. An optimal design is one created to maximize some scalar measure of information about some model(s). Generally, though, an investigator has interest in more than one scalar measure of design quality. For example, orthogonality or near-orthogonality is a desirable design characteristic. A measure of the orthogonality of two design columns is their correlation. If there are many factors in a design and if there is interest in two-factor interactions, the number of pairwise correlations to consider can be daunting. In this post, I introduce a diagnostic plot that incorporates all the pairwise correlations in a form that is easy to digest. Using this plot allows a quick and informative evaluation of how close a screening design is to being orthogonal.

Currently, there are several commercial software tools for generating optimal designs. Design quality is a valid concern of any user of such tools. Even if the supplied design is optimal according to the advertised optimality criterion, there are many quality characteristics of a design. Box and Draper in their book Empirical Model-Building and Response Surfaces provide a list of 14 such characteristics. In another, more recent text on response surface modeling, Myers, Montgomery and Anderson-Cook provide their own list of qualities of a good design. Two characteristics of interest here are the ability to precisely estimate model parameters and the ability to assess the adequacy of the model.

In screening experiments, the primary interest is in finding the few factors of many that have the most influence on the response of interest. Such experiments generally have few runs due to budgetary constraints and the preliminary nature of the investigation. In such cases, the investigator hopes that a first-order model including only the main effects of the factors will prove adequate to identify the vital few factors. However, the investigator may have a nagging concern that an active two-factor interaction may bias the analysis, resulting in the selection of an inactive main effect. A more unpleasant prospect is that an active main effect might be masked by an active two-factor interaction having a coefficient of the opposite sign.

A simple approach is to consider two models of interest – the model to fit (primary) and the model that may alias the fitted coefficients (potential). Typically, the number of terms in the combined models is larger than the number of budgeted runs. In a 1994 Technometrics article Bill DuMouchel and I considered this two-model framework and developed a Bayesian modification to D-optimal designs to allow some ability to estimate the potential model while guaranteeing that the primary model is estimable. Last year in an article in the Journal of Quality Technology, Chris Nachtsheim and I considered the same model setup. Our objective was to minimize the aliasing of the primary model due to active effects in the potential model while requiring that the statistical efficiency for the primary model not fall below some specified lower bound.

Considering the multiple criteria for design quality, it makes sense to have a diagnostic plot to allow direct evaluation of any design. In the next section, I describe how to make and use one such graph.

How do you make this graph?

For evaluating design orthogonality, I propose using a cell plot showing the pairwise correlation between two model terms as a colored square. Suppose the primary model has k two-level factor main-effects. One reasonable potential model would contain the k(k-1)/2 two-factor interactions. In this case, there are k(k+1)/2 terms of interest. The cell plot is a matrix of colored cells of dimension k(k+1)/2 by k(k+1)/2. For the design in the figure, k = 9 and there are 45x45 = 2025 cells in the plot, each depicting one pairwise correlation. A table containing these values has the same information but is impossible to scan quickly.

To construct this plot, start with the n by k matrix of factor settings where n is the number of runs in the experiment. Add to this matrix k(k-1)/2 columns obtained by the elementwise multiplication of every pair of factors. Now compute the correlation matrix of this matrix by finding all the pairwise correlations among the columns. The result of this computation is a square symmetric matrix of dimension k(k+1)/2 by k(k+1)/2.

The absolute values of the elements of this matrix range from 0 to 1. It is useful to create a color gradient that makes it easy to distinguish small absolute correlations from large ones. Two possibilities are to use a grey scale scheme where a zero correlation is white and a correlation of 1 is black. A more colorful alternative is to use blue for zero correlations with a color gradient to red for absolute correlations of 1. The diagnostic plot below shows an example using the blue to red color gradient.

Correlation plot for the minimum aberration fractional factorial design.

So what does this tell me?

Note that the diagonal of the plot is a line of red squares. These squares show the correlation of each column with itself, which is, of course, 1.

It may be useful to divide the plot into four sections that divide the terms in the primary model from the potential terms. The figure has a vertical line separating the main-effect terms from the two-factor interaction terms. The horizontal line makes the same separation. These lines divide the plot into four meaningful sections.

The upper-left section of the plot shows the correlations of the primary terms with each other. The most desirable color of all the off-diagonal squares is blue. Colors ranging toward red in this area mean that the terms in the primary model are correlated. The result of this correlation is inflated variance of the coefficients in the model. The figure shows the correlation plot for an orthogonal design, so all the off-diagonal squares in this section are blue.

The upper-right and lower-left rectangles show the correlation between terms in the primary model and the potential terms. Again, the most desirable color for all these squares is blue. As the color of an individual square approaches red, the associated potential term, if it is active, will add increasing bias to the associated primary term. If a square in this region is red, then the potential term and the primary term are confounded, making it impossible to determine which effect is causal. The 16-run textbook design associated with the figure is resolution III, meaning that some two-factor interactions are confounded with main effects. For example, the main effect of factor A is confounded with the DG interaction. Ideally, we would like to avoid such confounding. In my next post, I will introduce two designs that do just what we want.

The lower-right square of the plot shows the correlations between terms in the potential model with other terms in the potential model. If there are fewer runs than terms in the combined models, all of the off-diagonal squares of this section of the plot cannot be blue. If a square in this section is red, then the corresponding potential terms are confounded. Note, for example, that the AB, CH, EI and FG interactions are confounded.

Happily, this plot is already one of the design evaluation tools in JMP so you do not have to follow the construction steps above by hand.

Experiments for most of us are demonstrations of scientific principles. We recall the science class where we put litmus paper into a beaker of lemon juice and watched it turn pink.

In scientific research, many investigators still construct experiments to add support to a current hypothesis or perhaps disprove it. Such experiments may require only one run of the experimental machinery. Any subsequent runs are just to verify the result from the first demonstration.

So, when I say that my research area is experiment design, people ask, “You mean you show people how to be very careful when they set up their experiments?” Certainly careful setup, observation and measurement are important in any kind of experiment. But this is not what experiment design is about.

I want to explain what experiment design is. I also want to motivate the use of designed experiments in engineering problem solving. But first I need to point out that today’s products and the processes for making these products are much more complex than they were even a decade ago. The manufacture of the processing units that go into your cell phone may require dozens of steps, each having multiple controls. Even making toilet paper is a high-tech process!

Engineers are responsible for both defining products and building systems or processes that work. This requires attention to detail and ingenuity. Competitive pressure means tight development schedules and careful cost considerations.

The first step on the road from concept to products in stores involves building and testing prototypes. One way to test a prototype product, system, or process is to change it in some way and measure the effect of that change on some measure of performance. In experiment design parlance, measures of performance are called responses. Factors are aspects of the prototype that we can change or control. Usually, we are interested in the effects of more than one factor, which requires multiple tests of various prototypes. Here, experiment design is about how to decide which prototypes to test and how to go about testing them.

Experiment design process flow

Initially, the goal is to determine which factors have most substantial effect on a response. A brainstorming session may identify a dozen or more factors that we think might affect the response, but generally, only a few factors account for the vast majority of the variation in the response. Imagine trying to differentiate the effects of a dozen factors by trial and error. This is a common practice for engineers who have no experience with experiment design. Trial and error studies often waste resources and give ambiguous results. An expensive approach is to test all possible combinations of the factors. But with a dozen factors, this approach requires 4,096 trials, which is usually prohibitively time consuming and expensive. By contrast, a well-designed experiment can provide clear results while dramatically reducing the required amount of testing.

For example, screening experiments separate the vital few factors from the trivial many.  Experiment designs constructed for factor screening generally require only a small investment  in testing that is between one and two times the number of factors. The ability to distinguish the effects of so many factors with such a small number of test runs is due to the elegant structure of experiment designs for screening.

Until recently, engineers wanting to use experiment designs for factor screening had to rely on catalogues of designs in textbooks or other reference materials. These design tables were restrictive because they were only available for certain combinations of numbers of factors and test runs.

Nowadays, commercial software like JMP can construct screening designs on the fly based on the requirements of each unique study. A computer-aided approach to experiment design provides several advantages over the use of reference designs. Computer generation of designs is immediate. This allows for creation and comparison of multiple design alternatives. An additional benefit is the flexibility of creating a design for each unique problem rather than reformulating your problem to match the requirements of a reference design. Finally, the training or knowledge requirements for appropriate use of computer-aided designs are lower than those for traditional designs. This makes this technology potentially accessible to any engineer.

Imagine a world where the use of experiment design was a standard engineering procedure. Products would get to market faster at lower cost and with higher quality and reliability. Such a world would be a better place to live.