By Stefan Ahrens, Sr. Solutions Architect, SAS Germany
Recently, there has been a hot debate about a Facebook experiment where users unwillingly participated in a psychological study with manipulated news feeds. While the fact that Facebook customers are involved without their prior consent is an entirely valid discussion, let’s not throw away the idea of experimentation entirely.
Quite the contrary, experiments – done the right way – can help us gain knowledge with a minimum amount of exposure by those that are affected. In other words, if you know how to set up the experimental conditions in a proper way, you may be able to get the same amount of information – with a lower number of what in statistics we would call „experimental units“. It goes without saying, that it’s still a good idea to seek approval by your customers before you start experimenting on a grand scale.
Lessons from Design of Experiments
Scientists and statisticians in different industries – most notably in process manufacturing – have done this for a very long time now established a field called Design of Experiments (DOE). While modern DOE in can be used to set up and analyze very complex experiments, you probably wouldn’t need to go that far in the business world. Still, you can learn a few tricks from the art of DOE. So, I wonder why the concept of experimental design hasn’t caught up there yet.
Take for example the design of a web site’s landing page. There are certain elements such as layout, type of font, color scheme, wordings of text, images, header message that are within your control, and you might want to find the optimal setting to induce a certain behavior of your website visitors.
Limits of One-Factor-At-A-Time (OFAT) method
Traditionally, people that are responsible for this, have done this with a series of A/B tests where they would vary one of those elements at a time and compare the response to that of a control group in a pairwise fashion, thinking of this as being „the scientific method“.
However, as soon as several elements are involved and those elements have various alternatives, things get a little bit more complicated, and the One-Factor-At-A-Time (OFAT) method is no longer the most efficient. It is exactly here where DOE can make the most valuable contribution.
Getting started with the DOE method
Yet, in order to do DOE the right way, there are couple of things that you should ensure before you start:
(1) Have a corporate culture that embraces experimentation. In other words, make sure that you are allowed to vary all those aspects that are relevant to the business problem that you are trying to address in a systematic manner, even if some of the combinations are more preferred than others. It is here where change agents are needed to bring a fresh perspective to the matter. For example, they might help you bridge the gap between how experimentation is seen traditionally in marketing and new media departments, and what are DOE best practices in the scientific world and R&D departments.
(2) Think thoroughly about what it is that you want to achieve. What kind of behavior (response variable in DOE speak) do you want to measure? What type of elements (called factors in DOE) do you want to test? And what are the different alternatives per element (called factor levels in DOE)? Which of those elements are within your control, which are beyond your control? (This is where a technique called blocking comes into play in DOE.)
(3) Generate a proper layout of the series of experiments that you want to run. What combinations of factors (i.e. treatments) do you want to test? How do you deal with the issue of a control group? How many observations do you need per treatment and how do you assign your observations (i.e. experimental units) to the treatments? Is there a cost associated with each observation? If so, how many observations can you afford in total?
(4) If possible, randomize the order of runs. Don’t try to run the treatments in a sequential or any other systematic order. There might be things that can affect the behavior that you want to measure over time and that you cannot control. Randomization will help you to make sure that those things do not influence the results of your experiment.
(5) Have a clear understanding of how to interpret and analyze the results of your experiments once you collected the data. How do you compare the response between different treatments? How do you measure the difference between the responses (i.e., the size effect of the effect in DOE)? Do you see interaction effects, i.e., change in response for one factor depends on the levels of another factor? How can you display the results in a visual manner? Do you need to look at statistical significance testing? What kind of software will help you in analyzing your data?
If you keep these 5 things in mind, I’m sure DOE can help you get answers to your business questions faster than experimenting in the traditional way using a series of A/B tests.
