If you’re a marketing professional it doesn’t get much better than Andy Pulkstenis’ talk at Analytics 2012. Do You Know or Do You Think You Know: Creating a Testing Culture at State Farm documented the value of implementing experimental design into your marketing efforts.

During the talk, Pulkstenis outlined the multifactor testing State Farm executed on its web site and the outstanding results they received through testing to maximize their metric of interest.

As far as multifactor experiments, the test Pulkstenis’ team executed was fairly simple. He measured four “Test Factors” with two or three possibilities for each factor. The result was 24 possible combinations, 12 of which were randomly surfaced when a potential customer visited the State Farm web site. After testing the subset (and running predictive models on the other 12), State Farm was able to determine the optimal web site design to surface, resulting in a 12% lift over their current page. But results were only part of the success story Pulkstenis shared; he also discussed the how his organization has embraced testing at all levels and reaped the benefits.

Fortunately, at SAS Education we’ve also developed a culture of testing when it comes to marketing our SAS courses and certifications. And that’s good news for our users as well. As a service organization with a mission to help SAS customers get the most out of their software investment and advance their professional careers, our primary goal isn’t to maximize revenue, but to expand our reach. To help us achieve this goal, we’re measuring our campaign results and matching the results with student profiles to surface the right kind of information to match them to the right course, certification or learning experience. It’s exactly the sorts of thing Pulkstenis says mature testing organizations do.

Do you have a true testing culture in your organization? According to Pulkstenis, a testing culture demonstrates the following characteristics:

1)      They execute designed experiments to inform (and determine) strategy – They “do a test” before making a change.

2)      They know when to settle for a simple A/B test and when to use “advanced” multifactor tests.

3)      They impose internal standards and best practices for test design.

4)      They have formal internal governance.

How did you do? If you’re not quite there yet, Pulkstentis gives us a road map for launching testing in your organization.

1)      Be a teacher – get people excited about the possibilities.

2)      Be Opportunistic – find impactful opportunities.

3)      Be Persistent – you need to be passionate.

4)      Be Ambitious – aim for moderate or greater business impact.

5)      Be Careful – partner with business partners in a cautious way.

6)      Be Ready – Build the “success story” deck before you need it.

So there you have it – results from strategic testing in marketing and a road map for those who aren’t there just yet. If you’re interested in learning more about the talk, visit our web site, where we hope to post presentations from Analytics 2012 soon.