A few words on experimentation and measurement


Last week, I had the great pleasure of hearing Dan Ariely, behavioral economist and author of the excellent book, Predictably Irrational: The Hidden Forces That Shape Our Decisions, speak at a seminar in Toronto. This seminar was part of a series from Richard Ivey School of Business and SAS Canada: Enabling Transformational Change.

Dan's perspective on experimentation, learning, and overcoming some of our predictable irrationalities is truly illuminating. In many ways, the brain is among the last frontiers for research—one could argue that we know more about the world we live in than we do about the inner workings of the brain, its potential, and limitations. Being mindful of influencing forces on our decision-making is relevant for individuals as well as organizations. Dan’s experiments are great examples of applying analytics to help us all do better.

Designing such clever experiments to extract truths from our world is a noble endeavor and one we should adopt to a greater extent--taking the time to observe, assessing what we are measuring and what important things we could measure to gain better understanding and make better decisions. Even measuring (where we can) the impact of mixing judgment with quantitative modeling: for example, much has been written about judgment and statistical forecasts. Often, the minor tweaks and over-rides people make to the statistical forecasts make the overall forecast accuracy worse. When large over-rides are made and people have to more carefully consider adjusting the statistical forecasts, forecast accuracy can be improved. (Paul Goodwin of the University of Bath gave a most interesting talk on this at F2006.)

Measurement is often overlooked, but it needs revisiting from time to time if you consider data and analytics strategic assets (key take-aways in Competing on Analytics by Tom Davenport and Jeanne Harris). So many organizations seem to be so focused on opportunistically collected transactional data, they have not considered if there are other things they could be measuring--often at very little cost--to gain greater insights and further compete on analytics.

In a recent conversation with Bill Kahn, PhD statistician with a wide range of interesting experience in several industries, he said "In times of rapid change, you need to be a rapid learner." Clever experiments enable you to learn very quickly, to see what matters most and why.

Some of the examples in Dan's books are discussed in talks you can view online. Links for those interested are below:


About Author

Anne Milley

Sr Director, Analytic Strategy, JMP

Anne oversees analytic strategy in JMP Product Marketing. She is a contributing faculty member for the International Institute of Analytics. She enjoys organic gardening and spending time with her family.

Leave A Reply

Back to Top