What magic teaches us about data science

Teller, the normally silent half of the magician duo Penn & Teller, revealed some of magic’s secrets in a Smithsonian Magazine article about how magicians manipulate the human mind. Given the big data-fueled potential of data science to manipulate our decision-making, we should listen to what Teller has to tell us.

“Magicians,” Teller explained, “have done controlled testing in human perception for thousands of years. Magic is not really about the mechanics of your senses. Magic is about understanding — and then manipulating — how viewers digest the sensory information.”

In his article, Teller explains seven principles that magicians employ to alter our perceptions. The first principle is pattern recognition. I have previously compared its role in data-driven decision-making to how we listen to music. We search for any pattern in data relevant to our decision that allows us to discover a potential source of insight. Once our brain finds a decision pattern, we start making predictions and imagining what data will come next. But sometimes the music of the data is the sound of pattern recognition directing our search for decision consonance among data dissonance toward comforting, but false, conclusions.

This leads us to the sixth principle that Teller explained. “Nothing fools you better than the lie you tell yourself. When a magician lets you notice something on your own, his lie becomes impenetrable.” The most impenetrable lie in data-driven decision-making is when the only thing we notice in data is what we were looking to find.

“Magic is an art,” Teller concluded, “as capable of beauty as music, painting or poetry. But the core of every trick is a cold, cognitive experiment in perception: Does the trick fool the audience? A magician’s data sample spans centuries, and his experiments have been replicated often enough to constitute near-certainty.”

Data scientists are not magicians, even if they are sometimes perceived that way. However, data science, just like magic, is an art with a cognitive experiment in perception at its core. We need data scientists to help us perform those experiments. We also need data engineers to build the infrastructure around data and algorithms, data artists to help us visualize data, and data philosophers to help us see through our own illusions about data.

tags: analytics, big data, Big data analytics, data quality, data science

2 Comments

  1. Posted March 19, 2014 at 10:00 am | Permalink

    Excellent insight, Jim. I love the tie in to Teller and magic. You have nailed well a big challenge of analyzing data - going in with a preconceived notion and then stopping when you find it.

    Insightful users of data will let the data take them on a journey that goes where the data wants. And, yes, we absolutely need the data scientists, engineers, artists and philosophers to take this journey. Let us hope the river of data is clean because if the waters of accuracy are muddy, our journey can get mighty messy.

  2. Posted March 21, 2014 at 5:55 am | Permalink

    Freaky, Jim. I just blogged on Magic vs Data Management today too...! http://informationaction.blogspot.com.au/2014/03/now-thats-magic.html

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <p> <pre lang="" line="" escaped=""> <q cite=""> <strike> <strong>