What magic teaches us about data science

2

Teller, the normally silent half of the magician duo Penn & Teller, revealed some of magic’s secrets in a Smithsonian Magazine article about how magicians manipulate the human mind. Given the big data-fueled potential of data science to manipulate our decision-making, we should listen to what Teller has to tell us.

“Magicians,” Teller explained, “have done controlled testing in human perception for thousands of years. Magic is not really about the mechanics of your senses. Magic is about understanding — and then manipulating — how viewers digest the sensory information.”

In his article, Teller explains seven principles that magicians employ to alter our perceptions. The first principle is pattern recognition. I have previously compared its role in data-driven decision-making to how we listen to music. We search for any pattern in data relevant to our decision that allows us to discover a potential source of insight. Once our brain finds a decision pattern, we start making predictions and imagining what data will come next. But sometimes the music of the data is the sound of pattern recognition directing our search for decision consonance among data dissonance toward comforting, but false, conclusions.

This leads us to the sixth principle that Teller explained. “Nothing fools you better than the lie you tell yourself. When a magician lets you notice something on your own, his lie becomes impenetrable.” The most impenetrable lie in data-driven decision-making is when the only thing we notice in data is what we were looking to find.

“Magic is an art,” Teller concluded, “as capable of beauty as music, painting or poetry. But the core of every trick is a cold, cognitive experiment in perception: Does the trick fool the audience? A magician’s data sample spans centuries, and his experiments have been replicated often enough to constitute near-certainty.”

Data scientists are not magicians, even if they are sometimes perceived that way. However, data science, just like magic, is an art with a cognitive experiment in perception at its core. We need data scientists to help us perform those experiments. We also need data engineers to build the infrastructure around data and algorithms, data artists to help us visualize data, and data philosophers to help us see through our own illusions about data.

Share

About Author

Jim Harris

Blogger-in-Chief at Obsessive-Compulsive Data Quality (OCDQ)

Jim Harris is a recognized data quality thought leader with 20 years of enterprise data management industry experience. Jim is an independent consultant, speaker, and freelance writer. Jim is the Blogger-in-Chief at Obsessive-Compulsive Data Quality, an independent blog offering a vendor-neutral perspective on data quality. Jim is the host of the popular podcast OCDQ Radio, and is very active on Twitter, where you can follow him @ocdqblog.

2 Comments

  1. Excellent insight, Jim. I love the tie in to Teller and magic. You have nailed well a big challenge of analyzing data - going in with a preconceived notion and then stopping when you find it.

    Insightful users of data will let the data take them on a journey that goes where the data wants. And, yes, we absolutely need the data scientists, engineers, artists and philosophers to take this journey. Let us hope the river of data is clean because if the waters of accuracy are muddy, our journey can get mighty messy.

Leave A Reply

Back to Top