Why an agile mind-set will lead to better data prep and analytics

young woman with laptop studying agile development While the terms and technologies have changed, analytics has been with us for a long time now.

Let's go back in time for a moment to the early 2000s. Equipped with new databases, data warehouses and even BI tools, mature organizations were eager to turn data into knowledge, insights and even action. When embarking on analytics "projects," many would contemplate the following data preparation questions:

Do you know what story you want to tell before you prepare the data?
What's the end goal?
What are you trying to see with your data? (Implicit in this question is the importance of working closely with people who understand the data.)

Note how simple these questions seem.

To be sure, these weren't irrelevant or irrational queries. After all, thanks to rampant data quality issues, data often wasn't ready to be released into the wild – and that continues to be the case today. In fact, according to some estimates, organizations spend a mind-boggling 80 percent of their time preparing data for analytics instead of gleaning insights.

Still, I have to wonder whether that type of linear, phase-gate, and open-loop approach to data prep and analytics will lead to success. (Gartner sure doesn't think so.) After all, the speed of business has doubtless intensified in the last decade. Is it sage to apply arguably antiquated methodologies to the same analytics "projects"?

Better questions

It's a fair question and one of particular interest to me. I spent the last few months preparing for my new gig as a faculty member at Arizona State University. One of the courses I'm teaching is CIS450: Enterprise Analytics. The course emphasizes agile development methodologies, specifically Scrum.

Against that backdrop, what if organizations focused instead on addressing the following types of ongoing questions?

What types of stories (plural) can organizations address via analytics and data preparation? The notion that the data can only tell a single story today seems misplaced and even irresponsible.
What types of data and data prep are most important today? What about tomorrow? (It's silly to think of data prep as a binary.)
What's the end goal today?
How will we even know if we've achieved our goal?
How can future iterations and data prep allow us to hit our subsequent goals?

Note the relatively open-ended nature of these questions, especially compared to those at the beginning of the post.

Simon Says: Close the loop.

Many organizations suffer because their analytics projects never generate sufficient and meaningful feedback. No, vanity metrics don't count. Put differently, these projects resemble washing machines that sequentially plow through cycles irrespective of whether the clothes are really clean.

Instead, it's best to think of analytics and data prep in terms of closed-loop systems. Even "dumb" thermostats constantly seek feedback and input – never mind smart ones such as the one from Nest/Alphabet.

Shouldn't your organization's analytics efforts endeavor to do the same? If this sounds impossible, check out The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win.