As I write these words, AMC is gearing up for the final eight episodes of its award-winning show Breaking Bad. In the press leading up to the premiere, show creator Vince Gilligan has been giving plenty of interviews while concurrently keeping a lid on how the show ends. Gilligan has said this much: It won't be pretty, but it will be definitive.
That is, we will not see an ending redolent of The Sopranos, a show whose finale "Made in America" sparked a great deal of controversy. Millions of people (myself included) thought that their TVs or cable connections died at the most inopportune time. Nope. David Chase's intentionally vague denouement left the door ajar for more episodes and maybe even a movie, at least until the death of star James Gandolfini on June 19.
Netflix and big data parallels
I think about the endings of series quite a bit, and it seems to me that big data is much more Sopranos-like. That is, big data never really "ends."
Consider Netflix for a moment, one of the most prominent big data companies out there. In a recent interview with SiliconAngle, head of data platform architecture Jeff Magnusson discussed one particularly thorny technical issue: the difficulty in understanding Apache Pig raw code due to the complexity of certain scripts. Maria Deutscher writes that “Netflix solved this problem with a visualization tool called Lipstick. The homegrown program transforms code into directed acyclic graphs, or DAGs, that make it easier to spot bugs in large projects." (DAGs are useful for modeling different kinds of structures in mathematics and computer science.)
And Netflix is never really finished with big data. Lipstick will not be the last data visualization tool the company develops. Far from it. As the company continues to grow and evolve, so too will its big data and dataviz needs.
Simon says: get gray
Embrace this vagueness. What other choice does an organization have? The maxim "set it and forget it" just doesn't apply to big data. To be sure, looking at exception, audit and standard reports still has its place, but just don't expect a single, remotely definitive big data "report."
Employees at progressive companies like Amazon, Apple, Facebook, Google, Twitter and other big data companies understand the iterative nature of our world -- and the data behind it. Don't make the mistake of looking for only the black and the white. There's a great deal of value in the middle.
Feedback
What say you?
