In this era of big data hype, it's easy to understand the hesitation of many organizations to take the plunge. Finding a signal in noisy petabytes of unstructured data isn't easy. Companies like Netflix, Amazon, Facebook, Twitter and Google that "do" big data well have spent hundreds of millions of dollars (or more) deploying, modifying and utilizing new applications and filesystems.
Make no mistake: These technologies aren't WYSIWYG. While we may have decades of experience with relational databases, the same can't be said about nascent technologies like Hadoop, NoSQL and others.
So, where does an organization start?
That's not a simple question to answer, especially in a single blog post. Operationalizing big data is a book-worthy subject, as I discovered when I wrote Too Big to Ignore. With so much hype, it's tempting for hidebound organizations to dismiss the topic altogether.
And I've met many people who do that very thing. In my view, dismissing the ability of data to solve business and societal problems is a big mistake. Still, some men you just can't reach, to steal from Cool Hand Luke.
Yet there is a middle ground with respect to the expectations of big data. No, data can't solve every business problem, but don't think for a moment that data can't help discover new insights and find elusive solutions, a point echoed by Scott Berkun. On his blog, Berkun writes about the limitations of data. From his post:
Data is a flashlight. Data gives you specific information about a singular vector of information. Data, like a flashlight, is only as useful as the person wielding it and the person interpreting what it shows. It has no magical powers. To get good information you want multiple sources so you can triangulate information and compensate for the inherent biases each kind of data has. For example, A/B testing can tell you things customer interviews can’t and vice versa.
It's an interesting post, and I don't disagree with anything in it. At the same time, though, I wonder if similar viewpoints only reinforce the dataphobia of far too many individuals. In other words, how many people read this and wonder if they can continue to get by exclusively on intuition?
Now, maybe. In the future, though, I wouldn't bet on it.
Simon says
Big data is like religion. If you're staunchly opposed to it, it's unlikely that a talking head or a book will convince you otherwise.
Recognize that managing expectations is key. What's more, it's folly to believe that everyone is on the same page. Dataphobes are lurking everywhere. Thwart them by setting expectations at a reasonable level. Promising the moon will only give fodder to naysayers.
Feedback
What say you?
