Thoughts on big data and new tools

"Man is a tool-using animal...Without tools he is nothing, with tools he is all."
—Thomas Carlyle

Let's go back to 2010 for a moment when the Great Recession was in full swing and IT budgets were on the chopping block or, at best, flat.

In hindsight, the arrival of big data caught most organizations off guard. At that point, relatively few had taken care of the basic blocking and tackling of data management – never mind discovering valuable insights through "analytics." (Tony Fisher makes this point in his book The Data Asset: How Smart Companies Govern Their Data for Business Success.)

In hindsight, perhaps it was never realistic to expect companies not named Amazon, Netflix, Facebook and Google to find the signal in our increasingly noisy world.

Few intelligent professionals dispute the import of big data today. (Whether this is mostly lip service is up for debate.) The question remains, though, how will organizations make sense out of things such as event stream data, petabytes of largely unstructured information and an increasing number of external data sources? Interesting subquestions include:

Do they need to buy and deploy totally new data management technologies?
Can certain tools such as ETL be retired for more contemporary equivalents? (Pfizer recently 86'd its ETL applications in favor of data virtualization tools.)
What types of information can/should be stored on-premise versus "in the cloud?"
Is building a new tool a viable option? What about forking an existing open-source application? (This is precisely what Internet-advertising firm Quantcast did.)
Can they adapt the tools they've already purchased? (Companies like Oracle are promoting "appliances" that can tie existing relational databases to Hadoop.)
Can they rent data scientists as needed via sites such as Kaggle?
Can they find a true partner such as Hortonworks or MetaScale?
What about other solutions?
How about a combination of the above as budgets and business and technology needs merit?

To make a long story short, there are no single right answers to these types of questions, let alone one "right" or "best" technology. Truth be told, there never have been. Even in the relatively simple era of Enterprise 1.0 (read: e-mail, intranets, ERP and CRM), many different enterprise applications, relational databases and data repositories could get the job done.

Simon Says: Tools by themselves won't get you home; culture is important, too.

Of course new tools matter. You'll never hear me say otherwise. With respect to big data, try cramming a bunch of photos and videos into an Oracle database and running an SQL statement against it. Good luck with that.

As I wrote in Why New Systems Fail, spending a great deal of money on shiny new things guarantees nothing – especially when employees stick to their hidebound, data-free ways. Generally speaking, it's folly to think of software and data in isolation. Perhaps it's more instructive to ask, How can our organization build a culture of analytics? By answering that largely human and "squishy" question, the confusion around tools will abate.