Is it time for a big data reality check?

It’s been more than a year since I asked the question, “Is big data overhyped?” My answer today is the same as it was then: No. Big data is not overhyped. It’s real and it’s growing.

Do we need a reality check, though? Do we need to talk about how we got here, and who should be most concerned about big data?

Big data didn’t just happen. It’s been growing as the use of computers, smart phones and the Internet has grown. It’s been growing as more and more devices are outfitted with smart meters and sensors and GIS transmitters. It’s been growing for decades.

What was missing, until recently, was the ability to gain meaningful and useful insights from big data. But now, low-cost storage and in-memory computing have converged to help organizations make proactive decisions about the future marketing, product, and customer decisions with big data.

Of course, not everything is big data. We know that instinctively. There are still a lot of analytical problems that you can solve without big data storage or big data analytics. Fraud detection. Quality control. Basic data mining. Most of these things can be accomplished without big data.

So who does have big data and big analytics problems? Well, banks, retailers and pharmaceutical manufacturers are some of industries facing the most obvious big data challenges, especially the larger companies in these industries with hundreds of thousands of customers and hundreds of thousands of products or treatment options. Smaller retailers and even regional banks might not have to scale quite as high.

But here’s the most important thing to remember, even if you don’t have big data in house: Every organization has the potential to benefit from big data. Why? Because so many of today’s big data sources are public. Think open government data. Think weather and meteorological data. Think Twitter. The data is out there, it’s free, and it’s waiting for you to analyze it.

A few examples:

  • A hospital in the Netherlands is incorporating weather forecasts to predict increases in pulmonary problems for patients with lung disease.
  • A GPS manufacturer incorporates government and meteorological data into its systems to augment the data it already receives from drivers and traffic patterns.
  • A chemical manufacturer stores as much Web data as it can get its hands on to better understand the use of its plastic products around the world.
  • Economic analysts are analyzing trending topics on social media to predict changes in unemployment rates, before national unemployment rates are released.

Even if you’re a small player in your industry, if you’re the first to store and incorporate some of these open data sources into your existing analytical work, you’ll have an advantage over your competitors. It might mean you’re dealing with big data for the first time, but data visualization and open source storage options are making the entry point feasible for almost any organization.

Does everyone have big data? Not even close. But is there big data opportunity for almost everyone? Yes, there is. And that’s the reality, not just hype.

tags: big data

One Comment

  1. Posted November 13, 2013 at 1:55 pm | Permalink

    Well said Jim.

    We seem to go through this with every advance in technology. First we are enamored. The technology, whether it is Big Data, Relational Data Bases or ERP... is touted as the key to future prosperity.

    Then we become disappointed, partly because it doesn't work as advertised, but partly because it doesn't do what it was never intended to do.

    For example big data may uncover useful results, but are these results incorporated into business process. Is it answering the right questions? These are beyond the scope of the technology.

    If you want to get value out of a technology, then you need to understand what it does do and what it doesn't. If you expect Hadoop to integrate itself into your business process you are going to be disappointed. Processing and Analyzing big data opens up a world of possibilities. But it still doesn't do the thinking for you.

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <p> <pre lang="" line="" escaped=""> <q cite=""> <strike> <strong>