Is big data over hyped?

We’ve definitely hit the point in the “big data” hype cycle where people are looking critically at the term and asking what all the fuss is about. Some pundits have even speculated about a big data bubble that’s sure to burst after everyone realizes the term has been over-used, and technologies like in-memory processing and high-performance analytics (HPA) are not actually in high demand.

Folks, there’s no bubble here. You can’t put the data back in the bottle, so to speak. Big data – however you define it – isn’t going away and it isn’t getting smaller. It’s going to keep growing.

Maybe the term “big data” will change or devalue in significance, but you can’t poke holes in the concept itself, which is this: there are significant business benefits to be gained from storing and analyzing large volumes of data more efficiently.

That doesn’t mean big data is an issue for everyone or that high-performance analytics is the answer for everyone.   But there is a business case to be made for the use of high performance analytics in many arenas, and those that see the whole idea as overhyped are looking at it too narrowly.

Here’s how I like to look at it: High-performance computing is, simply, an enabler. Most importantly, it enables you to get answers faster than before. But – and this is important – high performance computing is only as good as what you’re computing. If you’re getting summary statistics about your business portfolio, HPC can give you those reports faster. However, if your system is predicting risk exposure on thousands of assets, you’re going to get those predictions faster than before. Or, if you’re optimizing markdowns for millions of SKUs at hundreds of retail locations, you’ll be able to optimize those prices more quickly. That's high-performance analytics.

You see the difference?

A lot of big data proponents are promising things bigger, better and faster. But if the information you’re getting is backward looking, it’s still going to be looking at the past when you get it in a shorter timeframe. You’re still only understanding the past faster than before. No matter how fast you go with summary statistics, you’re never going to get to the future.

Only predictive analytics like forecasting and optimization will bring you out of the past and into the future. When you use high performance analytics to predict things like risk, customer satisfaction or marketing optimization, you’re getting your predictions sooner than before, and you can react more quickly. When you’re computing forward-looking results, the speed really can make a difference.

At its most basic level, high-performance computing reduces the time dimension. You have to decide what types of answers you want more quickly: standard reports or predictive analytics.

Once you know that, you can start asking questions and making decisions that could change your business. Or your world. I don’t think that’s hype. It’s as real as you can get, and the organizations that get it first are going to be the ones that make it further into the future.

tags: big data, high-performance analytics, high-performance computing, predictive analytics

3 Comments

  1. Sidharth
    Posted February 8, 2012 at 11:41 am | Permalink

    Epic blog .. right on the bulls eye ..

  2. amol
    Posted June 1, 2012 at 1:21 am | Permalink

    Spot on with "No matter how fast you go with summary statistics, you’re never going to get to the future."

  3. Ken Comer
    Posted February 1, 2013 at 9:35 am | Permalink

    I got to this page from a Google search in 2013 on anything negative about 'big data'. It's interesting that this is one of the few articles (in English) that comes to bury Caesar not to praise him.

    I am one of those 'doubters' that you speak of. Analytics on big data will allow us to make predictions faster on a more robust set of information, but will we be any more accurate? Or, does all this new high-speed computing just allow us to reach the wrong answer faster?

    If our tools did not properly identify risks before, have they become measurably more effective as big data and big data analytics arrive on the scene?

    That having been said, congratulations for being one of 2012's very few big data contrarians.

    - KC

5 Trackbacks

  1. By HPA once a day: day one - SAS Voices on February 13, 2012 at 12:02 pm

    [...] to "The Age of Big Data" from The New York Times Sunday Review. More than a surface read about the trendy big data theme, author Steve Lohr gets into the technology needed to find answers in big data right from the start: [...]

  2. [...] Davis talks about high-performance as an enabler and says big data alone is not the answer: A lot of big data proponents are promising things bigger, better and faster. But if the information [...]

  3. [...] one is a no-brainer and I’ve covered number two before (see previous post: Is big  overhyped?). I’d like to discuss number three here by talking about this new career title: the data [...]

  4. [...] Is big data over hyped? [...]

  5. [...] been more than a year since I asked the question, “Is big data overhyped?” My answer today is the same as it was then: No. Big data is not overhyped. It’s real and it’s [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <p> <pre lang="" line="" escaped=""> <q cite=""> <strike> <strong>