It’s been more than a year since I asked the question, “Is big data overhyped?” My answer today is the same as it was then: No. Big data is not overhyped. It’s real and it’s growing.
Do we need a reality check, though? Do we need to talk about how we got here, and who should be most concerned about big data?
Big data didn’t just happen. It’s been growing as the use of computers, smart phones and the Internet has grown. It’s been growing as more and more devices are outfitted with smart meters and sensors and GIS transmitters. It’s been growing for decades.
What was missing, until recently, was the ability to gain meaningful and useful insights from big data. But now, low-cost storage and in-memory computing have converged to help organizations make proactive decisions about the future marketing, product, and customer decisions with big data.
Of course, not everything is big data. We know that instinctively. There are still a lot of analytical problems that you can solve without big data storage or big data analytics. Fraud detection. Quality control. Basic data mining. Most of these things can be accomplished without big data.
So who does have big data and big analytics problems? Well, banks, retailers and pharmaceutical manufacturers are some of industries facing the most obvious big data challenges, especially the larger companies in these industries with hundreds of thousands of customers and hundreds of thousands of products or treatment options. Smaller retailers and even regional banks might not have to scale quite as high.
But here’s the most important thing to remember, even if you don’t have big data in house: Every organization has the potential to benefit from big data. Why? Because so many of today’s big data sources are public. Think open government data. Think weather and meteorological data. Think Twitter. The data is out there, it’s free, and it’s waiting for you to analyze it.
A few examples:
- A hospital in the Netherlands is incorporating weather forecasts to predict increases in pulmonary problems for patients with lung disease.
- A GPS manufacturer incorporates government and meteorological data into its systems to augment the data it already receives from drivers and traffic patterns.
- A chemical manufacturer stores as much Web data as it can get its hands on to better understand the use of its plastic products around the world.
- Economic analysts are analyzing trending topics on social media to predict changes in unemployment rates, before national unemployment rates are released.
Even if you’re a small player in your industry, if you’re the first to store and incorporate some of these open data sources into your existing analytical work, you’ll have an advantage over your competitors. It might mean you’re dealing with big data for the first time, but data visualization and open source storage options are making the entry point feasible for almost any organization.
Does everyone have big data? Not even close. But is there big data opportunity for almost everyone? Yes, there is. And that’s the reality, not just hype.