After reading Gartner's 2015 Hadoop adoption study results by Analysts Nick Heudecker and Merv Adrian, the first thing that comes to my mind is Goethe's phrase from Egmont, "Himmelhoch jauchzend, zu Tode betrübt." Translated: heavenly joy, deadly sorrow.
What happened to yesterday's hype around the cute yellow elephant - which I still think can repeat the success of Linux in data centers - and what happened to all the talk about its potential to revolutionize the market for data storage and processing?
Let's take a step back and look at what has changed. From a technology perspective, the Hadoop ecosystem has progressed significantly for the past few years. In some areas, it still requires a lot of expertise and knowledge to get things done. Other areas like loading data into Hadoop and visualizing data stored in Hadoop, have become more user friendly for even non-techie users. Tools like the SAS Data Loader for Hadoop now make the platform accessible for business users.
Why then are so many of the respondents to the Gartner survey refusing to have a look at Hadoop? (54percent have no plans to invest and only 18 percent plan to do so over the next two years.)
One of the reasons is that Hadoop has become too closely associated with big data, and the Hadoop platform is perceived to be only relevant for companies dealing with big data. This buzz is not only overhyped, it is a deterrent for many potential users who could benefit from Hadoop now. Because yes, not everybody has big data (however you might define it) right now.
BUT does Hadoop = Big Data? No!
Although a lot of the success for Hadoop has come through its adoption at massive scale at Web companies like Facebook, Yahoo and Google, it has been around for much longer than the big data buzz.
And I would predict that Hadoop will stay around long after the hype around big data dies down. (Note: the forecast below is not based on any sophisticated analytical method, but purely on my industry experience and my rudimentary Photoshop skills. ;))
So is Gartner completely wrong about this?! Well, Gartner did a survey. So the results reflect the current situation at their customers’ organizations - and this is partially in sync with what I am seeing here in Switzerland.
Many of the survey respondents report a massive skills gap (57 percent) and only about half know how to draw value from Hadoop.
Similar results are shown in the upcoming, BARC Hadoop survey (sponsored by SAS), which will be published soon. The study asked about 250 respondents in the DACH region about their Hadoop plans and 77 percent say that it is less important (39 percent) or not important at all (38 percent) to them. Adoption is even less than the Gartner survey suggests.
My recommendation to everyone following the discussion around Hadoop is the following: don’t be too afraid of the naysayers or the skills gap. The Hadoop platform has great potential and is still progressing as expected, with more clients of all sizes finding greater uses for Hadoop all the time. Even with the big data hype experiencing a downturn, Hadoop is a technology framework that stays. Hadoop is not necessarily about extreme data volumes or massively scaled clusters. It can (and will) deliver value in many areas (and sizes), and it’s time now to jump on the train and be a leader instead of a follower.
Approach us if you are looking for use cases and ROI estimations in a modern SAS and Hadoop deployment. We can bring some fresh ideas and best practices on the table that are worth looking at.
If you are in the DACH region, have a look at the Big Data Lab, which is a risk free way for you to find the right use case(s) fast and cost efficiently for Hadoop, whether your data is truly "big" or not. For readers outside of the DACH region, consider the benefits of an innovation lab.
