Ask anyone who has been involved in the IT space over the past few years – they'll confirm that there have been quite a few predictions about Hadoop. These vary drastically from one person to the next.

While one person will say Hadoop is just a fad, another insists that Hadoop will revolutionize the way we do business. Someone may say Hadoop is just another shiny object, but their office neighbor declares Hadoop will replace the data warehouse. (Granted, you're far less likely to hear this comment now as compared to a couple of years ago.)

No doubt, there are a multitude of opinions about Hadoop. Likewise, there have been many usage surveys on Hadoop, and it's informative to review the results.

And the surveys say...

Two main takeaways tend to be consistent across the various Hadoop usage surveys that have been done:

• Interest in Hadoop and the number of Hadoop pilot projects rises year after year.
• The rate at which organizations have put Hadoop into production is lower than expected.

Consider a 2016 survey conducted by the Business Application Research Center (BARC) in conjunction with SAS. For organizations with 2,500 or more employees, 83% either have a Hadoop pilot running already, are planning a Hadoop initiative, or say that a Hadoop initiative is conceivable in the future. But among that same group of organizations, only 18% have Hadoop in production. For midsize organizations, the number drops to 7%. Why, you might wonder – why are Hadoop adoption rates so low? Especially considering how much attention it has received over the last four years.

In three words – Hadoop is difficult.

What are the challenges of implementing Hadoop?

The same BARC and SAS survey highlights the main difficulties organizations face when implementing Hadoop:

• 54% of organizations surveyed cited a lack of professional know-how as a barrier.
• 50% of the organizations said they lacked the knowledge needed to construct a big data architecture.
• 41% said they were unsure how to use new findings from data research.

Many organizations today simply don’t have the resources or skill sets in place to take Hadoop head on – in other words, there is a Hadoop "skills gap." Despite this longstanding lack of skills, most people recognize the benefits of Hadoop and understand that it will be part of their infrastructure in the future. That's because Hadoop complements and optimizes existing platforms as customers modernize their analytics and data architectures.

Our data management solutions at SAS have always been developed with the goal of enabling fast implementations and making things easy for both IT and business people to use. It's no different for Hadoop. Many of the challenges that prevent organizations from putting Hadoop into production can be solved by using data management in conjunction with Hadoop.

How SAS Data Management makes Hadoop easier

Every organization's Hadoop challenges are different, and each must follow the route that works best for them. At SAS, we've found that our customers benefit the most by adopting the following practices:

• Read and write data to and from Hadoop as if it were any other source that SAS can connect to.
• Choose from the option to push processing into Hadoop – or extract from Hadoop to SAS – for a multitude of data management work.
• Profile, transform, cleanse and load data into Hadoop and in memory without writing code.
• Call data quality functions within Hadoop for casing, extraction, gender analysis, identification analysis, parsing, pattern analysis, standardization and match-code generation.
• Keep data in place and integrate SAS Scoring processes on Hadoop.
• Organize Hadoop data into a file format that includes a data component, a metadata component and indexes.
• Employ enterprise data management capabilities such as governance, mastering, monitoring, transformation and more.

As one of our customers in the health and life sciences industry explained: “With SAS Data Loader for Hadoop, our business analysts have self-service access to prepare and cleanse data without writing code or burdening IT.” Because it's easy for business people to use – but still robust enough for data scientists – Hadoop offerings from SAS are a must-have for many of our customers.

Yes, the hype over Hadoop has subsided. But now, a more exciting time is upon us – a time for results.