Yes, you've heard about it, unless you've been so busy that you haven't seen magazines, internet news or even television for the last couple of years: big data. This development has exploded onto the enterprise data management scene, and like any hot new concept, the marketing is breathless and urgent. Take advantage of this data or fall behind your competitors.
If you're a seasoned enterprise executive, you may be asking the obvious question: "How much should I care about this?" The answer depends on the structure and nature of your business. Here are a few things to consider when making the decision on pursuing big data.
Don't panic
When researching big data, you will likely see some relevant case studies on how a company is using big data to gain competitive advantage in some way, and it may make you anxious to get going.
At this point, consult the Technology Adoption Lifecycle curve. Big data is arguably somewhere between the Innovator and Early Adopter stages. This is still a rather unproven area, and it may not match up with your standard risk profile of technology adoption. Since the big data segment is in such an early stage of adoption, it’s also unlikely that many of your competitors have a coherent strategy for exploiting big data, unless you are in an industry with high levels of technical innovation.
You also should look at your current information architecture and compare operations with the techniques of big data, primarily in the area of data integration. You may be pleasantly surprised to find out that you are already performing some of these processes, perhaps on a smaller scale. This is important because one of the major constraints on big data adoption is the lack of skilled staff members to process the mountains of data available. Having current staff available that understand some of the concepts is an advantage.
Become versed in analytic concepts
The momentum of the reporting and analytics industry has been moving from creating large central data repositories for data analysis to a mode of reporting and analyzing data wherever it resides. This shift has an effect on the traditional information lifecycle, and big data only accelerates this effect (see my previous post).
The increased focus on analytics means that more and more of your staff will be communicating in analytic terms, whether performing traditional analytics or in the new area of determining what data to utilize from external sources. This means that executives that understand statistical and analytic concepts are better positioned to effectively manage the new analytical enterprise.
Perform cost-benefit analysis on every source
The cost of acquiring and pre-processing big data sources is relatively high since one of the distinguishing characteristics of big data is that it is often incomplete, and it may not have enough identifying characteristics to be useful. This is true even if the new data is aggregated, as is usually the case.
For any data source, you should evaluate the quality of data at the outset of any project. All new big data sources should be profiled in advance of formal acquisition, determining its ultimate value to the organization (including “soft” benefits that may not have an adequate definition at the time). You can compare this value to the cost of acquisition and pre-processing. Your organization then needs the discipline to only move forward with data sources with positive cost-benefit ratios.
Employ trusted advisors
Big data concepts are evolving rapidly, so it is virtually impossible to keep an effective handle on the situation and attend to all of the other responsibilities modern executives face. So, it’s prudent to assemble a team of trusted advisors to assist you in making the best use of big data, whether you choose to proceed or not. This team should consist of both internal resources that understand your business and external advisors familiar with data acquisition, analytics and management. Both groups should understand the dynamics of the big data market space. The assembly of this team is an effective way to derive maximum utility from big data sources at the optimum time for your organization.
The most important thing to remember is that big data is much like any other new, shiny technology that has emerged over the past fifty years. There is tremendous promise and opportunity, but executives must weigh this promise against the cost of adoption and integration, ensuring that it is used in the most effective way possible.
