At every conference I attend it seems that everyone is talking about Big Data, yet very few organisations are truly applying Big Data principles to their businesses. This is still pioneer territory. It’s fair to say the technologies are in their infancy (by traditional IT measures).
I did meet one employee of a large telco recently who confirmed that they’re actively looking at ways to analyse sentiment in vast amounts of social media data. This seems to be a popular route to take for those getting started with Big Data. By analysing clickstream traffic, social media data and other vast data sources, companies can start to infer meaning and behaviour.
The problem I see is that most people misunderstand what Big Data is. It isn’t just more of the same relational, structured data. It’s completely new and alien to the enterprise in most cases. It requires a new way of governing, assessing, integrating, storing and interpreting.
The key differences I read about in the cases of companies who succeed with Big Data is that they are truly “information-powered.”
Take Google, for example. Every single element of their web design is tested millions of times using Big Data analytics. What colour should that click-through button be? How many search items should be displayed? Where is the best place for Google Ads - right banner or footer? They have an end in sight for their Big Data exercise: Find out the optimal design to maximise user experience and ad spend.
Amazon was able to transform its IT infrastructure by analysing Big Data to understand how its data centres performed at a minute detail. As a result they discovered huge improvements in cost and utilisation. Again, Amazon had an end in sight: reduce infrastructure costs and increase utilisation.
These companies are “information-powered.” The decisions they make are driven by data. The same just isn’t true of most companies and I believe these are the ones that will fail the “Big Data journey” because it’s not a technological journey, it’s a philosophical one. We have to change the philosophy that most senior leaders adopt which is largely “listen to my instincts.”
Phil Simon made a great point recently in his post “5 Things I Know About Big Data” when he said:
“An organization that keeps its Small Data in respectable shape seems more likely to embrace Big Data”
This is so true. If you’re considering the Big Data route but haven’t already got your Small Data in order I think you’re in for a very interesting journey.
3 Comments
Thanks for the props, Dylan. I'm totally with you on Google and a/b testing. When you have a sufficiently large sample size, why leave something to chance?
It isn't just about how executives make decisions, either, but all up and down the organization. In healthcare, for example, we have a huge population of clinicians who believe strongly in their training and instinct, and an industry that is being driven hard toward information-driven decision making. It's a challenging new world to define.
@Thanks Phil, it's so true. If a company can't even get the basic decisions underpinned by small data they're stuffed when it comes to Big Data.
@Paul: Thanks for the comment, I think Healthcare must be one of the toughest to govern for the reasons you cite. Ironically I just did an interview that discusses the impact of poor document governance in the UK health system: http://www.dataqualitypro.com/blogpost/703684/153679/How-to-Improve-Healthcare-Data-Quality-Through-Policy-and-Procedure-Management