Some recent press articles question the value of big data while a book takes the opposite approach; I’ll choose the middle way. The New York Times article ‘Is Big Data an Economic Big Dud?’  questions the value of digital data and the resulting increase in the amount of data. This CNBC article points out that NASCAR hasn’t seen any impact to their bottom line.

These articles on the negative end of the spectrum are countered by a recently released book.

‘Big Data: A Revolution That Will Transform How We Live, Work, and Think’ takes the opposite perspective. This book argues that big data will transform our world and goes to the extreme making statements such as ‘with big data you don’t have to understand why a correlation occurs.’

The truth lies somewhere between these viewpoints.

Here are some thoughts and examples from recent projects where my consulting team has been directly involved with customers wrestling with these issues. Results we’ve seen at SAS support the idea of fundamental change underway in how we work with big data and the business results. Unlike the aforementioned book authors, I view the progress being made as an ‘evolution’, not a revolution. The market is already multiple years into working with in-memory and in-database technologies. It’s not as if there has been an overnight shift.

Utilizing in-memory technologies, along with time-tested predictive algorithms, our clients have seen an orders of magnitude decrease in time to do their analyses. One large customer saw several different predictive models decrease from taking hours to run down to minutes, and even seconds in some cases.

In addition to reduced number-crunching time required, our clients are also now able to work with larger amounts of data. Given this ability our clients can work with entire databases instead of subsets of their data. One implication is the ability to model an entire credit portfolio, and to do so in a shorter time window, than previously possible with only a sample of that database.

Now that we are able to leverage a larger volume of data at increased velocity we can do more analyses, looking at more and different business problems. We can consider modeling entire databases without sampling in these big data environments.

Though we are in the early days of the big data evolution, it’s clear the book ‘Big Data’ is more on the mark than the negative articles that have recently been published. If one considers we are at the ‘crawling stage’ in working with big data, and already seeing orders of magnitude changes in working with data, one can only imagine the impact on applications like data mining, forecasting, text mining, optimization and quality control.  In turn business will realize quicker time to value, increased profit margins, reduced costs, and more satisfied customer by leveraging big data and the technologies to work with that data.