I’ve had a lot of discussions with business leaders around the discrepancy between big data investment fears and successful use cases. Most of them say that "the quest for the golden use case" takes too much time and is usually not successful in the end. Ultimately, this quest can lead to paralysis and unpleasant questions from executives.

Everyone I talk to agrees that allowing experimentation is key to changing a culture and enabling digital transformation. During these discussions, the concept of the big data lab drew interest as a pragmatic way to drive innovation forward.

The big data lab as an icebreaker to drive innovation

Laboratories are dedicated spaces for testing and experimenting. The use of a laboratory is a recognized practice when investigating ingredients for a specific formula, determining how to generate a desired effect, or establishing which combination of materials results in a new robust product.

A laboratory is a separate area in an organization, where experts like chemists or physicists can experiment to test their ideas and assumptions or to prove expected effects. Exactly this kind of environment is also useful for identifying reasonable use cases for big data. The ingredients are new data, new technologies and new skills.

We know our well-structured data in our warehouses, but we don’t know much about our unstructured data, the additional data that comes out of digitized processes, or streaming data from the Internet of Things. We know our BI and OLAP tools but we may be missing experience on emerging technologies like in-memory, visual analytics or Hadoop.

These are precisely the issues that can be addressed with a big data lab. As in a traditional laboratory, the big data lab is not connected to the production processes or the factory and contains all the necessary prerequisites to handle big data (storage like Hadoop, data management, visualization, analytics). The lab also includes experts to help business users execute their experiments and to provide the necessary skills to orchestrate the diverse technologies.

Business departments can approach the lab with new ideas or hypotheses. Together, with the data experts and data scientists, needed data is acquired in the lab. By collecting, enriching and analyzing this data, ideas can be explored and validated, and any new questions can be investigated more deeply until the idea is validated, sorted out or replaced by an even better insight. After being proven through the laboratory exercise, the idea will be elaborated into a business case to start a project and to implement the results into affected business processes.

3 main reasons to have a big data lab

Why do we need this extra step in the process of implementing big data use cases? During my discussions with organizations we agreed on three main reasons:

Eliminating the resources spent in the “quest for the golden use case”

I have talked with many companies that initiated virtual innovation teams, conducted workshops, hired external consultants for a lot of money but still have not been able to find the use case to convince managers to invest in big data.

Missing concrete requirements or quantifiable expectations are the characteristics of big data use cases. Therefore it’s hard for organizations to determine business cases and to justify investments in advance.  Plenty of time is invested in trying to identify those parameters by using traditional methods or supposed experience from external and expensive advisors. A CIO of a large German manufacturing company made me aware of a potential fatal consequence: instead of using the immanent enthusiasm of people for new challenges and technologies, the results of the workshop marathon on “searching the use case” included the resignations of the people involved with the project.

By turning this process around and changing the mindset from “we only invest money if we gain a short term ROI” to “allowing experiments in order to identify new insights,” the process of innovation becomes more agile and creative. As a result, organizations know much earlier, whether to further invest in a use case or whether the idea should be sorted out.

Reducing expenses when integrating innovative big data technology

Integrating required technologies into your existing environment is a complex, expensive and time-consuming exercise. Since there is little knowledge in your organization on most of these new technologies and little experience on what is really needed to build big data into your infrastructure, a lot of pre-investment would be necessary to set this up across departments.

The big data lab allows companies to experiment with new technologies outside of their production environment. The lab not only enables clear visibility into the right use cases but generates a lot of experience on additional technologies, including how to incorporate new technologies into existing architectures. By experimenting during a couple of use cases, organizations will generate an increasing solid picture, which of the new methods and technologies would be valuable for their business.

Addressing the skills issue and building a big data service

We are experiencing a shortage of skilled people who can work with the combination of big data technologies. Needed skills include: managing big data with new technologies like Hadoop, understanding streaming data, and preparing and analyzing large volumes and new types of data with advanced analytics technologies.

Data scientists are in short supply and have to be developed for the digital transformation. The big data lab approach could provide a perfect place to locate appropriate staff to spread this capability across for the whole organization. Therefore the big data lab could provide a full service to the organization for experimenting with big data ideas. At the same time, ideas coming from one side of the business can be shared with others, enabling a virtual collaboration between parts of the organization.

Equipped with the necessary techniques and the right skills the big data lab will evolve into an incubator for innovation through big data.

Innovation, experimentation and enthusiasm

The discussions during the last weeks support my conclusion that the big data lab is the right approach for most organizations. Innovation requires experimentation. Experimentation requires enthusiasm. Enthusiasm is driven by speed, teamwork and fast results. Exactly this challenge will be addressed by a big data lab.

What are your thoughts on a big data lab? Could you see a use for it in your organization? Follow me on twitter @AndiGoedde to start a discussion.

Learn more about the SAS Big Data Lab, or watch this video (in German) to see the concept in action.