Recently I have been out speaking with a number of organizations about the idea of the innovation lab concept , which I discussed in a previous blog post, as the way to unleash the power of big data and make even the largest of companies as agile as a startup. During my discussions there are a couple of things that I am observing, that I wanted to share with you, since it seems there are different types of innovation labs in organizations:

1. Many companies have something I am going to label an IT innovation lab where they are experimenting with "big data" technologies. These IT innovation labs are NOT the same as the “data related” innovation lab that companies need to put in place to remain agile in this new digital world. The focus of the IT innovation lab is to test the technology and its integration whereas the focus of the "data related" innovation lab is to test hypothesis around mashups of data and different analytical approaches In the digital world, information gleaned from data is your best competitive weapon, and speed is a critical component to your success.  It is my opinion these should be separate and distinct in a companies strategy as each has a role to play. This post will focus on the tale of two labs and how they differ.
2. Tight budgets, and the significantly different focus of the “data focused” innovation lab, are causing organizations to ask for support to obtain funding given it is generally a new concept to not have a concrete business problem solved as part of asking for an investment. A second post in this series will focus on how I suggest organizations can build the business case for the data focused innovation lab which I believe is vital to the future success of all organizations no matter how large or small they are.

Separating the labs

In most organizations there already exists a lab where IT tries out new technologies to assess their fit within the broader architecture of the organization and their ability to integrate with existing technologies. These labs are generally IT focused playpens used as part of the selection process for new technologies in organizations. They are also the first thing I get pointed to when I ask about the data related innovation lab.

With a plethora of technologies purporting to deliver big data capabilities, into their existing infrastructures, it is clear that IT wants a playpen to ensure that the new technologies deliver what they say they can in terms of integration with existing security systems, integration with existing data stores and integration with existing end user tools for example. In essence they allow them to see the impact of any procurement that will touch their broader enterprise architecture.

Most organizations know they have to start to tackle the big data challenge. This has led to a clamor for use cases that might be applicable to show the value of big data with the best being selected and trialed as companies try to show how they can benefit from big data. Because the IT innovation lab already exists, in most organizations, it has become the natural start point for anything big data related. The issue is that the IT innovation lab might have the skilled people to stand up an environment that can support data driven innovation but the rest of what it is focused on differs vastly from that of the data driven innovation lab.

This chasm between the two world means that, in most organizations, big data projects are being treated as IT infrastructure testing and use case proofing generally in a one-off manner and not as a service. It also means big data projects and hypothesis testing are being slowed down by controls put in place to ensure smooth operational deployment of technology proven to deliver value. Lets look at how an IT-driven lab works when it comes to trying out new ideas. This is how the process generally goes:

1. Someone decides that a big data project is needed
2. An idea is identified and a use case is generated
3. A business case is built to justify the project and assuming it passes a review (often as part of a broader council), then a budget is allocated and a project is initiated.
4. Next a team is formed to identify the right technology with a heavy IT focus.
5. The team decides new technology is needed to solve the business issue.
6. IT wants to enable the business to try the new technology but also wants to ensure it works in their existing environment. In fact this check on if it will work in the existing environment is the primary purpose of the IT innovation lab.
7. An RFP is created that contains BOTH sets of requirements - for new technology / integrating into the existing environment and also for the specific needs of the use case.
8. A lengthy selection process starts. Often this process is run with 2 or 3 vendors being asked to do a PoC or a Pilot
9. Each vendor deploys into the IT lab and the vendors help to deliver the PoC often with little to no business involvement apart from presenting back the solution. Much time is spent on the building out of the infrastructure, integration aspects and data acquisition. Only then is time spent actually executing the business use case to explore the data and build out analytic models resulting in a very long winded process to obtain a result and see if the use case works out as anticipated.
10. At the end of that process business states which vendor they like and IT states which one they like. Often they do not 100 percent align. Sometimes the business requirements are not met and the project dies here with a lot of IT cost incurred.
11. Assuming the PoC or Pilot has shown some business return IT states their preferred vendor and the business either goes with it or they decide they do not like them and the project dies. The number of times I have seen a business team decline to adopt the chosen IT offering is not insignificant.
12. If it passes all these phases there is then a process to turn what was done into something that can be operationally deployed, it is tested and then we go live.
13. It is only at this stage we know if the idea really succeeded or not.

Sound familiar. Now, of course, you might say your process misses some of these steps, that it runs differently or that your process is cleaner. You may also say that this is how every “Big Data Project” starts out life in your organization – you would not be alone.

The point is really not to debate that process but to understand that this very process is what we are trying to short circuit with a data focused innovation lab when it comes to analytical style projects. In addition, the intention of the data focused innovation lab is not to integrate into what we are doing today (well…. apart from perhaps accessing corporate data sources). Instead, the point is to provide a platform where business can try out their ideas, on whatever software stack they deem appropriate as an ongoing service. And yes, we should aim to standardize that for a few reasons, without having to go through a whole round of IT related tests for every idea we want to try. To this end, the data driven innovation lab might produce an immediate result you can use or it will prove the value up front reducing the risk of failure having gone through a lot of intermediate time consuming steps. It becomes your business case proving ground.

Before all my IT friends scream at me, let me just say that I am not saying that IT requirements are not relevant. Clearly, there will be a point where ideas turn into things we want to take into the operations of an organization, and at that stage we need to ensure we have the right technology in place, the right security integration, the right integration overall and a proper IT support structure – if you like, this is the time to take something into the IT innovation lab as part of the process to productionize it as you will already know its value and impact up front. The good thing is that it will be much clearer what needs to be done in that process so even that testing will be relatively quick.

The question then is, should you use an IT innovation lab for experimentation or can we free the business from those constraints to let them innovate. At the same time can we reduce the overhead of having IT testing integration and pulling together data via strongly governed processes they will never use nor need in the majority of the cases?

Having spent time focusing on the IT innovation lab, let me say that my view is that while an IT innovation lab certainly has its place in an organization, it should not, in my opinion, be used as a proving ground for business ideas around big data. (Note that when I say big data, I mean data from inside and outside the org, structured and unstructured, traditional sources and wacky left field ones you might be ignoring today.) Inherently IT innovation labs are too infrastructure focused and often replicate a lot of the overhead of the operational/production environments which are exactly what is slowing us down in the first place.

An IT innovation lab does, however, have a place for testing things once you know they are valuable, in order to iron our kinks and avoid operational issues as you put into place any new infrastructure to operationalize the outcomes of your experiments. In other words, you need both worlds ultimately even if the IT innovation lab is nothing more than a development and test environment for things you need to harden before you put them into production.

The data driven Innovation Lab

The data driven innovation lab is well covered in my previous post, with respect to what it is and how to get started, so I will not repeat it here.

The core difference between the data driven innovation lab and the previously discussed IT innovation lab is that when it comes to the data driven innovation lab, you are putting into place a separate, and often disconnected, platform ONCE to enable all parts of the organization, through a service, to play with big data to see if they can find innovative ways forwards. This means you need an innovation lab technology stack and architecture designed for the rapid ingestion of many different types of internal, and external, data followed by exploration, discovery and analytics to prove or disprove any idea.

If we revisit the previous process it looks a little different.

1. We establish the data driven innovation lab which means establishing a platform for innovation at the organization. My second post will focus on how to build the business case to get that put into place
2. Someone in the organization has an idea of how your company can benefit from big data decides that a big data project is needed and that triggers the involvement of the innovation lab service
   1. They approach the data driven innovation lab and book an appointment to try out their idea
   2. Data is quickly sourced into the environment by one of the data driven innovation lab staff based on what the business identified they needed and respecting any security and privacy constraints
   3. Business users sit with analysts and data science staff from the innovation lab team and they jointly explore the data, try out their ideas and come to conclusions
   4. At this stage we may have an outcome already with no need for further hardening of the idea or we might have proven out the idea and want to start a service transition process to move the idea as quickly as possible into production.
3. Assuming we want to harden the idea we still need to:
   1. Build the business case. This time we know it is proven to work up front and can better state its impact
   2. Send the said business case to the council which can probably expedite it as due diligence has been carried out already
   3. We know the technology and data we need to take the idea production and as such we can focus the IT project on making that data and the relevant technology available into the IT innovation lab (if not already there) based on what was used in the experiment ultimately to get the result we wanted
   4. Once testing is completed and any issues ironed out we can go live and the solution is deployed into the organization

What I hope you can see is that the data driven innovation lab will dramatically help shorten the time we are spending doing IT heavy tasks and testing in the IT innovation lab. Essentially once the platform and service is in place, it should be made available to enable the entire business to try out new ideas working on data which could be things like adding open data to the data being consumed by an existing analytics model to see if it makes a difference or bringing together data sources internally that have never been combined to see if new insights are uncovered. The possibilities are almost endless and that is the point.

We need to provide a place for this nurtured experimentation with little to no IT overhead if we want to encourage experimentation. I am often asked for use cases for big data and I can share many of the ideas of what organizations are testing  but ultimately the best use case could be one locked up in your data and someone inside your organizations head. That nugget is something hard to copy whereas a generic use case will soon become the norm even if you have to do it too to remain competitive.

So just what is the difference?

In a nutshell, the difference is in the focus. In the IT innovation lab, the focus is really on proving/disproving the technology works in the current infrastructure, which means that is a core part of every project running via that lab where a business issue is often the cover for testing technical integration in the hopes of delivering business value at the end. In the data driven innovation lab, the focus is on proving/disproving ideas around the use of data and analytics quickly and with as little overhead as possible in a programmatic way. Keeping it away from the broader IT world helps keep it agile in its make-up and makes it a more business friendly place to work!

Next Up

In my next blog post, I will chat about how the business case for a “data driven” innovation lab can be made either alongside an existing IT innovation lab or as an investment on its own.

The market waits for no company in today’s digital age. Agility is the key to your survival and growth and a programmatic and pragmatic approach to big data is the way forwards. If you are interested in speaking to me more about this topic you can add a comment to the blog or register to meet with me at the Hadoop Summit. If you will be there, between April 15th and 16th in Belgium, SAS will be a key sponsor.