What does the future of analytics look like in your organizations enterprise architecture? Does it include thinking about a two speed approach to analytics which includes both:

• An agile rapidly changing analytics platform for innovation (a lab) seperated from operations and broad enterprise audience usage
• A slowly moving systematic enterprise analytics platform (a factory) that supports operations and a broad enterprise audience

Moving forwards with analytics in the enterprise architecture is an important IT topic at the moment. Working out how to innovate and modernize at the same time is one of the biggest challenges organizations are facing right now. In the past this has been like changing to new wings while flying the plane, which has always been slow and risky. There is potentially a better way to handle this moving forwards this blog explores.

Introducing Lambda architecture

Many of you are familiar with the Lambda architecture. If not then this Wikipedia page is perhaps a good place to start. In short, it’s a data processing framework which Wikipedia states:

“… attempts to balance latency, throughput, and fault-tolerance by using batch processing to provide comprehensive and accurate views of batch data, while simultaneously using real-time stream processing to provide views of online data”

Though not everyone’s a fan, Lambda’s been on the rise as an architecture for data processing for some time. The reason? There’s a clear need to handle not only regular batch updates of data, but to also handle rapidly arriving streaming data.

I won’t go into the weeds of Lambda architecture here, but if you’re exploring it, be aware that SAS provides a broad range of technologies to support your Lambda architecture efforts. Event Stream Processing and our wide portfolio of Data Management technologies, amongst many others, can help you consume, manage and govern your data.

The reason for starting the discussion with Lambda is that it got me thinking about the broader challenges in enterprise architecture around analytics in particular.

I started to wonder if those same principles of one thing moving quickly and another moving slowly could offer a solution to the challenge of bringing in the new analytics technologies and approaches of the day alongside established well-trodden paths.

As I’ve explored this further, I see real merits in the dual approach.

The modern organization

Today’s CIOs/CTOs face significant challenges. Unless they’re with a start-up, they generally have a legacy mix of IT systems supporting critical business processes, and they’ve added new analytical technologies over time.

This overall mix of analytic technologies and systems has gotten so complex and engrained into the organizational fabric, that there’s an inability to change it. That’s an obstacle to becoming more digital and exploiting the latest analytical capabilities.

Modernizing the "factory," as I like to call the platform supporting operations, is not as simple as just removing technology and replacing it with the latest and greatest.

To some extent this is akin to the batch data processing side of a Lambda architecture. Here we need time to make change, and any change needs to be carefully assessed to determine its impact.

Changes impact processes, people, culture and much more. It’s not just about the technology -- the integration points, the downstream systems and all the normal things we consider when we think about IT technology projects.

This is why change is hard on the factory side of things, and why many companies are unable to move quickly to modernize and transform into the digital organization they’ll need to be in the future.

For these organizations the question is: What can they do to overcome this inertia and reduce risk when they do modernize?

Supporting digital transformation

There are two approaches organizations can take to move forward:

Approach One: On one hand, you can work exclusively on the modernization of your enterprise architecture to update all your analytics to drive a better overall factory filled with the latest technologies and ways of working.

Many organizations are taking this approach. They’re trying to understand the capabilities of new analytic technologies on the market; react to the need to do something with "big data;" and look for cost savings and efficiencies. And they’re doing all of this while the same exercise is being conducted with the core operational systems which are also under scrutiny.

All of this is being done to remain relevant in the market, cut IT costs to free up budget for other things, support regulatory requirements and meet new business requirements. There’s nothing wrong with this approach. It all needs to happen-- but it’s a limited focus and you can’t expect to stay on the cutting edge since your forward movement will generally be slow and deliberate as your business is “in flight.” You might make a big jump once, but will you really do this at every generation of major capability evolution?

Approach Two: The second approach marries the ongoing aim of modernizing the enterprise architecture (the factory) with the addition of a second platform (the lab) to support analytics-driven innovation.

This second platform, often provisioned as something like an innovation lab, provides an agile environment to develop a dynamic, frequently changing platform free of the constraints of the more slow moving enterprise platform. In this secondary environment, ideas can be tried and incubated with a clearly defined set of services and processes to use the agile environment on an ongoing basis throughout the organization.

The secondary platform can provide the directional focus for deciding where to make changes to the analytics architecture in the main enterprise architecture for the best returns -- and give guidance on your target architecture as you make that change.

To some extent, the second platform is akin to the streaming or real-time side of the Lambda architecture.

Thinking forward - acting quickly

This dual approach will allow IT to meet organizational requirements for agility and innovation, while also ensuring that changes to the enterprise platform around analytics are well-defined and scoped, with a clear benefit lined up once that work is complete.

This is also similar to Lambda, which clearly states that the two worlds (fast and slow) have to come back together at some stage.

Your feedback wanted

I am interested in your thoughts around this – do you have other ideas on how to tackle this issue? Feel free to follow me on Twitter @mark_torr to see what else I’m passionate about.