"Temporary solutions often become permanent problems."
—Craig Bruce
For a long time now, many large, mature organizations have struggled with effective data integration.
Of course, this should surprise exactly no one. As I have seen firsthand in my career, organizations often cannot rid themselves of legacy applications, technologies, systems and mindsets. (For more on this, see Why New Systems Fail.) And one can easily apply that general statement to data integration tools, many of which organizations deployed in an era of small, structured and predictable data sets.
True data integration: More theory than practice
I can't help but think that having a bunch of disparate data integration tools inhibits effective modernization. Put differently, perhaps it's best to consider modernizing and/or simplifying your data integration tools. Rather than continue using a patchwork of antiquated processes and applications that don't talk to each other, why not deploy a portfolio of integrated applications? That might be the ideal, but it turns out that few companies seem to be doing so. As Philip Russom writes in the TWDI Checklist Report: Modernizing Data Integration:
A few years ago, a TDWI survey about next-generation data integration asked users if they’re “using a DI tool that’s part of an integrated suite of data management tools from one vendor.” Only 9 percent of respondents reported using one, although 42 percent would prefer one. The same survey asked what would drive users to replace their primary DI tool. The most popular answer was: “We need a unified platform that supports DI, plus data quality, governance, MDM, etc.”
This begs the natural question: What's stopping the deployment of new tools and applications that would promote true and contemporary data integration?
Generally speaking, I believe that the answers are more human and cultural than technological. The "ain't broke, don't fix it" mentality pervades quite a few organizations, especially ones that have been relatively successful. To this end, rare is the CXO willing to fall on the sword for a new and ostensibly unproven technology or application, even if he or she will admit that the status quo is suboptimal. An organization's data integration efforts may not be ideal, but they often work well enough. Why rock the boat?
Simon Says: Modern data integration starts with unity.
I understand that position but I don't agree with it. Data streams certainly aren't getting any smaller and simpler – quite the opposite. What worked in 2002 may kind of work now, but many legacy tools are sitting on a burning plank.
Piecing together a bunch of independent MDM applications, data warehouses and ETL programs with workarounds and Band-Aids just doesn't strike me as an intelligent way of doing things. Seamless connectivity via APIs (where possible, of course) just seems like the more sophisticated way to go.
As Queen Latifa knows, unity can be a good thing.
Feedback
What say you?
Get some tips from TDWI about data integration modernization.
