sascom voices - The sascom magazine blog

There will never be a single, omniscient master file. What there can be, however, is a data integration process that draws together all of the available data, performs data cleansing and merge resolution functions, and sets up routines to make sure that process continues as new data become available from those systems.

But data integration is not the stuff of headlines. For most people, data Integration is very boring stuff. Data integration is the stuff of IT journals, computer science courses, and data governance seminars held at the budget hotel by the airport. To tell the truth, even those of us charged with data integration tasks recognize that data work can be mind-numbing and thankless.

The IT staff charged with merging two data sets will be the first to have to report that the two systems, which everyone had assumed were basically parallel and almost duplicate data, don’t match at all. The IT staff will be the ones who need to find a way to determine if the two twelve-year-old boys named “John Smith” in Capitol City are really a duplicate entry or the same boy. The IT staff will be the ones who find a way to merge the massive, older FORTRAN mainframe records with the newer, sparsely populated MySQL database built by a programmer who left last year. The IT staff will be the ones who document every minor irregularity in metadata so that an analyst who will use those data next year will know that “Flu1” and “Flu1a” are actually the same variable, drawn from two different databases, one represented as numeric and the other represented as alpha and, between them, there is only an 87% match. That is data integration.

Statistical Consultants get bonuses for creating remarkable statistical models that make highly accurate predictions. Report Writers get popular creating beautiful dashboards and graphs that make the data easy to understand. data integration staff get ulcers, absolutely certain that the update to the SQL database that houses Table 456.2B is going to invalidate the automatic merge algorithm used to correct for weaknesses in the old Table 456.2A extract.

Data integration tools have done much to increase the speed, ease, and efficiency of data integration. At the same time, databases and software have become more complex with larger systems and more one-off, undocumented custom builds floating around the enterprise. All of these must be brought together if you want a denominator you can trust.

William Carlos Williams, the American poet, once wrote:

so much depends
upon

a red wheel
barrow

glazed with rain
water

beside the white
chickens.

Sometimes the fate of the world can depend on the most ordinary things. Simple things that seem almost irrelevant. Things you would think you could take for granted.

Today, one of the most important pieces of information medical experts need to know is a denominator: What is the total number of people who may have contracted swine flu? Meanwhile, in countries around the world, in basement offices and cubicles and even on laptops in a conference room that just happens to be empty from 10-2 today, IT professionals are working on data integration processes. That work may be exceptionally frustrating and time consuming. But developing a process for getting the most complete and accurate model of all of your available data is critical to being able to respond to opportunities… and emergencies. And nothing is more important than that.