If you work in data quality long enough you’ll meet detractors of data quality software. The viewpoint in this camp is that poor quality data should be driven out at the time of design, not retrospectively detected and fixed. They perceive data quality tools as a costly overhead, something that is superfluous in a well-designed information landscape.
In a perfect world, perfectly designed systems would have defect prevention built in. All of the various design authorities would share the same vision for absolute quality management and developers would follow a strict "data quality rule book" and never create code that allowed defects to emerge.
Back in the real world we know that while this perfect nirvana is an admirable goal, it is nevertheless a fantasy. The average organisation simply lacks the ability to make these kind of design changes in a timeframe that can impact data quality in the short-term. Data-driven projects come in late, over-budget and with lots of corners cut. Tactical design solutions are taken, and there are a plethora of improvement projects that get cancelled in the face of competing budgets.
But, of course, we can't forget that business users and customers are experiencing real pain, right now, and poor data quality is so often the cause.
An example of real frustration with data quality was typified in my personal story of the duplicate healthcare identifier. It was clear to me that the "system was broken," and the organisation actually had to create an entire process and team to resolve the issue. This took many weeks, and if this was a business where I could vote with my custom I would have clearly looked elsewhere for healthcare services.
But what is the reality for an organisation when trying to fix an issue as widespread as this by adopting the "defect prevention" strategy proposed by data quality software detractors? It is simply impossible for an organisation of that size to turn around design flaws that stem from years of implementation.
What could they have done instead to prevent the issue I experienced?
Data quality software could have helped track duplicates nationally so the mistake could have been identified in a much faster time-frame. If the problem had been spotted early then the identifiers would not have flowed through endless patient booking systems in at least four healthcare institutions.
Is this the most desirable solution? No, of course a design change would have been preferable but for the vast number of organisations I meet, data quality software can serve an absolutely essential role that more than compensates for the initial outlay required.
A tactical solution? Yes, perhaps, but the benefits are definitely strategic because they align to improving customer service and driving down costs.
What do you think? Is prevention the only cure? How have you deployed data quality tools? Please share your views below.
