When tackling data quality, a lot of companies focus on the problems they can fix in the short-term. If they can clean data they will. If they can fix a broken process they will. If they can improve the logic of data validation logic they will.

Quite often, people won’t stop to think of the broader issues surrounding these problems.

• Why did the data become invalid in the first place?
• Why was there a broken process?
• Why didn’t the validation logic work as expected?

I find that most companies are concentrated on solving data quality dimensions that relate mostly to values, but often the bigger problem is one of architectural failure.

Hand on heart, I truly believe data architecture is one area where we have failed to progress as an industry.

In the eighties I was given a thorough grounding in data modelling at University, and in my first role every data analyst in the organisation was adept at data modelling to a high standard. Whenever we received data from an external organisation we learned to create physical and logical data models that were compared to our enterprise model for gaps.

What I discovered is that over the years, as I moved around various organisations, this same kind of rigour was far less visible. Less emphasis was placed on core architectural techniques such as not only maintaining accurate data models but using modelling as a core technique to design the right data structures for the task in hand. I often meet data analysts today with no modelling skills whatsoever and that is deeply worrying given their importance.

Documentation is another grey area for many organisations. I was initially trained by one of the software engineers who designed flight control systems for commercial airliners. I still remember the "hairdryer treatment" I once got from him when he discovered that my software design documentation had a number of major gaps in it. He came from a background where poor documentation could lead to fatalities and it was a lesson I never forgot.

Far too many organisations pay lip-service to good documentation, and there are always major gaps missing that only come to life when you need to really understand the functionality - perhaps during a data migration or merger and acquisition activity. I think the trend to outsource system development or buy off-the-shelf solutions only exacerbates the problem.

So, with modelling and documentation, we’ve barely scratched the surface of modern data architecture. But it’s clear that many organisations are failing just in these areas alone and as a result are creating data quality defects.

Let’s imagine one of your web developers creates an excellent data validation routine to prevent poor quality customer details being captured. Is that process being documented and shared across the organisation?

When you buy new SalesForce accounts and add Cloud-based solutions to your enterprise architecture, are you reflecting these new systems in your enterprise models and logical models?

When data quality defects are observed to stem from architectural failings are you quickly able to discover copycat issues elsewhere in the organisation from looking at documented architecture patterns?

In summary, now more than ever, organisations need to invest in some of these traditional data management disciplines. Otherwise, you’ll never get off the data quality ferris wheel. If you’re seeing the same type of issues repeated across the organisation, take a look at the broader picture or you’ll be firefighting for a long time to come.

What do you think? Are core data management disciplines like data architecture slipping behind or am I being too critical?