Why do so many data migration projects fall off the rails? I’ve been asked this question a lot and whilst there are lots of reasons, perhaps the most common is a bias towards finding the wrong kind of data quality gaps.

Projects often tear off at breakneck speed, validating and cleansing all manner of data quality problems, without really understanding the big-picture issues that can call into question the entire migration strategy.

Why does this problem persist?

Many project teams now have access to data quality tools and, quite rightly, deploy them at the earliest opportunity. It feels like the project is moving forward when profiling statistics and cleansing plans are created.

However, data profiling and validating the data in front of your eyes doesn’t tell you much about data or relationships that are missing, it just tells you the quality metrics for the information that is known.

There are a lot of other considerations, such as:

• Information found in systems that are not under scope because they were never considered important.
• Information requirements of the target system that have not been firmed up yet.
• Information relationships that exist between systems that are not discoverable electronically.

So we have lots of conceptual gaps that the project team often ignore. These gaps can cause a major derailment during your data migration initiative.

How do you incorporate conceptual gap analysis into your existing data migration framework?

We know that data quality tools are critical during data migration. We can use them to reverse engineer our data landscape and present back to the business a high-level view of the data assets under scope.

We then need to perform a range of conceptual modelling workshops with the business in a top-down approach so that we find all the high-level subject areas that the business believes will be critical to operations and services in the target system.

The conceptual modelling workshop provides the focus for investigation and the data quality tools (especially profiling and linking systems) and helps to find the gaps.

It’s this combination of techniques that helps project leaders ascertain the really big conceptual gaps that can often be virtually undetectable using data profiling and data discovery in isolation.

Likewise, conceptual modelling helps you build a more focused and prioritised strategy for your data quality tools so that they deliver maximum value.

What do you think? What are some of the big failure points that you’ve witnessed on data migration projects? Share your views below.