How do you structure your data migration plan? How do you allocate resources and timescales so that you maximise your chance of success? In this article I want to explore some data migration project planning misconceptions and give you some pointers for an alternative approach.

One of the big improvements with data migration in recent years has been the splitting out of data migration as a distinct project. Historically, data migration was seen as a bit player in the target application implementation project, so it was just bundled into the planning for the major initiative.

Today, projects treat data migration as a distinct process that is best managed separately but feeds into the goals of the broader program.

This obviously means that you get a separate team of project management staff to help coordinate the flow and governance of the project. That’s the good news. The bad news is that they often try to approach data migration as just another software project.

Unfortunately, that is the path to failure.

The typical software development approach to data migration goes something like this:

Start > Analyse > Design > Build > Test > Execute > Validate > Close

More and more resources are added as each phase progresses. On some projects I've witnessed 30+ developers by the build phase, which is frankly insane from a project management perspective. You just can’t coordinate that level of development and expect a quality result.

What you end up with is a huge imbalance in your project plan with a small amount of resource and activity at the start and a huge amount of frenetic energy at the end.

Obviously, this is not ideal. Instead, I prefer something like the following approach:

Impact Assessment > Start > Discover > Analyse > Prototype > Release > Repeat > Close

I believe the days of the huge two-year ‘big-bang’ migration project are numbered. You simply can't hit a moving target two years off. Even 12-month projects are incredibly difficult to plan for at the outset.

I believe it is far better to front-load your project with highly specialized technology, seasoned migration experts and solid deep business engagement to create a more agile series of iterative migration milestones.

The impact assessment before the project start is critical because you simply don’t know what will lie ahead. You need activities such as data discovery, scoping and prototyping to be taking place before project initiation so you know what you’re dealing with and whether the project is even feasible.

When the project kicks off for real my preference is for a very small team of highly skilled practitioners, as opposed to a cast of thousands. I've found that migration projects don't scale by adding more developers; the converse is actually true, in that it makes life far more difficult to coordinate and quality issues creep in.

Another key requirement for this leaner approach is to use the right technology. "Hand cranking" migration platforms using custom scripts and whatever homegrown tools the IT team have to hand is a recipe for disaster. You need to use the right profiling, discovery, data quality, data transformation and data validation tools to have any chance of delivering an agile, results focused migration.

So the lesson is a simple one. Swap tail end panic for front-end discovery, prototyping and iterative release.

But what do you think? Welcome your views in the comments below.