Companies across the globe perceive bad data quality as a negative. We use terms like “cleansing” and “defect removal” to describe some of the core activities involved when improving data quality.

The problem is that many people don’t stop to ask, “How did our data get into this state? What is this poor quality data telling me about the people, processes and technologies that helped create it?”

This is a real shame because unlocking that knowledge can often create considerable opportunities for increased profits, morale and operational efficiency.

Why aren’t the secrets of bad data being unlocked?

If you look at most business intelligence (BI) surveys as an example of data usage, one of the biggest challenges cited is always - drum roll, please - data quality.

Why is this?

It’s because BI teams operate downstream of operational data, so they often take a “fixer-upper” mentality. It takes too long for them to go back to the business and make the required changes. If they can see a pattern in the data that is wrong and it can be corrected with a simple rule, why not change it and keep everyone happy?

This also happens on data migration projects. When faced with fixing bad data many teams simply shunt data into the target and ask the business to deal with it.

Bad data quality is seldom mined to understand the underlying problems simply because people think “it’s not our job.”

As a result, this short-term, reactive thinking is damaging your organisation in so many ways.

But what does the alternative look like?

What are the benefits of discovering the true stories of bad data?

Several years ago I consulted on a telecoms system that contained some less-than-perfect field operations data. The client was looking at ways to clean up the information for a data migration but also create better quality reports on engineer performance, fixed asset costs, contract charges, etc. After finding many issues I started to pull some of the threads that related to this bad data story.

In one of the notes fields on the main engineering assets table I found a lot of coarse language. (I had created an expletives search tool to find the questionable comments - great fun and I recommend you develop one, too!)

What I found was a hidden story in the data: engineers hated the system.

Imagine you’re cold and wet, out in an English ditch on a winter day, trying to splice severed copper cables together. You then have to go back to base and record the new equipment layouts that you’ve implemented. Would you tolerate a system that foiled you at every step?

The design of the system was counter-intuitive to the way engineers worked. They hated it with a passion and weren’t afraid to use “fruity language” whenever they had an issue.

Just by listening to the underlying story of the bad data I was able to get a much clearer view of why this system was so loathed - and, more importantly, how the new system should address their concerns.

Next Steps - Unleash the data quality whisperer in you!

So how can you transform the way you approach bad data?

It’s simple: find the hidden story.

Track problems back to their source and ask questions of those who created the data.

Did they have enough training? Are there conflicting performance incentives? Is the application a pig to navigate and enter information with? Was the data they need already present? Could they understand it?

The next time you’re confronted with dirty data, pull the thread of its life story and see if you can unlock its past.

You never know what you might find, but I guarantee it will always help your business become smarter and more profitable.

Post your favourite "data quality whispering" story in the comments below.