Clean-up woman: Part 2


In my last post, I talked about how data still needs to be cleaned up – and data strategy still needs to be re-evaluated – as we start to work with nontraditional databases and other new technologies.

cleaning supplies for clean-up womanThere are lots of ways to use these new platforms (like Hadoop). For example, many organization are using a data lake as a staging area for minimally processed data. That said, this may be the place used to analyze and profile data. Not only for quality of data, but for integrity and completeness of the data between processes. Kind of like an audit area. For example, maybe we made a change to configuration of a source system, and we want to make sure the data coming out of that system is processed according to the testing and/or use scenarios.We could store data in many formats in the data lake. It could be XML or structured data. Technology is changing so quickly; it has become difficult to create a strategic vision.

We could also use the data lake as an archival platform. Heck, they do say it's cheaper – right?

If we do start loading up the data lake, information about the data residing in the lake will become very important. I would still call that "metadata," wouldn’t you? So, how the data is used and who is using the data are still very important questions.

To manage this environment will still require guidelines, governance or whatever you want to call it. We cannot just dump stuff in the lake, give everyone a fishin’ pole and say HAVE AT IT!

Issues like this arise: Do I still need tool platforms that include a combination of data integration and data quality? Must MDM still be considered?

In our fast-moving technological world, getting data to the right people, in the right amount of time, with the right amount of clean-up will be key to business prosperity and expansion. New platforms and self-service are just the tip of the iceberg.

Download a paper about data management for analytics best practices


About Author

Joyce Norris-Montanari

President of DBTech Solutions, Inc

Joyce Norris-Montanari, CBIP-CDMP, is president of DBTech Solutions, Inc. Joyce advises clients on all aspects of architectural integration, business intelligence and data management. Joyce advises clients about technology, including tools like ETL, profiling, database, quality and metadata. Joyce speaks frequently at data warehouse conferences and is a contributor to several trade publications. She co-authored Data Warehousing and E-Business (Wiley & Sons) with William H. Inmon and others. Joyce has managed and implemented data integrations, data warehouses and operational data stores in industries like education, pharmaceutical, restaurants, telecommunications, government, health care, financial, oil and gas, insurance, research and development and retail. She can be reached at

Related Posts

Leave A Reply

Back to Top