Data-driven design, Part 2


Part 1 of this two-part seriesBusinessman explains lessons learned as he worked toward being data-driven suggested a definition of data-driven design and described how technology is driving a change in mindset about how data is captured, prepared, integrated and used. In Part 2, we'll concentrate on lessons learned and things that could go wrong as you embark upon data-driven design.

You've probably seen articles about business users doing their own data preparation for an analytics project. (I'm pretty sure we wrote a few of those articles!) Some business users have skill sets that give them the luxury of pairing together data from multiple sources and creating what they need quickly. And their work may be funded by projects that require the team to do the data work without (sometimes) including IT.

Drawing from real-life lessons learned by a few of my clients, let's look at some things that could go wrong as you strive to be more data-driven.

Data redundancy

With no governance and control on where data is moved, you could end up with the same data in different data stores across the enterprise – which could be used incorrectly. It is important that we suggest using enterprise data assets, and meet the needs of the users without moving data around. For example, if you are creating a data lake, we might suggest that we look there first. Most data lake implementations are getting data feeds from our enterprise governed data assets. Usually, some sort of view may be created, based on our business users data requirements. The business user may want streamed data from a portal, as well as, strategic data for a join to analyze the speed that an inquiry is completed.

Data overload

I'm thankful that we have software to profile and inventory data as we bring it into our big data platforms. This allows us the insight required to determine how to use the data. Without it, we have no way of knowing what shape the data is in, where it came from, or how to assess its quality and integrity.

No governance

There may be some early-on data that was loaded onto our big data platform that did not go through the profiling/inventory software. This data needs to be identified and processed correctly. Why did this happen? Because at the time, we weren't quite ready to govern this data – or we didn't have data governance software to help us. This sort of thing happens – we just need to remember to fix it.

No business rules on data usage

We're living in a corporate world where business rules are not always documented, and may only live in the mind of someone at the organization. That said, the data misuse will occur if there is not an enterprise understanding of how the data is used. Profiling and doing inventory of the data will help to a certain extent, but it won't alleviate the misuse of data. Financial controls and balances will still be required for any data pertaining to dollars that flow through our big data platform. Without these controls, you risk reporting incorrect regulatory data.

Business users – get the data quickly

Most of my clients are using a big data platform for analysis but are continuing to use enterprise strategic data assets for corporate reporting. I believe this is a good way to get data to business users faster without compromising any regulatory requirements. Our enterprise data assets are governed – and hopefully managed and audited – for completeness, viability, regulatory standards, etc. Relying on these assets allows us to get the data to business users faster, while continuing to do corporate reporting that usually does not require real-time data.

Download a paper about data management in the real world


About Author

Joyce Norris-Montanari

President of DBTech Solutions, Inc

Joyce Norris-Montanari, CBIP-CDMP, is president of DBTech Solutions, Inc. Joyce advises clients on all aspects of architectural integration, business intelligence and data management. Joyce advises clients about technology, including tools like ETL, profiling, database, quality and metadata. Joyce speaks frequently at data warehouse conferences and is a contributor to several trade publications. She co-authored Data Warehousing and E-Business (Wiley & Sons) with William H. Inmon and others. Joyce has managed and implemented data integrations, data warehouses and operational data stores in industries like education, pharmaceutical, restaurants, telecommunications, government, health care, financial, oil and gas, insurance, research and development and retail. She can be reached at

Leave A Reply

Back to Top