20 encounters of the information management kind – #3 New data (to file or not to file)

0

To allow file uploads into your business intelligence environment or not to allow file uploads – what is your decision?  I have a tendency to look at this issue like this:

  1. Someone on the business side needs this data – and probably needs to looking at existing data another way.  For example, rolling up sales data to different sales regions or categories.
  2. The data may not "age" well for integration in the data warehouse.  In other words, it may not be needed in two years, because our business changes so quickly.
  3. The load process requires automated data checking and validation.

Here's how to address these problems to make this a non-issue:

  1. Understand the needs of your business users, and change the data strategy to meet the business needs (that are always changing).
  2. Create a table in the staging area to load the file into, and consider this a place to validate and check the data for errors.
  3. Never integrate this data with subject areas like Customer or Product.  But, instead create a data structure that allows you to "pull" data from Customer or Product, as well as the table you created for the uploaded file data.
  4. Consider profiling the data after it is loaded into the staging area, and generate any error reports.  Make sure the data is OK before it is propagated any further in the data warehouse.
  5. Understand how and why the business users need this data.  What if this data drives some important business for the company?  Would you consider the upload then?

 

Tags
Share

About Author

Joyce Norris-Montanari

President of DBTech Solutions, Inc

Joyce Norris-Montanari, CBIP-CDMP, is president of DBTech Solutions, Inc. Joyce advises clients on all aspects of architectural integration, business intelligence and data management. Joyce advises clients about technology, including tools like ETL, profiling, database, quality and metadata. Joyce speaks frequently at data warehouse conferences and is a contributor to several trade publications. She co-authored Data Warehousing and E-Business (Wiley & Sons) with William H. Inmon and others. Joyce has managed and implemented data integrations, data warehouses and operational data stores in industries like education, pharmaceutical, restaurants, telecommunications, government, health care, financial, oil and gas, insurance, research and development and retail. She can be reached at jmontanari@earthlink.net.

Leave A Reply

Back to Top