Yes, "Big Data is cool" but "all the cool kids are doing it" is no reason to forget about two of the most fundamental practices of data management: data archiving and data purging. Data management professionals need to work with their business and leaders to determine when data is no longer needed. We then need to work with our other technical teams to determine the most appropriate data archiving and purging strategies. I strongly suggest having these strategies & guidelines set forth both at the enterprise level and operational levels of an organization.

Isn't disk storage cheap these days? So what's the big deal?
The whole "disk is cheap" is a bit of a pet peeve for me. Yes, the cost of disk storage per byte has decreased, but if you ask anyone who has to actually write checks for storage and all the associated costs, they will tell you that the total costs are still significant. The amount of data that organizations store has skyrocketed and the associated costs for production-quality storage solutions, backups, disk administration, etc. are still a significant enough budget line item that most companies find paying for these things painful.

You have to look at the complete picture for "storage costs," and most people don't do that. They think that buying a terabyte of cheap storage at the local electronics store only costs a hundred bucks, so "disk is cheap". The reality is most organizations continue to spend a significant amount of money on storage management as a whole.

But haven't you always said "Data is an Asset"?
Of course data is an asset, but if you keep data around that is no longer of immediate value for you, it becomes a liability. How do you know when this is? The minute the data no longer adds value to your operations (including your analytical operations), and you no longer have a legal obligation to keep it the data, it should be removed from your systems. If this seems like a scary proposition, consider archiving it to less expensive storage solutions and purging it from the more expensive storage and online systems.

Do You Have To Let It Linger?
It's the new year and just like many of us are considering slimming down our waistline, it's time to ensure we have the correct processes in place to slim down our databases. Let's trim off the fat which is no longer of value for your organization.

Until next time...Rich