“Field of dreams warehouse”– a historic phrase I used in the early days of data warehouse development. It describes the frenzy of activity that took place to create enterprise data infrastructure, before the business rationale for the data use was even understood. Those were the early days.

In some ways the Hadoop rush has parallels, beginning in a similar fashion. Many organizations have switched on Hadoop clusters to collect any and all data that may possibly have value, because they know they can sift through the noise with technology to discover the golden nuggets. A significant difference, however, is that Hadoop provided an immediate cost reduction strategy and so has had the benefit of a more tangible, measured value.

But a predefined, clear data strategy was generally not at the forefront of either of these initiatives. Build it and they will come … well, not necessarily. Store it and we’ll find use for it later … maybe so, but perhaps not.

So how can we turn any such field of dreams initiative onto its head? How can we create a dream data warehouse – one that is useful for fielding any organizational need? What does enterprise data management need to become so that it will be agile enough to address any current and future business needs?

It starts with the data.

The sources, the formats, the accelerating velocity of volumes that will soon overflow data lakes. Storing first and deciding what to do later won’t be a long-term viable strategy. In the IoT, we have educated guesses about the extent of data we’ll need to wrap our infrastructure around. But right now, even those are estimates. We are only beginning to imagine the efficiencies, just starting to realize the cost reductions and envision the opportunities that exist to monetize and use IoT data.

In the IoT, the things are distinguished by their data. What they do, measure, transmit and communicate – it’s all part of thing DNA. Instead of subscribing metadata to data elements, data will self-define, auto-select and automatically transform based on the guidance of data governing policies and application usage.

Creating a dream data warehousing environment is within reach.

Learn more about what’s coming from the Future of Data Management paper.