With all the hype over big data we often overlook the importance of modeling as its necessary counterpart. There are two independent limiting factors when it comes to decision support: the quality of the data, and the quality of the model. Most of the big data hype assumes that the data is always the limiting factor, and while that may be for a majority of projects, I’d venture that bad or inadequate models share more of the blame than we care to admit.
It’s a balancing act, between the quantity and quality of our data, and the quality and fit-for-purposeness of our models, a relationship that can frequently get significantly out of balance. Or more likely, complete mismatches between data and modeling can crop up all over our organization. In one instance we may have remarkable models starved for good data, and on the other hand, volumes of sensor or customer data sit idle with no established approach to exploration, analysis and action.
This imperative to balance the data with the model reminds me of an espionage story from WWII. Read More