The case for automating data management

Artificial intelligence is everywhere these days. Forget Siri, Cortana, Alexa and their ilk. Robots are coming to food stores, financial services, law firms and just about everywhere else. Enterprises are using AI to automate their IT networks. The results are promising: AI can identify and fix IT issues faster than humans can.

If AI can cook for us, then it is downright preposterous to claim that organizations' data management practices are immune from the relentless tentacles of automation.

Today I'll make the case for automating data management.

Taking a step back

Before going too far, let's start by arriving at a common definition of data management. At a high level, it represents the practice of integrating, cleansing, migrating and improving enterprise data. Its goal is to produce consistent, accurate and reliable information that employees can use to make better business decisions.

Sounds easy, right?

Hardly. A quick Google search of "data management problems" yields nearly 1.6 billion results. Anecdotally, both on the consulting and client side, I've seen organizations' poor data management result in incorrectly paid employees, lawsuits, missed business opportunities and other catastrophes.

Those of us who have seen the perils of suboptimal data management can identify the usual suspects. Many organizations' data management efforts are manually intensive. Sure, organizations may have automated parts of the process by kicking off ETL jobs at 3 a.m., but it's not uncommon for data management to require a good deal of human intervention:

There are a few limitations of this model. First, it is an open loop, not a closed one. As a result, organizations will have a harder time improving their process.

Second, this high-level data management process has its roots in data warehousing principles stemming from the 1970s or 1980s. In case you hadn't noticed, the world has changed a bit since then. No longer is data management all about controlling the values of structured data in an internal, on-premise relational database. As I've written before, traditional data warehouses simply aren't enough anymore.

Next, relational databases and ERP and CRM applications remain important – but it's folly to think that they are the only games in town. These days, managing and understanding data from social media sites can be critical. Unstructured data is exploding as is the use of JavaScript Object Notation (JSON) – a lightweight data-interchange format.

Arguably most important, traditional data management is predicated on data at rest – not its streaming counterpart.

Brass tacks: We're drowning in data. To successfully manage disparate data sources and types, new tools and automation are no longer nice to have. They are becoming essential for organizations to properly manage their data. At a high level, they'll allow for tighter, closed loops and better feedback.

Simon Says: AI is already leading to better data management.

Models aside, data management is already benefiting from AI and complementary technologies. From a recent WSJ piece:

Unilog’s ever-expanding library and search engine would be unusable without the use of data preparation software, machine learning, and natural language processing that allow the contractors to quickly find the products they need.

Still, the model above won't eliminate all firms' data management problems. As Unilog shows, increased automation is poised to improve things. Ideally, AI will allow employees to identify data management issues sooner – perhaps even before they become full-fledged crises.