There is yet another AI productivity gap

When I first started as a data scientist, there was a gap. I met with dozens of organizations who would invest time and resources into building accurate and tuned models and then ask, “What now?” They had a fantastic model in hand but couldn’t get it into a place and form where it could be used to make a better decision or improve a specific outcome. At the time, I called this chasm “the gap between finding insights and using insights.”

Earlier this year, research from MIT found that 95% of GenAI investments have a 0% return on investment. When you take into consideration that the enterprise investments into GenAI projects have been to the tune of $30 - $40 billion, you realize just how much money gets thrown into GenAI haphazardly. MIT has called this new gap “The GenAI Divide.” I call it “Yet Another AI Productivity Gap” or YAAPG for short.

But within the YAAPG, there is a paradox. Personal GenAI tools are improving personal productivity. Anthropic recently reported that Claude says it speeds up individual tasks by about 80%. Yes, Claude, the LLM. The LLM that also said that it could deliver products in person at one point, so I won’t fault you if you take Claude’s statements with a pinch of salt. Yet, when used thoughtfully, many individuals report productivity gains from GenAI. Power users understand how to use GenAI in their personal work to create value and avoid AI Slop. In fact, All Things Open released a whole guide that features tips and use cases for improving personal productivity using GenAI.

As we look across the GenAI Divide, what prevents organizations from seeing a return on their investments? Is it poor data processing capabilities? Is it GenAI systems that can’t learn? Is it a lack of trust in these systems? Is it a lack of AI Governance? Or is this just the same problems we’ve always seen when organizations are operationalizing AI? And what can we learn from personal productivity gains?

Lessons from Russian Novelists

In the opening line of Anna Karenina, Leo Tolstoy wrote, “All happy families are alike; each unhappy family is unhappy in its own way.” This led to the Anna Karenina principle in that success relies on several key factors coming together correctly, and failure can occur in any number of ways. Operationalizing AI requires a lot of things being done correctly.

All happy families are alike; each unhappy family is unhappy in its own way.

And when we look at power users of GenAI for personal productivity, they’re often doing several things right. First, they know what problem they’re trying to solve. Second, they understand the benefits of solving that problem. They find the right tool for the problem, make adjustments to the tool, and understand how to feed their inputs to that tool. They observe the outputs, judge their quality, and make adjustments. They’ll try new things, learn how best to work with the tool, and trust their usage of the tool.

Let’s start with a personal productivity example to demonstrate those points. Many people use generative AI tools as coding assistants. These assistants may write comments to document code, answer questions, debug problems, suggest changes, write unit tests, and more. Each time a software developer uses their coding assistant, they use it to address a specific problem — and they understand the time-saving benefits of the tool. Software developers may be limited to using the tools their organization purchases, but these tools are often vetted and provided context through the organization’s code base. The input to the tool is often just the developer’s questions and relevant code. Importantly, the programmer is in the loop. The software developer sees what the coding assistant suggests, determines if it is satisfactory, and either approves it or makes adjustments. And, over time, the developer learns the ins and outs of using the coding assistant. The programmers begin to understand how they can ask better questions or create better prompts to get more satisfactory answers from the coding assistant. Through this experimentation, the software developer learns how to better use the GenAI tool, and not just trusts the tool, but trusts their use of the tool.

Scaling to Enterprise Problems

How does this scale to enterprise projects? First, before any project starts, you should know what you’re trying to accomplish. Many organizations see GenAI as a hammer where everything is a nail. I’ve spoken with individuals across organizations that were tasked with creating a project where they could use an LLM, GenAI tool, or Agent. An indiscriminate use of AI does not create ROI. A better approach would be to find a pain point or something the organization can do better. If you don’t know what your pain points or gaps are, ask. Employees, customers, or users are a great place to start.

By having a clear problem to solve or an outcome you’re trying to achieve, you can next state the benefits of solving that problem. If you can prevent customers from abandoning their carts in your online store, you can increase your revenue. If you can help customers answer simple questions or perform basic tasks using an online chatbot, you can give your employees time to work on more productive tasks, improving output. Now that you know the benefit of solving the problem, you can create a metric to measure project success. At this stage you can estimate if this problem is worth solving by weighing the expected benefit against the estimated costs.

If you’re ready to move forward, now you need to find the right tool for the job. Sometimes this is a shiny new AI tool, but sometimes it’s just an ETL pipeline feeding to a dashboard. Sometimes you need to look at the simplest solution that will get the job done. AI is more expensive and less predictable than business rules and code. If this problem truly is best solved using AI, next determine which AI tool is best for the job. There are many choices among AI tools with options like building in-house, getting access to a generalized model, acquiring a model fine-tuned for a specific task, or even getting a tool that wraps a model and other enhancements. Here you have to weigh the purchase cost, hosting costs, and employee time to implement the tools, with the expected benefit of the model towards the task.

Before you can productionalize your new tool, you need to understand and set up the downstream processes that feed data to the tool and upstream processes to monitor and observe the tools outputs. For some use cases, the user will provide their prompts or questions directly to the tool. In others, the tool may be operating as a part of an automatic pipeline. In that case, your organization may provide the tool access to a variety of documents, data files, or a fine-tuned specific prompt. Additionally, if any information is collected from a user, you may need to clean that input to remove instances of Personally Identifiable Information (PII), Intellectual Property (IP), toxic input, or attempts of prompt injection.

Beyond downstream data, the tool’s outputs should also be monitored, with guardrails in place to evaluate, approve, reject, or adjust outputs from the GenAI tool. Again, you want to ensure that the tool doesn’t return PII, IP, or toxic responses, but you may also want to review if the response is formatted correctly, is on task, and references accurate information. If a response has a flaw, perhaps there is logic to provide why the response was incorrect and let the GenAI tool try again. Downstream and upstream processes can be performed manually, as a part of an automated pipeline, or as a hybrid approach where manual intervention can take place when specific conditions are met. But before investing into a GenAI project, it’s important to have some idea about the level of effort to integrate the tool into a wider production pipeline.

Finally, you should encourage teams to experiment with their tools in a responsible manner. By having a safe space to experiment, teams can learn, see what works, see what doesn’t work, what can be adjusted in their user input, report problems, and trust their usage of the tool to improve outcomes for the organization.

Conclusion

In conclusion, before enterprises can see ROI on their GenAI investments, they must master the following:

Know what problems they’re trying to solve

Understand the benefits of solving that problem

Find the right tool for solving the problem

Know their data pipeline

Monitor, observe, and adjust outputs from the tool

Experiment, learn, and trust their usage of the tool

Traversing the GenAI divide will take a deliberate and thoughtful approach to solve a real problem rather than an indiscriminate use of AI.