Human-In-The-Loop AI: What It Takes To Find The Right Balance

Ryan Johnson is the chief product officer at CallRail, an AI-powered lead engagement platform that serves over 225,000 businesses worldwide.

getty

​Every new wave of technology promises speed, efficiency and cost savings. AI does, too. As an example, for many overloaded companies, it can help triage incoming requests, separating what’s urgent from what can wait.

​Still, history has shown that adopting new technology does not guarantee immediate returns. Those lessons are not lost with AI. Despite 88% of organizations citing regular use of AI in McKinsey’s research, less than 20% have seen AI make a tangible impact on enterprise-level EBIT.

This dichotomy isn’t new. Prior tech implementations, such as ERPs and CRMs, also had high failure rates. Just like with those implementations, the disconnect with AI often stems from a misunderstanding of how it changes the way a business runs.

In the case of AI, one of the most important concepts to understand is how to architect a system that augments rather than replaces human judgment.

Human-In-The-Loop AI: The Missing Piece

AI typically performs best when tasks are structured, clearly defined and predictable. Its strength lies in analyzing vast datasets within a confined set of rules and providing recommendations based on observed patterns. That’s why it’s effective in generating reports, routing calls or identifying duplicates in a spreadsheet.

However, AI falls short when it lacks the context and specificity to be truly intelligent and effective. Without domain or market knowledge that usually resides in a person's mind, the output becomes unreliable.

In a business setting, this lack of accuracy can cause negative downstream effects, such as when Deloitte had to refund much of its fee for producing a report that contained AI-generated errors or when lawyers for MyPillow CEO Mike Lindell were fined for filing briefs with errors generated by AI. ​​

The implication for businesses is the need to layer human oversight into AI workflows. While AI is getting better at handling more complex tasks, human intervention is still needed in the process to ensure the right outcomes.

Why Workflow Orchestration Sits At The Center Of AI Success

To successfully integrate humans in the loop, answer this question: Where in the process should you trigger the AI-to-human handoff? ​The answer becomes even more pressing as the deployment of AI agents ramps up, since they act autonomously, often without natural points for human intervention. ​

The goal is to blend the speed and efficiency of AI with the accuracy and quality of human input to improve outputs. A study by Stanford and Carnegie Mellon, for example, found that AI-human teams are 68.7% more effective and accurate than AI or humans alone. ​

A way to think about it is to view tasks on a spectrum, with AI and humans at opposite ends. The AI handles ​repeatable and operational tasks, such as answering inbound calls or routing messages, while the human focuses on edge cases that require judgment before execution. ​

Take, for example, a pool company using an AI voice agent to manage inbound call leads. The AI voice agent can be used in the first half of the workflow to triage and qualify calls based on the lead-scoring rules the pool company has established.

Depending on the complexity, the AI voice agent can prompt the caller to book an appointment or a quick service request, at which point, the workflow is complete.

But if the caller requests a more complex service, like new pool construction, the AI voice agent will need to be orchestrated to route the call to the best-fit sales agent based on the conversation. ​

The net positive for the pool company is that it can deliver better customer experiences by ensuring timely and consistent support. Likewise, the company benefits from ensuring that no inbound leads slip through the cracks, thereby supporting bottom-line results. ​

Architecting The AI-To-Human Handoff

As a chief product officer, I view the orchestration of AI-driven workflows with human-in-the-loop the same way I approach solving product challenges. It starts with understanding the problem, outlining next steps, finding where technology can help and then engineering the system based on these outputs.

The same logic applies to designing the AI-to-human handoff. Specific business tasks need to be broken down into steps. Using the spectrum example above, the tasks need to be organized into those better suited to AI and those that require human approval or feedback.

As the system takes shape, the next step is to map the required data streams to facilitate the workflow. Not all businesses have the context for AI to reference, and that’s okay. This insight can guide how to optimize the perfect handoff as more data becomes available with use.

While automating end-to-end tasks can be difficult, the best approach is to start simple. See what AI can automate as much as possible without compromising accuracy for efficiency. Once the workflow design is completed, deciding which AI solution to implement, if one has not been selected, becomes easier.

Achieving AI-Human Harmony

AI might be one of the most powerful tools we’ll see in our lifetimes. But the promise of immediate productivity benefits exists only when we carefully balance how AI and humans work together as a complete system.

At face value, this might seem like a slower approach. Yet AI can also be used to expedite discovery and workflow design by running interactive tests that enable you to make iterative changes in real time.

Ultimately, it’s the integration of AI and humans working together as one system, not the AI solution alone, that will deliver the best results for businesses and their customers. ​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?