Your AI Pilot Shipped. The Value Didn't. Here's Why

Sandeep Shivam is an Associate Director at Tavant, building AI-powered lending products that improve efficiency and customer experience.

getty

This is a common pattern in regulated AI programs: A pilot launches, demos well and gets celebrated. Six months later, no one knows its value, who owns it, or whether it still works as intended. By then, the team has already moved on to the next pilot.

If you build or lead product and engineering teams in fintech, you've seen this. In August 2025, the Commonwealth Bank of Australia (CBA), the country's largest lender, publicly reversed a decision (registration required) to cut 45 customer service jobs that it had said an AI voice bot made redundant. The bank had claimed the "bot reduced call volumes by 2,000 per week." However, union members showed call volumes had risen, with overtime issued and team leaders pulled onto the phones. CBA apologized to the workers, brought them back and acknowledged that its assessment of which roles were needed should have been more rigorous. The post-mortem framing was about AI. It almost never should be.​​

The Data Stopped Being Ambiguous

MIT's 2025 Project NANDA study of 153 enterprise leaders and 300 AI deployments found that 95% of generative AI investments produced "no measurable P&L impact" against $30 to $40 billion in enterprise spend. The widely cited number isn't a deployment failure rate. MIT defined failure specifically as a value-capture gap: pilots that shipped but produced no enterprise-level financial return. BCG's framing of why has held up across every credible study: AI success is 10% algorithms, 20% data and technology and 70% people, processes and culture. The model is the easy part.​

Five Operating Model Failures Product And Engineering Teams Own

The framing that AI failure is a C-suite problem lets product and engineering off the hook too cheaply. The C-suite owns budget and named accountability. Product and engineering own the five failures between funding approval and value capture.

1. Kill Criteria

Every pilot should ship with a documented usage floor, accuracy floor, cost ceiling and customer experience guardrail at which the program is killed. Most don't. The result is a portfolio of half-dead initiatives that consume engineering attention indefinitely because nobody has authority to stop them. If you can't say in advance what would make you turn this off, you've built an initiative that will be hard to govern and harder to shut down.

2. Problem Framing

RAND's 2024 root-cause analysis of 65 practitioners put misunderstood problem definition at the top of the failure list. In fintech, this shows up as generative AI being used to summarize meetings or draft customer emails. These uses never aggregate into P&L because no underlying business process is redesigned.

The team that only builds AI to make an existing workflow slightly faster has usually chosen the weakest version of the opportunity. The team that builds AI to enable a workflow that wasn't previously possible has chosen the harder, smaller, more measurable problem.

3. Data Foundation

Fintech firms aren't data poor. They're AI-ready data poor. The gap is lineage, governance, access controls, master data stewardship and a consistent customer identifier across customer relationship management (CRM), core banking, ERP and policy admin. A RAND interviewee called this the absence of "the plumbers of data science." If your data engineering team is smaller than your model team, you've staffed for the demo, not for production.

4. Evidence-Based Escalation

In May 2025, the fintech Klarna publicly walked back its claim that AI could replace 700 customer service agents. This wasn't a technical failure. The firm escalated agent autonomy faster than its measurement infrastructure could catch the quality drop. Mastercard did the opposite. It scaled its generative AI fraud system only after demonstrating doubled compromised-card detection and up to a 200% reduction in false positives. The pattern matters: assistant, then copilot, then agent, each transition gated by quantitative evidence.

5. Workflow Redesign Before Model Selection

Morgan Stanley's Debrief tool is often cited as an AI win, but it wasn't an AI notetaker bolted onto Zoom. It was a redesign of the assistant that hit 98% adoption among wealth management advisors. The model was the last decision, not the first.​

Before You Approve The Next AI Pilot, Ask These Four Questions

Most AI failure happens after the model is chosen and before the workflow is redesigned around it. Before approving the next pilot, ask the following:

1. What would make us shut this down? Without clear usage, accuracy and cost guardrails, it's less a pilot and more a budget item.

2. What workflow are we redesigning, not decorating? If AI only adds to the existing workflow, you may be missing the bigger opportunity to rethink it.

3. Do we have the data plumbers, not just the model builders? If the team structure is misaligned, the program will likely feel misaligned too.

4. What evidence promotes the next step of autonomy? Autonomy should advance only when measurable evidence proves it's safe, reliable and valuable.

AI leadership in financial services will be driven by people, processes and culture, not by the model an institution chooses. Model selection matters, but it isn't the main driver of AI success.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?