Why QA Fails Because Of Coordination, Not Testing Code

Karim Jouini is the CEO of Thunders AI, an AI-powered test automation platform.

getty

I recently spoke to an enterprise leader who described what a bad release looked like for their organization. Every team's monitoring showed clear. Every test in every suite had passed. The post-mortem ran for 90 minutes and reached a familiar conclusion: Nobody had done anything fundamentally wrong, yet the system had failed.

If you run engineering or quality at scale, you've sat in that room where you know the testing worked exactly as designed, but where the system didn't work as intended. ​

The problem is that this testing model was built for a different era of software delivery. The old quality model assumed one team could observe and own everything while keeping all of the necessary assumptions in their head. ​

Modern enterprise software is too fragmented for that, and we keep trying to manage it with tools, metrics and org charts built for that simpler past. QA stopped being a testing problem somewhere around the move to microservices, and it might be time to rebuild the model to match our current environment.​

What Scale Actually Changes

The standard story is that enterprise systems got bigger and faster, so we need more tests that run more often and more automation. But that story treats quality as a property of components, when in a system with twenty teams shipping into shared production, quality is a property of interactions.

I've watched engineering leaders run the math on this and reach the wrong conclusion. They look at coverage going up and incident rates not going down, and they assume the answer is more rigor, more gates and more sign-off. What they are actually measuring is the gap between a single feature working in isolation and the entire platform working as a whole. You can only close this gap by finding the seams.

In my experience, the unit of quality at enterprise scale is the user journey, not the service itself. Almost everything distinctive about the product follows from the user journey.

I learned this the hard way while building an expense management platform in a previous role. A user submitting an expense touched six teams in four organizations, from receipt capture and optical character recognition through to the policy engine and the accounting export.

Every team's suite was green most of the time, but something kept breaking. Rates and recoverability rules differ by country and category, and what one team treated as a valid VAT line was not always what the team downstream was expecting.

In other words, even if each piece behaved correctly on its own, the user still got the wrong reimbursement. Both things were true at once.

One of the biggest frustrations of the old testing model is the meeting where everyone presents green metrics, and the release still goes badly. Pass rates measure activity, not confidence. They tell you what was tested. They don't tell you whether you tested the right things, or whether the assumptions behind the tests still hold.

The goal with this model is fair: You need some metrics, and pass rates are at least cheap and consistent. But cheap and consistent is what they're good for. Treating a 98% pass rate as a release signal is how you end up in a post-mortem, staring at green across every dashboard with a broken user experience.

Where AI Helps, And Where It Doesn't

I run an AI company in the testing space, and I've found that most leaders assume that AI will make testing faster for them. It does, but the bigger leap is changing who owns the work. ​​

The bottleneck is ownership, not throughput. No human engineer volunteers to write and maintain the test that crosses four teams' code. So that test doesn't exist, and the failure it would have caught surfaces in production.

The thing AI is actually good at is managing the cross-team intent that no human is going to own. Agents can generate the journey-level scenarios that span service boundaries, regenerate them when the underlying components move and keep them readable in plain language so product, engineering and QA can argue about the same artifact.

Organizational memory is the point. The part of the system view that used to live in one architect's head, and stopped fitting there around the time the company hit two hundred engineers.

This is why journey-level testing is a better model in the AI era, because the journey is the only level at which the coordination problem is actually visible. When that memory exists outside any one team, the coordination problem becomes a coordination conversation. There's finally an artifact to point at.

With AI, product owners can write tests for their own user stories during the build phase using the same language as everyone else, so that everyone can see and reuse, including professional services teams.

With a user-journey model, QA can stop being a function at the end of the pipeline and evolve into a critical part of the process that produces and deploys software in the first place.​

What This Means Monday Morning

If QA is a coordination problem masquerading as a testing problem, then the next thing to look at isn't your test suite. Instead, look at the meeting where you decide whether to release. Who's in it? What artifact do they look at? Can anyone in that room name the three user journeys that, if they broke, would actually cost the business something?

If nobody can answer that question, no amount of testing is going to save the release. You need a platform, memory and collaboration.

From what I've seen, most enterprise outages today are failures of shared understanding rather than code quality. By addressing this, teams can both ship faster and have processes that can be clearly understood at your next post-mortem.​​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?