RAG Didn't Die—It Moved Up The Stack

Co-Founder of Superlinked, building enterprise-grade open-source inference for production-scale Agentic AI search and document processing.

getty

The narrative that Retrieval-Augmented Generation (RAG) is dead has been repeated by enough credible voices over the last year that many engineering leaders have started to believe it.

The argument is understandable: context windows got larger, long-context recall improved and some early RAG patterns deserved to be retired, but the conclusion is wrong. Retrieval did not disappear. It moved up the stack, from a visible pipeline step into the broader context layer that decides what evidence a model sees, in what order and at what cost. The teams that saw it early are the ones that I see pulling ahead.

What The Numbers Actually Show

Cursor's Developer Habits Report, based on agentic coding usage, makes the shift hard to argue with. The input-to-output token ratio rose from 4.52 times on January 1, 2026, to 11.41 times on May 9. Input now accounts for more than 90% of input-output token volume. Once cache is included, cache reads alone account for around 90% of total token activity; fresh input has grown from about 3% to 7%, and output has remained below 1% throughout.

This is not a RAG benchmark. It is a context-construction signal. Modern agentic systems are increasingly readers before they are writers. They consume, reuse and reshape large amounts of context before producing relatively small amounts of output. That makes the question behind retrieval even more important: What should the model be looking at right now?

Where The "RAG Is Dead" Claim Came From

The claim grew out of a real observation. Context windows expanded from a few thousand tokens to hundreds of thousands, and in some cases, a million or more. Long-context recall improved. Models began handling messy, unstructured input without falling over.

Some of the old RAG stack did deserve to die. Simple chunking, cosine-similarity-only retrieval and prompt stuffing weren't enough for production AI. That is not the death of retrieval. It is the death of treating retrieval as a thin pre-processing step.

Retrieval was never about workarounds for short context windows, but rather giving the model the right information at the right time. While the shape of that work changed, the work itself did not go away.

Where Retrieval Actually Went

Retrieval moved from a step in the pipeline to the layer that shapes the context window itself. The work now lives in places that are not always called retrieval.

Query reformulation rewrites the user's request into something the system can answer. Hybrid retrieval blends keyword and vector matches because neither alone is enough at production scale. Reranking sorts the candidate set by relevance before the model sees it. Agentic retrieval pulls additional context on the fly as a task unfolds. Context compaction decides what to keep when the window fills up.

Each of these belongs to the same discipline: deciding what evidence the model should see, in what form and in what order. The names changed, but the job did not.

What Actually Moves Retrieval Quality

In the production retrieval systems we have observed, the single most consistent improvement is not a better embedding model. It is reranking applied to a wide candidate set.

Most teams retrieve their top 10 results and feed them directly to the model. Teams that pull a top 100 or top 200 candidate set and pass it through a cross-encoder reranker first often see larger gains in answer quality than teams that swap to whatever embedding model is leading the public benchmarks that month. The reranker reads the query and each candidate document together. The retriever can only compare them at a distance. The second look is where the largest gains appeared. In agentic systems, where retrieval runs many times per task, these gains compound.

Why This Matters Operationally

Teams that read "RAG is dead" and pulled back on retrieval are paying for that decision in three places:

1. Cost

Input tokens cost less per unit than output tokens, and cached tokens cost even less, but input context now dominates token volume and a growing share of price-equivalent cost. Irrelevant context gets read, cached, reused and degrades answer quality before generation begins. Bad context is one of the most expensive mistakes in the stack.

2. Quality

Long-tail queries, where much of the business value lives, are exactly the queries where retrieval shape determines the answer. Stuffing the full context into the window does not solve this. It hides it.

3. Iteration Speed

Without a retrieval layer that the team owns and instruments, debugging a degraded answer means debugging the model. This is rarely productive.

What Leaders Should Do

I've found that there are a few moves that help separate teams positioned well for the shift from the rest. The first is instrument retrieval quality on its own, independent of generation quality. Recall, precision, grounding rate, latency and context length belong on a dashboard that the team watches weekly. If the only quality signal is the final answer, the team will misdiagnose failures and over-tune the model when the context layer is the real problem.

It's also important to treat context construction as a product surface, not glue code. The source mix, the reranking strategy, the order of retrieved evidence and the compaction policy all shape the user experience. They deserve product ownership.

Finally, audit where retrieval already exists in the stack under different names. Query rewriting, reranking, memory, tool selection, context pruning and agent planning often live in different parts of the system, often without a single owner. Naming the layer is the first step to improving it.

Where The Work And Opportunity Moved

In my experience, the teams that built an advantage over the last year did not double down on bigger context windows. They built the layer above them. Although RAG did not die as many have said, its simplistic version did. The teams that confused those two things already lost ground to the teams that knew where retrieval went. The work moved, and the opportunity moved with it. ​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?