No. What died is its 2023 version: fixed-size chunking, unfiltered vector search, and the bet on an infinite context window. Targeted retrieval remains the cheapest and most reliable option once a corpus exceeds a few tens of thousands of tokens or query volume is high.

Do I have to choose between RAG and a long context window?

No, architectures that hold up in production combine both: RAG first narrows the number of candidate documents, then the long context window lets you send several full documents rather than isolated fragments once that number is small.

What knowledge base size justifies RAG over a full prompt?

The threshold we observe sits around 25,000 tokens for a stable, rarely-queried corpus. Beyond that, or as soon as content changes often, the cost math shifts strongly toward targeted retrieval instead of resending the whole corpus every time.

What does agentic RAG actually change?

It adds a decision step before the search itself: whether to search, what to search for, where and how. That step costs more to build but reduces the risk of imprecise answers when questions are varied and query volume is high.

Does an MVP need RAG from day one?

Rarely. A retrieval pipeline added before the real query volume is known slows down iteration without improving need validation. It is better to start with full context when that is enough, then add a retrieval layer once production data is available.

RAG in 2026: What Still Holds and What Got Replaced

RAG in 2026: the direct answer

RAG is not dead in 2026. What died is its 2023 version: fixed-size chunking, a single vector search with no filters, and the bet that a large enough context window would eventually make retrieval pointless. That bet failed: sending an entire corpus into the prompt on every request costs, across the architectures we see in client work, 8 to 80 times more than targeted retrieval, often with lower precision on large corpora. The right question is no longer "RAG or long context?" but "what data volume, what budget per request, and what latency can my use case tolerate?" The rest of this article covers what was genuinely abandoned, what still holds, and the method we apply on client projects to decide, including the three situations where building a retrieval pipeline is simply the wrong call.

What got abandoned since 2023

Three practices that defined first-generation RAG lost their technical justification.

Fixed-size chunking

Splitting a document every 500 tokens, regardless of headings, paragraphs or meaning, produced fragments cut off mid-idea. The model would receive a sentence without its conclusion, or a conclusion without its premise. Semantic chunking, which respects document structure and adapts block size to content, has replaced this approach in nearly every recent project we come across.

Vector search alone, without filters

Cosine similarity between two embeddings says nothing about a document's freshness, its status (draft or approved), or its source. A system that only "searches for what looks similar" sometimes surfaces an outdated version of an internal policy simply because it is linguistically close to the question asked, or worse, two contradictory versions of the same document with no indication of which one is authoritative. Hybrid search, combining lexical and vector similarity with metadata filters (date, status, scope), made this weakness manageable without ever removing it entirely. An exact keyword often stays more reliable than semantic proximity for a contract reference, a standard number, or a product name: that is precisely what pure vector search misses most often.

Betting on an infinite context window

The assumption that million-token context windows would make RAG unnecessary did not hold up in production. Two documented problems explain why. First, the "lost in the middle" effect: a model retrieves information placed in the center of a very long context less reliably than information placed at the start or the end. Second, time to first token increases sharply with the size of the context sent, which makes the approach hard to sustain in an interactive application. Vendors kept raising the advertised context limit year after year, but the practical ceiling for reliable recall has moved far more slowly than the marketing number suggests.

What still holds, and why: the cost math

On a thousand-page knowledge base (roughly 600,000 tokens) queried a hundred times a day, the math clearly favors targeted retrieval over resending the entire corpus every time. The table below summarizes the order of magnitude reported across several 2026 architecture analyses.

Approach	Tokens sent per request	Relative cost	Main weakness
Full context (entire corpus)	~200,000	~30x baseline	Cost and time-to-first-token
Full context with caching	~200,000 (cached)	~3 to 4x baseline	Corpus must stay stable between requests
Targeted retrieval (RAG, top 8 chunks)	~6,000	Baseline (1x)	Depends on chunking and filtering quality

That math flips once the corpus is small and stable: below roughly 25,000 tokens, with low query volume, loading everything into context is often simpler to maintain than a retrieval pipeline, for an overhead that doesn't yet strain the budget. Caching narrows the gap but doesn't close it, and it comes with its own constraint: the cached content has to stay identical between requests, which rarely holds for a knowledge base that gets edited during business hours.

Long context and RAG: complementary, not rivals

The most useful reading for an enterprise project in 2026 does not pit the two approaches against each other, it combines them. RAG first narrows the field of candidates among thousands or millions of documents; the long context window then lets the model receive several full documents, rather than isolated fragments, once that field has been narrowed to a handful of relevant items. It is this combination, not one replacing the other, that shows up in architectures that hold up under production load.

Agentic RAG: the real 2026 shift

The most concrete change is not in the search algorithm, it's in when the decision to search gets made. Recent architectures add a reasoning step before retrieval itself, typically organized into four successive decisions.

Should we search at all? A general-knowledge question doesn't warrant a call to the document store; a question about an internal procedure does.
What to search for? The user's raw question gets rewritten into a structured query, with entity extraction and relevant filters (date, product, scope).
Where and how to search? Choosing the document collection and the strategy (lexical, vector, or both combined and reranked).
Generate the answer. The model receives a reduced, already-sorted context, which limits the "lost in the middle" risk described above.

This architecture costs more to build than a classic RAG pipeline, because it adds a decision before every search. It earns its keep once query volume and the diversity of questions asked justify it; below a certain volume, it adds complexity without a measurable benefit.

A complementary piece, often skipped in fast rollouts, is reranking: a second, smaller and cheaper model reorders candidate chunks before handing them to the main generation model. Without this step, the system trusts the first pass of vector search, which favors overall semantic proximity and sometimes misses the single most relevant chunk for the precise question asked. Adding a reranker costs a few extra tens of milliseconds; it's a measurable investment once answer precision drives how much users trust the tool.

Method: choosing for a Swiss SME project

On the projects we run for Swiss SMEs and startups, the decision comes down to three questions, in this order.

Situation	Recommended architecture
Stable corpus under 25,000 tokens, low query volume	Full context in the prompt, no retrieval pipeline
Large or frequently changing corpus, high query volume	Classic RAG with hybrid search and metadata filters
Heterogeneous questions, multiple sources, high precision requirement	Agentic RAG with a decision step before retrieval

One point we measure systematically on engagements: source freshness matters more than pipeline sophistication. A RAG system connected to documentation that hasn't been resynced in six months answers with the same confidence on outdated information as on current information. No retrieval architecture fixes an upstream documentation governance problem, and no amount of reranking recovers information that was never updated at the source.

Before choosing an architecture, measure three numbers for your own case: how many tokens your current corpus represents, how many requests per day you expect once the tool is live, and how often the source content changes. Measuring these three numbers before writing a single line of pipeline code avoids most of the architecture regrets we see six months in: either a RAG built for a corpus that would have fit in a plain prompt, or the opposite, a full-context prompt that becomes unsustainable as documentation grows.

When RAG is not the right choice

Three situations where building a retrieval pipeline adds cost without adding value:

A small, stable corpus (a few dozen pages that rarely change): full context, optionally cached, is simpler to maintain.
A prototype or MVP meant to validate a use case before investing in infrastructure: a retrieval pipeline added too early slows down iteration without improving need validation.
A need to reason over an entire corpus rather than over excerpts (summarizing a full report, for instance): RAG, which only returns fragments, is structurally a poor fit for this kind of question.

What we see on engagements

On the AI integration projects we deliver, the RAG question almost always comes up too early in the conversation with the client, before the real query volume or corpus size is known. Our approach is to start with the simplest architecture that covers the need, full context when that's enough, then add a retrieval layer only once production measurement shows it's necessary. This staged approach limits the risk of building a sophisticated search infrastructure for a use case that never needed one, and it keeps the first deployable version in front of real users weeks earlier than a project that starts with the full pipeline. You can browse our work to see how this method plays out on real cases, or read our article on what breaks first when an AI agent goes into production, which details the failure points of a system already in place.

Going further

This topic belongs to our Forward Deployed Engineer cluster, where we document what AI integration actually changes about engineering work in a business. Browse all blog articles to dig into related topics.