RAG, long context, or both? A decision guide for enterprise knowledge

There is a meeting happening in a lot of companies right now where someone says the context window got huge, so the retrieval system is legacy. The window really did get huge: Google's current documentation (updated June 2026) describes Gemini windows of a million tokens or more, and every frontier vendor has moved the same direction. The mistake is what happens next in the meeting: an architecture gets chosen because of a model feature.

A model feature is an input to exactly one of the variables that should decide this. I scope enterprise knowledge systems for a living, and the durable question is never what the window can hold. It is what your knowledge is like and what your answers owe people.

The two patterns, plainly

Long context means assembling your knowledge into the prompt itself and letting the model read everything on every query, with context caching (the provider stores the stable part of the prompt so repeat queries do not re-pay for it) to keep the bill sane. Retrieval, the RAG pattern, means indexing knowledge outside the model and fetching only the relevant pieces per query. The original RAG paper (NeurIPS 2020) is worth remembering for its motivation, because window size was never it: parametric models, meaning models that answer from whatever training baked in, leave “providing provenance for their decisions and updating their world knowledge” as open problems. Retrieval was built to answer where a claim came from and to update knowledge by updating an index instead of a model. Those two jobs did not get smaller since 2020. The windows did get bigger, honestly and dramatically, and that changes real decisions below.

What long context genuinely wins

Concede the strong case first. For a corpus that is small, stable, and readable by everyone who can query the system, stuffing it into context beats building a retrieval stack. No chunking strategy, no index freshness, no retrieval failures, and the model sees everything in full. Anthropic's contextual retrieval guidance (September 2024) says this outright: under roughly 200,000 tokens, about 500 pages, skip RAG and put the corpus in the prompt with caching. The economics cooperate for the right workload: Google's long-context documentation describes context caching cutting repeated-query input costs materially (their doc cites about 4x on one model), with cached storage billed hourly. One tradeoff rides along in the same documentation: time-to-first-token grows with input length, which suits patient chat and punishes tight product loops. Stable corpus, repeated queries, uniform access: long context, and a retrieval stack there is over-engineering. The numbers here are snapshots, the threshold from September 2024 and the caching economics from mid-2026, and they will move; date-check them before deciding.

What a big window does not buy

Four things, and they map straight onto the six variables.

Uniform capacity. In July 2025 the research team at Chroma, a vector-database vendor with a declared stake in retrieval, ran 18 models through tasks whose complexity was held constant while input length grew. Performance degraded non-uniformly: distractor damage amplified with length, answers phrased differently from the question got harder to find, and even trivial tasks turned unreliable at scale. Hold the framing with the vendor stake in mind; the direction still stands. A window's rated size is not a promise about its working quality, and filling it has a cost before you pay a bill.

Freshness. A prompt is assembled; it does not watch your knowledge. When policy changed this morning, a retrieval index updates one document; a cached mega-prompt gets rebuilt and re-cached, and until then the model reads yesterday.

Permissions.Context is all-or-nothing per call. Whatever is in the window is available to the answer, so if access differs by user, the assembly step must re-derive who may see what on every query. Retrieval handles this where it belongs: managed stacks like OpenAI's expose attribute filtering on the index, meaning per-file tags the query filters on, so the permission boundary lives in the query instead of a hand-built prompt assembler.

Citations. A window is not a citation mechanism. If consequential answers must show their evidence or decline, that is the cited-or-declined contract, and it needs retrieval traces: which sources, which passages, bound to which claims. Reading everything and asserting an answer is exactly the posture the contract exists to prevent.

The decision matrix

Six variables, three honest answers.

Read your system down the right-hand column. Zero or one match: long context, enjoy the simplicity. Two or more: retrieval is earning its complexity. A fifty-page HR policy handbook that changes yearly and is readable by every employee sits cleanly in the left column. A product knowledge base that updates daily, splits by customer tier, and feeds quoted answers lives in the right. And when the columns split, which is the common case, the answer is the hybrid.

The hybrid is the normal end state

Most enterprise systems I scope land on a routing design, and they land there deliberately. A small, stable core rides cached in context: definitions, standing policies, product fundamentals, the material every answer needs. Retrieval handles the rest: the large, the living, the permissioned, and everything that must be cited. A router decides per query whether the cached core suffices or the index runs. The retrieval side deserves real engineering while you are at it. Anthropic's contextual retrieval work cut retrieval failure rates roughly in half on their benchmarks by enriching chunks before indexing, and the retrieval column of the matrix assumes that kind of care, never a weekend vector store.

The router's thresholds are the part you revisit. Window sizes, caching prices, and vendor guidance move faster than any article can track, which is why the matrix carries variables and the numbers carry dates. And the hybrid's cost is real: the router is itself a system to own, with routing logic, cache invalidation, and two code paths to test. That operational weight is exactly why the matrix comes first; do not buy the hybrid until two or more variables have earned it.

How to decide this quarter

Build a small evaluation corpus before committing: fifty real questions from your actual users, including questions your documents cannot answer. Failure behavior matters as much as the success rate, and a system that answers unsupported questions smoothly has already failed the audit in the grounding contract. Run the set against the candidate architectures with your own documents. Score answers, evidence, latency, and cost per query, and write down the thresholds that made the decision. Then put a revisit date on the decision, one quarter out, because the crossover point will move again. The variables will not.

Questions worth asking next

Sources

1. Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020, arXiv 2005.11401. https://arxiv.org/abs/2005.11401
2. Anthropic, "Introducing Contextual Retrieval," September 19, 2024. https://www.anthropic.com/news/contextual-retrieval
3. Google, "Long context" (Gemini API documentation), updated June 22, 2026. https://ai.google.dev/gemini-api/docs/long-context
4. OpenAI, "Retrieval" guide. https://developers.openai.com/api/docs/guides/retrieval
5. Kelly Hong, Anton Troynikov, and Jeff Huber (Chroma), "Context Rot: How Increasing Input Tokens Impacts LLM Performance," July 14, 2025. https://www.trychroma.com/research/context-rot