New models accept enormous context windows. Should we drop RAG and just put all our documents in the prompt?

TL;DR: Sometimes, for small, stable, shared document sets, especially with prompt caching. For a large, changing, permissioned corpus, keep retrieval as the baseline. Sending hundreds of thousands of tokens per query costs far more than retrieval, adds latency, can lower answer quality on details buried in the middle, cannot enforce per-user permissions on a shared prompt, and still fails once the corpus outgrows the window. Use long context selectively: for a single long document the user is working on, or after retrieval.

How to approach it

Ask how big the corpus is, how often it changes, whether different users may see different documents, and the query volume. Then compare the two approaches on cost, latency, quality, permissions and scale, with numbers.

A strong answer

The appeal. No chunking, no embeddings, no retrieval failures: the model sees everything. For a small set of documents it can work very well.

The arithmetic. Suppose a corpus of about 500,000 tokens (a few hundred pages of policies and product docs), 200,000 queries a day, Claude Sonnet 5.5 list prices as of 6 November 2026 ($2 per million input tokens, $0.10 for cache reads, $10 output), and 500-token answers. Assume 30 days, all cached-prefix reads hit, and omit cache writes, storage, retrieval and other operating charges from this model-call-only comparison:

Caching changes the picture dramatically, from absurd to merely three times RAG's cost. That point is easy to miss, and it means long context is a real option for small, shared, stable corpora. It is still more expensive, and the cache must be rewritten whenever the corpus changes.

Run the rate limits too, because they kill the uncached design before the bill does. 200,000 queries a day is about 139 a minute on average. Without caching, that is roughly 69 million input tokens a minute; the published Scale-tier limit for Claude Sonnet 5.5 is 10 million (rate limits, checked 6 November 2026). Cache reads do not count toward that limit, so the cached design uses about 280,000 a minute for the fresh tokens. Pricing can also change with length: Claude Haiku 5.5 charges $0.10 per million input tokens up to 100,000 tokens of prompt and $0.50 above it, so a cheaper model can stop being cheap at exactly the prompt sizes this design needs.

Beyond cost.

• Latency. Processing hundreds of thousands of tokens takes time, even with caching helping considerably. Retrieval plus a short prompt usually answers faster.
• Quality. Models can miss or misweight details in very long inputs. Liu et al.'s Lost in the Middle (TACL 2024) found performance was highest when the relevant information sat at the start or end of the input and degraded when it was in the middle. RULER (COLM 2024) tested 17 long-context models: nearly all passed the simple needle-in-a-haystack test, yet only about half of those claiming 32,000 tokens or more held satisfactory performance at 32,000 on harder tasks. The advertised window is a capacity, and the effective window is something you measure. Retrieval puts the five most relevant passages in front of the model; a huge prompt asks it to find them. Test on your own questions; results vary by model.
• Permissions. If different users may see different documents, one shared prompt containing everything cannot enforce that. Build prompts from each user's authorized documents. Permission-scoped shared prefixes may still be cached, but access changes must invalidate affected entries. Retrieval filters by permission before anything reaches the model.
• Freshness. An edit invalidates the affected prefix or derived summary; include source versions and access changes in the cache contract.
• Scale. Corpora grow. A wiki of 50 million tokens will not fit in any window.
• Citations. Retrieval knows which passage it supplied; with a huge prompt, attributing an answer to a source is harder.

Where long context does win.

• A user working with one long document: analysing a 300-page contract, a codebase, a long transcript. Compare full context with retrieval on the actual task; neither wins by definition.
• Small, shared, stable reference sets (a few hundred pages that rarely change), with caching. Anthropic's contextual retrieval guidance draws the line at about 200,000 tokens, roughly 500 pages.
• After retrieval: retrieve whole sections or documents instead of tiny chunks, using the larger window to give the model more surrounding context.

My recommendation for a typical enterprise knowledge product: keep RAG, use the larger window to retrieve bigger, more complete passages, and use full long context for tasks centred on one document.

Price source: Anthropic rate card, checked 6 November 2026.

What interviewers probe next

"Retrieval keeps missing answers; long context would fix that." It might, but measure recall first. Inspect whether the source was parsed, authorized and indexed before trying hybrid search or reranking. Compare improvements on the missed cases and a held-out set.

"Prices keep falling. Won't long context be cheap soon?" Possibly; the permission and freshness constraints remain. Re-run the cost comparison when prices change.

How it's scored

Common mistakes

Ignoring prompt caching, or assuming it makes long context free.

Forgetting per-user permissions, which a shared prompt cannot enforce.

Assuming models use every part of a very long context equally well.

See RAG explained and prompt and prefix caching.