-
Choosing RAG vs. stuffing documents into a long context window
Whether to retrieve-then-generate (index a corpus, pull only relevant chunks into context) or put a whole document/corpus directly into a long context window is a live architecture decision: both answer 'how does the model get the knowledge it needs,' with different cost, latency, and reliability tradeoffs.
Why this verdict?
Sources genuinely disagree. Cost/latency data favors RAG heavily (cited near $0.00008/query vs. $0.10-6.75 for 100k-1M-token long-context calls, responding in under 2s vs. 30-60s); long-context proponents point to it beating chunk-based RAG on QA quality when the source is stable and fits the window. 'RAG is dead' and 'in defense of RAG' pieces are both circulating in 2026 with real evidence behind each -- the right call depends on corpus size/stability and latency/cost budget, not a universal winner.
-
Contextual retrieval: context-prefixed chunks + hybrid search + reranking
Before embedding/indexing a chunk for retrieval, prepend a short LLM-generated note situating it within the full document (e.g. 'this is from ACME's Q2 2023 SEC filing...'), then retrieve using both embeddings and BM25 keyword search together, optionally adding a reranking pass.
Why this verdict?
Anthropic reports this combination cutting top-20-chunk retrieval failure from 5.7% to 2.9% (49% fewer failures), and to 1.9% (67% fewer) with reranking added -- each layer contributes independently. It remains the reference approach for chunk-based RAG rather than having been superseded by something else.