Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Why naive RAG fails
Chunking — the decision that quietly caps quality
Embeddings and hybrid search (BM25 + vector)
Metadata filtering — narrow the search space first
Reranking — precision at the top
Contextual retrieval — fixing the isolated-chunk problem
Query rewriting — meeting the retriever halfway
Evaluation — retrieval metrics vs end-to-end, on your own data
A pragmatic build order
Closing
References

Introduction — Why naive RAG fails

The core idea of RAG (retrieval-augmented generation) is simple: find documents relevant to a question, paste them into the prompt, and let the model answer grounded in them. The 2020 paper by Lewis et al. formalized this as coupling parametric memory (model weights) with non-parametric memory (an external index). A demo takes half a day: split the documents, embed them, drop them into a vector database, and attach the top k nearest to the question.

The trouble is that this naive pipeline quietly falls apart in production — and the failure is almost always in retrieval, not generation.

Vector search captures meaning well but misses exact token matches — error codes, product SKUs, people's names, API names.
A fragment cut out of a document loses its own context. A chunk that says "revenue grew 3%" no longer knows which company or which quarter.
The retriever quietly returns a plausible-but-wrong chunk when it is wrong, and the model cites it confidently. It looks like a hallucination, but it is a retrieval failure.

The key intuition is one line: retrieval quality caps everything downstream. If the relevant document never enters the context, no amount of prompt engineering brings it back. So most of the techniques below aim not at "making generation smarter" but at "getting the right chunk into the context."

There is also a temptation to sidestep all this by enlarging the context window and "stuffing everything in." It is not a fix — models tend to miss information buried in the middle of a long context (the so-called lost-in-the-middle effect), token cost and latency grow, and above all the underlying problem remains: the right document still has to be in there in the first place.

Chunking — the decision that quietly caps quality

What it is. Splitting a document into the units you retrieve and embed. The common mistake is fixed-length slicing (say, by character count), which cuts mid-sentence, through the middle of a table, or across a code block.

When it helps. Aligning boundaries with meaning raises retrieval accuracy. Two basics carry most of the weight: (1) respect structure — split on paragraphs, headings, tables, code; (2) use a sliding window / overlap so adjacent chunks share a little text and a sentence straddling a boundary is not lost entirely. "Searching for Best Practices in RAG" (2024) reports that smaller chunks in the 256–512 token range tended to do better on faithfulness.

The tradeoff. Smaller chunks are more precise but thinner on context; larger ones carry richer context but mix in noise that dilutes the retrieval signal. The right size is corpus-specific — which is exactly why chunk size is a value to be measured, not guessed.

Embeddings and hybrid search (BM25 + vector)

What it is. Embedding search (dense) turns text into vectors and matches on semantic similarity. BM25 (sparse) is TF-IDF-family lexical matching that finds exact word overlap. They fail in different ways: vectors are strong on synonyms and paraphrase but miss rare tokens; BM25 is strong on exact tokens but breaks when the wording differs.

When it helps. Almost always. Most production corpora carry both kinds of query — "summarize the refund policy" (semantic) and "what is ERR_2043" (lexical). That makes hybrid search (run both and merge the results) a powerful default. Merging is often done with Reciprocal Rank Fusion (RRF), which combines the two rankings by reciprocal rank so neither score scale dominates.

The tradeoff. You now run two indexes (a vector store and an inverted index) and gain one more tuning knob in the fusion weights. Even so, it is usually the highest-ROI first move, because it recovers the exact-match queries that pure dense retrieval structurally misses.

Metadata filtering — narrow the search space first

What it is. Filtering candidates by document attributes before (or alongside) scoring them with vector or BM25 — tenant ID, creation date, document type, and above all access permissions (ACLs).

When it helps. In multi-tenant SaaS or a permissioned internal knowledge base, it is not optional but mandatory. Leaking a document the user cannot see into the results is not a quality issue — it is a security incident. As a side effect, narrowing the search space also improves precision and latency.

The tradeoff. You must attach clean metadata at index time, and filtering too tightly can cut out the very document you needed. Remember that a filter is a hard constraint, not a ranking signal.

Reranking — precision at the top

What it is. The first-stage retriever pulls a generous candidate set (say, the top 20–100), then a cross-encoder reranker reads each (query, document) pair together and rescores relevance. Embeddings vectorize query and document separately (a bi-encoder); a reranker looks at both at once, so it is far more precise.

When it helps. When first-stage recall is fine but top-of-list precision disappoints. The rule of thumb is "retrieve wide, keep narrow" — retrieve 20, rerank to 5, hand 3–5 to the model. The best-practices paper above emphasizes reranking's necessity, noting that removing the module caused a noticeable drop in performance.

The tradeoff. A cross-encoder runs one forward pass per candidate, so it adds latency and cost. That is why you apply it only to a short candidate list. Reranking an ever-growing pool of 100 or 200 candidates rarely pays for itself.

Contextual retrieval — fixing the isolated-chunk problem

What it is. Chunking creates one fundamental defect: a fragment loses the surrounding context of its source. Anthropic's contextual retrieval prepends, before embedding and BM25 indexing, a short LLM-generated explanation (typically 50–100 tokens) that situates each chunk within the whole document — something like "This chunk is from ACME's Q3 2023 earnings report."

When it helps. Especially on long corpora full of pronouns and relative references. The numbers Anthropic reported are striking: measured as failure rate at recall@20 (1 − recall@20), contextual embeddings alone cut failures by 35% (5.7% → 3.7%), adding contextual BM25 cut them by 49% (→ 2.9%), and adding reranking on top reached a 67% reduction (→ 1.9%).

The tradeoff. You pay an index-time cost of one LLM call per chunk. Anthropic put it at roughly $1.02 per million document tokens with prompt caching, but you must re-run it whenever documents change, and pipeline complexity goes up. It earns its keep most on static, reference-tangled knowledge bases.

Query rewriting — meeting the retriever halfway

What it is. A step that reshapes the user's original question before retrieval. It takes several forms: decontextualizing a follow-up in a conversation into a standalone query, expanding acronyms and jargon, decomposing a compound question into sub-queries (multi-query), or HyDE, where you first generate a hypothetical answer and embed that to search.

When it helps. Multi-turn conversations, short or vague queries, and cases where the user's vocabulary and the document's vocabulary diverge. When the question is already well-formed, it helps less.

The tradeoff. It adds another LLM call — more latency and cost — and introduces a new failure mode: a bad rewrite retrieves worse than the original query would have. So rewriting, too, is something to compare with and without via evaluation, not a switch you leave permanently on.

Evaluation — retrieval metrics vs end-to-end, on your own data

Separate the two layers. RAG evaluation must be seen in two strands.

Retrieval metrics — recall@k, precision@k, MRR, nDCG. They measure "did the right chunk make it into the context?" and need a labeled query-to-relevant-document set.
End-to-end (generation) metrics — faithfulness/groundedness (is the answer actually supported by the retrieved context, i.e. no hallucination), answer relevance, answer correctness. Often scored with an LLM-as-judge.

Why separate them: a strong generator can mask weak retrieval (and vice versa). Look only at the end-to-end score and things can seem fine while retrieval is leaking. If retrieval recall is low, no prompt saves you — splitting the layers is how you see where the bottleneck is.

In production it helps to sort failures into two bins: (a) the right document was never retrieved, and (b) it was retrieved but generation ignored or misused it. Bucket (a) is fixed in the retrieval stage — chunking, hybrid, reranking — while (b) is fixed in the prompt, citation enforcement, or the model. Without this split, staring at the end-to-end score alone keeps you fixing the wrong thing.

Public benchmarks do not predict your corpus. Build a small labeled set from real queries (a few dozen is enough to start) and A/B each technique with and without. This is the RAG version of the eval-first principle: the eval set comes before the model. Once the eval exists, chunk size, fusion weights, and reranker choice stop being taste and become experiments.

A pragmatic build order

Don't add everything at once. This order usually pays off best.

Structure-respecting chunking with overlap first. If it leaks here, everything downstream leaks.
Hybrid search (BM25 + vector, fused with RRF) — the cheapest large win, because it recovers exact-match queries.
Reranking — retrieve wide, keep narrow. It lifts top-of-list precision.
Contextual retrieval if your documents are long and reference-tangled.
Query rewriting if you have many conversational or vague queries.

If your corpus is multi-tenant or permissioned, metadata filtering (access control above all) sits outside this order and should be a default from the start — a day-one requirement, not a later optimization.

And before any of these, build the small eval set — so you can answer with numbers whether each step actually helped.

Closing

RAG is not magic — bolting retrieval onto an LLM does not make the knowledge problem disappear. It just turns it into a retrieval problem, and retrieval is a decades-old, still-hard field. The good news is that the biggest gains here usually come from the most boring places: not a bigger model, but better chunking, hybrid search, and the habit of retrieving generously and then reranking.

If you take one thing away, take this: evaluate on your own data. The techniques above are a toolbox, not a recipe. On one corpus contextual retrieval changes the game; on another, simply turning on BM25 is the whole story. That difference shows up not in a benchmark but only in your queries.