Split View: RAG가 엉뚱한 답을 할 때 — 검색 실패와 생성 실패를 가르는 디버깅 절차

When RAG Answers the Wrong Thing — a Debugging Procedure That Separates Retrieval Failure from Generation Failure

Introduction — the answer is wrong. Where do you look first?

When a RAG system answers something odd about content that is plainly in the documents, most teams fix the prompt. They rewrite "ground your answer only in the provided context" in a firmer tone. And it usually does not get better.

Of course it does not. In many cases the model did not disobey the instruction; the answer simply was not in the context it received. RAG is a pipeline with retrieval and generation wired in series, and when the front stage is wrong there is nothing to fix in the back stage.

This post lays out a debugging procedure that keeps the order. Split the cause first, check the most common culprit next, and build the measurement system last. Skip the order and you will spend six months fixing prompts.

Step 1 — retrieval failure or generation failure

The first thing to do is look at the retrieved chunks with your own eyes. Any debugging that skips this step is guesswork. Yet, remarkably, many pipelines never log the retrieval results.

If the logs are there, one experiment separates the causes.

def diagnose(question, gold_passage, retriever, generate):
    """compare feeding the gold chunk directly against feeding the retrieval result."""
    retrieved = retriever(question, k=5)
    ctx_retrieved = "\n\n".join(c.text for c in retrieved)

    ans_normal = generate(question, ctx_retrieved)
    ans_oracle = generate(question, gold_passage)      # skip retrieval

    gold_in_ctx = gold_passage[:200] in ctx_retrieved

    if not gold_in_ctx and is_correct(ans_oracle):
        return "검색 실패"          # correct when handed the answer → front-stage problem
    if gold_in_ctx and not is_correct(ans_normal):
        return "생성 실패"          # wrong even when handed the answer → back-stage problem
    if not is_correct(ans_oracle):
        return "생성 실패 (근본)"    # cannot solve it even given only the gold document
    return "정상"

Run this diagnosis across the whole golden set and you get the ratio of the two failures. In my experience, early pipelines skew heavily toward retrieval failure, and within that, most of it comes from chunking.

Cases judged as generation failure split into three again. The answer was in the context but the model ignored it and answered from prior knowledge; the information was scattered across several chunks and the model failed to combine them; the contexts contradicted each other and the model picked the wrong side. All three call for different treatment, so do not lump them together.

The limits of "say you do not know when you do not know"

The first thing people try against generation failure is an instruction that blocks unsupported answers.

SYSTEM = """제공된 문맥만 근거로 답하십시오.
문맥에 답이 없으면 정확히 "문서에서 찾을 수 없습니다"라고만 출력하십시오.
각 주장 끝에 근거 청크 번호를 대괄호로 표기하십시오."""

This instruction works. There are just two things to know when you use it.

First, demanding abstention forcefully also raises the rate at which the model abstains on questions it could have answered. That is a precision-recall trade, not a pure improvement. So do not report only "the hallucination rate fell" — report how much the answer rate fell alongside it.

Second, requiring citations gives no guarantee that the numbers the model wrote match the actual evidence. A citation number is merely a verifiable format; the verification has to be done by your code. Add a step that separately checks whether the cited chunk really contains the claim, and only then does it become a signal you can trust.

Chunking is half the cause

If the verdict is retrieval failure, look at the chunks before you swap the embedding model. Most of the time it ends right there.

The typical damage looks like this. A table gets cut through the middle so the header and the data land in different chunks. A code block is severed mid-function. A paragraph beginning "in cases matching the condition above" is separated from the paragraph before it and there is no way to tell what it refers to. A line holding only a clause number drifts away from its body text.

Fixed-length splitting knows nothing about any of this structure. So the first improvement is almost always splitting that respects document structure. Use heading boundaries for markdown, section tags for HTML, layout analysis results for PDF as the primary boundaries, and only cut inside them when the length overflows. Keep tables and code blocks whole as a single chunk each and treat them as exceptions to the length limit.

Size and overlap are the next question. The arithmetic itself is simple.

import math

def chunk_stats(doc_tokens, size, overlap):
    stride = size - overlap
    n = 1 if doc_tokens <= size else math.ceil((doc_tokens - overlap) / stride)
    stored = n * size
    return n, stored / doc_tokens

for size, ov in [(256, 32), (512, 64), (1024, 128), (512, 256)]:
    n, infl = chunk_stats(50_000, size, ov)
    print(f"size={size:4} overlap={ov:3} → 청크 {n:3}개, 저장 토큰 {infl:.2f}배")
# size= 256 overlap= 32 → 청크 224개, 저장 토큰 1.15배
# size= 512 overlap= 64 → 청크 112개, 저장 토큰 1.15배
# size=1024 overlap=128 → 청크  56개, 저장 토큰 1.15배
# size= 512 overlap=256 → 청크 195개, 저장 토큰 2.00배

Set the overlap to half the chunk size and storage doubles. The vector storage cost itself is trivial: 112 chunks of 1536-dimensional fp32 embeddings is a little over 690KB. The real price is that the candidate list fills up with nearly identical content. If you pull the top 5 and four of them are overlapping fragments of the same paragraph, your effective context is one. Ten to fifteen percent of the chunk size is enough overlap in most cases.

There is no correct answer for size, though there is a direction. Small chunks have higher retrieval precision and insufficient context; large chunks are the reverse. If your questions are single-fact lookups, go small; if they ask for procedures or explanations, go large. If you cannot settle on one, retrieving small and then expanding with neighboring chunks before handing them over is a practical compromise.

Embedding similarity is not semantic similarity

There is a typical pattern to how embedding retrieval fails.

The query contains an exact identifier. Error codes, product numbers, function names, employee IDs — in embedding space these tokens are nearly indistinguishable from other identifiers of similar shape. A vector is a device for compressing meaning, and the value of such a token lies precisely in the exact string.

Negation and conditions are flipped. "Conditions under which a refund is possible" and "conditions under which a refund is impossible" have very high cosine similarity. Embeddings capture topic, not logic.

The query and the document have different surface forms. A short question and a long descriptive paragraph come from different distributions to begin with. This is why some embedding models require separate query and document prefixes, and dropping that prefix costs visible performance.

The first and third are largely solved by mixing in lexical retrieval. BM25 is strong on exact string matches and assigns high weight to rare tokens. The most reliable way to merge the two result lists is reciprocal rank fusion, which uses only the ranks and never normalizes the scores.

def rrf(rankings, k=60, top_n=10):
    """rankings: per-retriever list of document IDs, in rank order. No need to align score scales."""
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)[:top_n]

hits = rrf([
    bm25_search(query, k=50),     # lexical
    vector_search(query, k=50),   # semantic
])

The constant 60 is the value proposed in the original paper, and using it as-is in practice is generally fine. The advantage of this approach is that there is effectively only one value to tune; the disadvantage is that discarding score magnitudes means it cannot reflect a case where one retriever is overwhelmingly confident.

I will not claim hybrid always wins. If your domain has few proper nouns and identifiers and its sentences are mostly descriptive, pure vector retrieval may be better. But running two retrievers costs little and the implementation takes half a day, so trying it first and measuring is the reasonable order.

Reranking — when does it earn its keep

Reranking is the step where the first-stage retrieval pulls a generous candidate list and a more precise model reorders the top slice of it. A cross-encoder encodes the query and the document together and observes their interaction directly, so it is more accurate than making separate vectors for each.

The price is computation. Vector search encodes the query once, but a cross-encoder runs one forward pass per candidate.

def rerank_budget(n_candidates, ms_per_pair, retrieval_ms):
    total = retrieval_ms + n_candidates * ms_per_pair
    return total, total / retrieval_ms

for n in (20, 50, 100):
    total, ratio = rerank_budget(n, ms_per_pair=1.2, retrieval_ms=15)
    print(f"후보 {n:3}개 → {total:5.0f}ms (검색만 대비 {ratio:.1f}배)")
# 후보  20개 →    39ms (검색만 대비 2.6배)
# 후보  50개 →    75ms (검색만 대비 5.0배)
# 후보 100개 →   135ms (검색만 대비 9.0배)

ms_per_pair varies enormously with model size, chunk length, hardware and whether you batch. The 1.2 milliseconds above is a value I put in for illustration, so be sure to measure and replace it in your own environment.

The criterion is this. Reranking only earns its keep when the first-stage retrieval does put the answer in the candidate list but ranks it low. That is, recall@50 is high while recall@5 is low. Conversely, if recall@50 itself is low, reranking can do nothing. You cannot promote something that is not there. So measure those two numbers before you even consider reranking, and keeping that order will spare you a pointless addition of latency.

Using it as a final filter before handing the whole context to the LLM is also valid. Dropping low-relevance chunks reduces input tokens and lowers the chance of the model being pulled toward irrelevant context.

Attaching context to chunks

If the text being embedded is only the chunk body, then that chunk is retrieved without knowing which part of which document it belongs to. A paragraph reading "the exceptions to clause 3 are as follows" has no way to be retrieved on its own.

The fix is simple. Prepend positional information to the text you embed.

def contextualize(chunk, doc):
    header = " > ".join(filter(None, [doc.title, chunk.section, chunk.subsection]))
    return f"[{header}]\n{chunk.text}"

# build both the embedding and the BM25 index over this text.
# but keep the same header in the context handed to the LLM so attribution is possible.

There are two reasons the effect is large. On the retrieval side, repeating the document title and section name in every chunk reinforces the topic signal. On the generation side, the model becomes able to tell documents apart, which reduces mistakes like blending the product A policy with the product B policy.

Going a step further, you can pre-generate one or two sentences summarizing the whole-document context for each chunk with an LLM and attach that. It is a trade: pay a per-document cost once at indexing time in exchange for retrieval quality. Note that this cost scales with the number of documents and is real money going out, so trying the simple header attachment first and only considering this when it falls short is the right order.

Metadata is both a retrieval target and a filter. Store date, version, department and document type as fields and you can cut off the "last year's policy got retrieved" class of failure with a filter. This does not work well as a ranking adjustment; only a filter makes it certain.

The golden set and recall@k — no improvement without measurement

Every change so far has to be reversible, and for that you need numbers. What you need is pairs of a question and the IDs of the gold evidence chunks.

The best way to build it is to pull from real user question logs. If you have no logs, generate questions from the documents with an LLM, but always have a human review them. Generated questions tend to copy the source wording verbatim, so an evaluation set made only of those is biased in favor of lexical retrieval. Be sure to mix in questions with rephrased wording, questions that require synthesizing several chunks, and questions the documents do not answer. That last kind is the only way to measure the ability to abstain.

def recall_at_k(dataset, retriever, k):
    """share of questions whose gold chunk landed in the top k"""
    hit = 0
    for q, gold_ids in dataset:
        got = [c.id for c in retriever(q, k=k)]
        if any(g in got for g in gold_ids):
            hit += 1
    return hit / len(dataset)

def mrr(dataset, retriever, k):
    """mean reciprocal rank of the first correct hit. Read alongside recall when rank matters."""
    total = 0.0
    for q, gold_ids in dataset:
        got = [c.id for c in retriever(q, k=k)]
        for rank, cid in enumerate(got, start=1):
            if cid in gold_ids:
                total += 1.0 / rank
                break
    return total / len(dataset)

for k in (1, 5, 20, 50):
    print(f"recall@{k:2} = {recall_at_k(golden, hybrid_retriever, k):.3f}")

Pull these two numbers per k and the diagnosis becomes mechanical. Low recall@50 means an index and chunking problem; high recall@50 with low recall@5 means a ranking problem; both high with wrong answers means a generation problem.

One caution. If this evaluation set is small, most of the differences you see in it are noise. Do not make a deployment decision on the result that recall rose from 0.70 to 0.73 across 30 questions. How to handle sample size and significance is covered in LLM evaluation without the vibes.

Closing — look at the retrieval results first

Half of RAG debugging ends at "actually read the retrieved chunks." Yet this is the step most often skipped, either because the pipeline was built without logging the context, or because fixing the prompt looks faster.

So two things are enough for today. Log the retrieved chunk IDs and their source text on every request, and build a 50-question golden set and measure recall@5 and recall@50. Without those two, every later improvement is repair work done blind; with them, the cause is usually narrowed down within an hour.