Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — The Answer, Up Front
The Three Are Not Solving the Same Problem
Can You Inject Knowledge by Fine-Tuning? — Why Two Papers Look Opposite
- Before You Take the Agriculture Numbers at Face Value
- And What Happens When You Force New Knowledge In
Where Long-Context Breaks Down — Position, Length, Vocabulary
Is RAG Cheaper Than Long-Context? — What Self-Route Actually Measured
But the Benchmarks Themselves Are Contaminated — The Most Important Part of This Post
What Changed in 2026 — Caching Doesn't Make Long-Context Free
So What Should You Look At — What You Can Actually Measure
When Not to Use Each
What No One Has Answered
Closing
References

Introduction — The Answer, Up Front

"We want to build a chatbot on our own documents — should we use RAG, fine-tune, or just stuff everything into a one-million-token context?" This is probably the single most common question in LLM architecture. And most of the answers you find when you search are decision trees without a single source. "RAG if you need fresh information, fine-tuning if you need style" — not wrong, exactly, but you can't decide anything with it. It never says why, or how much, or where things break.

So this post answers using only what has been measured. Here is the conclusion, stated up front.

The three are not competitors to begin with. That fine-tuning fails to inject new facts into a model is a result measured repeatedly across papers. So if "knowledge" is your problem, you effectively have two options, not three.
The remaining choice (RAG vs long-context) is almost entirely decided by one property you can measure yourself — where the recall@k curve saturates on your corpus. That is not a benchmark; it is a number you can measure in half an afternoon.
And the part where I have to be honest — no study, as far as I could verify, compares all three on one common benchmark. Every paper uses different models, different datasets, different metrics. So you must not line up the numbers from paper A next to the numbers from paper B.

What follows is the evidence. For each number I will also say what it measured and what it did not.

The Three Are Not Solving the Same Problem

First we have to sort out the axes. The three approaches are often placed side by side, but they actually change different things.

Fine-tuning changes the weights. It is a tool for changing how a model answers.
RAG changes what goes into the prompt. It inserts only the pieces retrieval selected.
Long-context also changes what goes into the prompt. It just puts everything in, without selecting.

So RAG and long-context are two extremes of the same axis (what to put in the prompt), while fine-tuning is a different axis. The concluding sentence of Gekhman et al. (arXiv:2405.05904) captures this framing most compactly — the paper writes that its findings support the view that large language models acquire most of their factual knowledge during pre-training, and that fine-tuning teaches them to use that knowledge more efficiently.

If that sentence is right, then the plan to "teach the model our internal documents by fine-tuning" picked the wrong tool from the start. Yet in practice a case study pointing the opposite way also circulates. We have to resolve that contradiction first.

Can You Inject Knowledge by Fine-Tuning? — Why Two Papers Look Opposite

The negative side. Ovadia et al.'s "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" (arXiv:2312.05934) measures knowledge injection head-on. The conclusion is clear — unsupervised fine-tuning does give some improvement, but RAG consistently outperforms it, both on existing knowledge seen during training and on entirely new knowledge. And models struggle to learn new factual information through unsupervised fine-tuning.

The positive side. Microsoft's "RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture" (arXiv:2401.08406) looks like the opposite. In the abstract's own words — fine-tuning lifted accuracy by more than 6 p.p., and this stacked with RAG for a further 5 p.p.

Do the two results conflict? No. They measured different interventions.

The fine-tuning Ovadia et al. mean is unsupervised fine-tuning — that is, continual pre-training. The paper states it uses this method throughout the study: take the raw text as-is and have the model predict the next token. More important is the reason the paper gives for excluding supervised fine-tuning (instruction tuning) from the candidates entirely — instruction tuning does not necessarily teach the model new knowledge, and therefore instruction tuning alone cannot be a solution to the knowledge-injection problem.

The agriculture paper's pipeline, by contrast, fine-tunes on Q&A pairs. It generates question-answer pairs from the documents and then trains on them. A completely different intervention.

So, to sum up: the single word "fine-tuning" points to at least two different operations, and the two papers each measured a different one. Blog posts cite the two interchangeably, and out of that mix comes the conclusion-free conclusion that "fine-tuning sometimes works and sometimes doesn't."

Before You Take the Agriculture Numbers at Face Value

If you are going to cite the positive evidence, fairness means looking at its limits too. The paper's Table 18 reads like this.

Model                Fine-tuned   Accuracy    +RAG
Llama-2-chat 13B                  76% ±2%     75% ±2%
Vicuna                            72% ±2%     79% ±2%
GPT-4                             75% ±3%     80% ±4%
Llama2 13B               O        68% ±3%     77% ±2%
GPT-4                    O        81% ±5%     86% ±2%

The arithmetic checks out exactly. For GPT-4, 75% → 81% is the abstract's "more than 6 p.p.," and 81% → 86% is "a further 5 p.p. with RAG." But three things need pointing out.

The error bars overlap. 75% ±3% spans 72-78, and 81% ±5% spans 76-86. They overlap over the 76-78 range. And the paper does not define whether this ± is a standard deviation or a confidence interval, nor does it offer any significance test. Reading 6 p.p. as a "sure improvement" goes beyond the evidence.
The table contains a case where domain fine-tuning actually lost. Llama2 13B fine-tuned on agriculture data scores 68%, below the 76% of Llama-2-chat 13B, a general-purpose instruction-tuned model. (The starting checkpoints differ — the former uses the base model, the latter the chat model. Still, it is clear that the intuition "tune on domain data and you win on domain tasks" does not hold automatically.)
The training labels are LLM-generated, and the grader is an LLM too. The fine-tuning answers were generated by Llama2-13B-chat with RAG attached, and evaluation was done by GPT-4. In other words, this experiment is in large part closer to "distilling the output of a RAG pipeline into the model."

And What Happens When You Force New Knowledge In

Gekhman et al. (arXiv:2405.05904) go a step further. Controlling the fraction of fine-tuning examples that introduce new knowledge, they observed that examples carrying new knowledge are learned noticeably more slowly than examples consistent with what the model already knows. And once they are finally learned, those examples linearly increase the model's tendency to hallucinate.

In the paper's own words — the results show that introducing new factual knowledge through fine-tuning carries a risk. The combination "learns slowly, and hallucinates more once it has learned" is the worst case in practice, because it shows up as training loss going down while production quality gets worse.

Practical implication: do not use fine-tuning to inject facts that change often. Use fine-tuning to change behavior — format, tone, domain terminology, output structure. How the hyperparameters tangle together when you do that with LoRA is covered separately in LoRA rank and learning rate.

Where Long-Context Breaks Down — Position, Length, Vocabulary

So why not just put everything in? The context window is a million tokens. Here there are three distinct, measured failure modes.

Position. Liu et al.'s "Lost in the Middle" (arXiv:2307.03172) measured performance while varying the position of the relevant information. The result is the well-known U-shaped curve — performance is highest when the relevant information sits at the very beginning or the very end of the input, and drops sharply when it is in the middle of a long context. The paper is emphatic that this shows up even in models that explicitly bill themselves as long-context.

Length. Databricks' "Long Context RAG Performance of Large Language Models" (arXiv:2411.03538) ran RAG workflows across 20 open-source and commercial models, varying context length from 2,000 to 128,000 tokens (up to 2 million where possible). It uses three domain datasets. The key result is in the abstract — retrieving more documents can raise performance, but only a very few of the latest SOTA models can maintain consistent accuracy at lengths beyond 64k tokens.

The body is more specific. For most of the models analyzed, RAG performance rose only up to 16k-32k tokens. Qwen 2 70B holds consistent accuracy out to 64k, Llama 3.1 405B starts to decline after 32k, and GPT-4-0125-preview declines after 64k.

More interesting is that the way they fail differs from model to model. Claude 3 Sonnet frequently refused to answer, citing copyright concerns; Gemini 1.5 Pro failed the task on long contexts because of an overly sensitive safety filter; Mixtral-8x7B emitted repeated or random content; and DBRX often ignored the instruction and summarized the content instead beyond 16k. So even "long-context degrades" is not a single phenomenon — refusal, safety filters, repetition, and instruction-ignoring each show up in a different model.

Vocabulary. NoLiMa (arXiv:2502.05167) is the most uncomfortable result. The paper's point is that conventional needle-in-a-haystack tests have literal word-for-word overlap between the needle and the haystack, which the model can exploit to make the task easy. So they designed the questions to minimize lexical overlap with the needle, and when they evaluated 13 models claiming to support 128K or more, models that did well on short contexts fell sharply as length grew. At 32K, 11 models dropped below 50% of their own short-length baseline, and even GPT-4o, which had been at the top, fell from a near-perfect baseline of 99.3% to 69.7%.

The important implication here: passing needle-in-a-haystack is not evidence that long-context works. That test was allowing a shortcut — lexical matching. Real questions usually do not use the same words as the document.

An honest limitation: the models the three studies above evaluated are all from the 2023-2024 era (NoLiMa was revised in 2025). I have not found a public study that re-measures, with the same methodology, whether the same curves appear on 2026 frontier models.

Is RAG Cheaper Than Long-Context? — What Self-Route Actually Measured

On the cost side, the most-cited work is Google's Self-Route paper (Li et al., arXiv:2407.16833). Let's look at the actual table. Contriever retriever, averaged over nine datasets.

Gemini-1.5-Pro       LC 49.70 | RAG 37.33 | Self-Route 46.41 | Tokens 38.39%
GPT-4O               LC 48.67 | RAG 32.60 | Self-Route 48.89 | Tokens 61.40%
GPT-3.5-Turbo        LC 32.07 | RAG 30.33 | Self-Route 35.32 | Tokens 38.85%

Here is how to read it. When resources are ample, LC consistently beats RAG — 49.70 vs 37.33 is not a small gap. But Self-Route (easy questions to RAG, and only those RAG judges "unanswerable" to LC) comes close to LC while using just 38-61% of LC's tokens.

Why this works is the paper's real finding — on 63% of queries the model predictions were exactly identical, and on 70% of queries the score difference was under 10. And the paper adds one thing: identical predictions are not necessarily correct ones, and RAG and LC tend to make not just the same correct answers but the same mistakes. In other words, on most questions the two approaches are indistinguishable to begin with. There is no reason to use the expensive one.

The cost figures must be cited with care. The paper's body says cost dropped 65% for Gemini-1.5-Pro and 39% for GPT-4O. For GPT-4O the arithmetic is exactly right — it uses 61.40% of the tokens, so 38.6% savings, which rounds to 39%. But Gemini-1.5-Pro uses 38.39% of the tokens, so the savings should be about 61.6%, and in Table 2 with the Dragon retriever the token ratio is 37.87% (about 62.1% savings). The 65% the paper states is off by about 3 p.p. from the paper's own table. Not a big deal, but you should know it before copying "65%" over verbatim.

And the decisive limitation: the models this paper used are gpt-3.5-turbo-0125, gpt-4o-2024-05-13, and Gemini-1.5-Pro. 2024 models. The datasets are LongBench (average context around 7k words) and ∞Bench (average around 100k tokens), so the "long-context" here is not a million tokens but 7k-100k.

There is a 2026 follow-up. "Route Before Retrieve" (arXiv:2605.10235, May 2026) points out that Self-Route is a passive fallback — it tries RAG first and moves to LC on failure — and proposes deciding the routing before retrieval, using metadata (document type, length, opening snippet). The direction itself is a signal: it keeps Self-Route's insight of "a different tool per question" while pushing to cut that cost further.

But the Benchmarks Themselves Are Contaminated — The Most Important Part of This Post

There is one finding that shakes all the numbers so far.

"Long Context vs. RAG for LLMs: An Evaluation and Revisits" (arXiv:2501.01880) re-examines prior work and points out inconsistencies with their key claims. And it does something very simple that no one had done — it filters out the questions that can be answered without any external context. The logic: modern LLMs can answer many questions from knowledge encoded in their parameters alone, so to judge which side helps more on long documents, you have to keep only the questions the model cannot get right without context. It uses GPT-4o for the filtering and applies strict exact-match grading.

The results are these.

Dataset            All Qs     Kept    Retention
Coursera              172       54     32%
NQ                  1,109      373     34%
NovelQA             2,283      869     38%
2WikiMHQA           2,300    1,036     45%
HotpotQA            2,200    1,113     51%
MultiFieldQA          150      121     81%
NarrativeQA         2,211    1,880     85%
QASPER              2,718    2,674     98%
QuALTY              2,725    2,725    100%
TOEFL-QA              962      962    100%
MultiDoc2Dial         158      158    100%
─────────────────────────────────────────────────
Total              19,188   13,628     71%

On Natural Questions the retention rate is 34%. Put differently, GPT-4o gets roughly two-thirds of NQ questions right with no context given at all. Coursera is 32%, NovelQA 38%.

Why this is fatal — if you measure "RAG vs long-context" scores on top of these benchmarks, a large part of that score is measuring the model's pre-training knowledge, not retrieval. And one of the three datasets the Databricks study cited earlier used is exactly NQ.

The practical conclusion that follows is the most important in this post. Your internal documents are not in the model's pre-training data. So you must not carry a public-benchmark RAG-vs-LC ratio over to your corpus as-is. On the benchmark the model already knew a large share of the answers; on your corpus it knows none. The conditions are different.

In addition, this paper's own conclusion differs subtly from the earlier work — on QA benchmarks, especially Wikipedia-based questions, LC generally outperforms RAG, summary-based retrieval is comparable to LC, and chunk-based retrieval lags. But on conversational queries and general queries, RAG has the edge. Which means "RAG wins/loses" flips depending on the kind of dataset.

What Changed in 2026 — Caching Doesn't Make Long-Context Free

Self-Route's cost model is simple. As the paper itself notes, LLM API pricing is usually based on the number of input tokens, so using fewer tokens is cheaper by that much. In 2024 that was the right model.

In the meantime, prompt caching became commonplace. Put a stable prefix in the cache and reuse is far cheaper. Take the structure Anthropic documents for its own models as an example: a cache read is about 0.1x the base input price, and a cache write is 1.25x at a 5-minute TTL and 2x at a 1-hour TTL. The minimum cacheable prefix length also varies by model (e.g., 4,096 tokens for Opus 4.8, 2,048 tokens for Sonnet 4.6). A natural objection arises here — if the corpus is stable and queries repeat, doesn't putting the whole thing in the cache collapse Self-Route's cost logic?

Only partly. "Beyond the Context Window" (arXiv:2603.04814), from March 2026, actually measured this. It builds a cost model that includes prompt caching and compares a fact-based memory system (Mem0) against long-context reasoning. The key sentence: long-context reasoning pays a per-turn cost that grows in proportion to context length even with caching applied, whereas a memory system's per-turn read cost is nearly fixed after a one-time write step.

The measured values are these (LongMemEval, 500 conversations, average context 101,601 tokens).

                        Cost
Memory write (once)     $0.0435 / conv      (total $21.76, 500 conversations)
Memory read             $0.0013 / query     (total $0.65, 500 questions)
LC GPT-5-mini           $0.0293 / request   (total $14.79, 504 requests)
LC first turn (modeled) $0.0265
─────────────────────────────────────────────────────────────
Break-even: ~10 turns (100k context). At N=20, memory is 26% cheaper.
As context grows, the break-even drops from 13 turns -> 9 turns.

The arithmetic checks out ($21.76 divided by 500 is $0.0435, $14.79 divided by 504 is $0.0293). So when the turn count is low, long-context is cheaper; when it is high, retrieval is cheaper. Caching only lowers the slope of this curve; it does not reverse its direction.

But there is something you must always say alongside this paper. On accuracy, long-context won by a wide margin.

Dataset           Memory system   LC GPT-5-mini   LC GPT-OSS-120B
LoCoMo                  57.68           92.85            81.69
PersonaMem v2           62.48           69.75            60.50
LongMemEval             49.00           82.40            48.20

On LoCoMo, 57.68 vs 92.85; on LongMemEval, 49.00 vs 82.40. The cheaper side lost badly on accuracy. The paper gives the reason too — passing the full history preserves every detail, but the memory pipeline compresses roughly 101,600 tokens of conversation into an average of 2,909 tokens of atomic facts per user, so information loss in the compression is unavoidable.

Scope limit: this study measured the memory system of a persistent conversational agent, not document-corpus RAG. The insight about cost structure ("retrieval is fixed per turn; long-context scales with length even with caching") generalizes, but the accuracy numbers belong to a conversational-memory benchmark. Do not carry them over as-is.

The effect of caching itself has been measured separately. "Don't Break the Cache" (arXiv:2601.06007) measured caching across more than 500 agent sessions over three providers and reports cost savings in the 41% to 80% range and TTFT improvements in the 13% to 31% range. That said, this paper too measured an agentic tool-calling workload (DeepResearch Bench), and notes that under a full-context caching strategy there was no TTFT improvement, or even a slight regression. And as the paper itself states, to the authors' knowledge no prior work had quantified these savings on multi-turn agentic tasks.

So What Should You Look At — What You Can Actually Measure

Now the decision rules. The core here is don't read benchmarks; measure your own corpus. And there is only one first number to measure.

Step 1: Plot the recall@k curve (this decides almost everything)

The table the Databricks paper tucked into its appendix is effectively the template for this methodology. It measured each dataset's recall@k by context length with text-embedding-3-large.

Context length       2k     4k     8k    16k    32k    64k    96k   128k
Databricks DocsQA  0.547  0.856  0.906  0.957  0.978  0.986  0.993  0.993
FinanceBench       0.097  0.287  0.493  0.603  0.764  0.856  0.916  0.916
NQ                 0.845  0.992  1.000  1.000  1.000  1.000  1.000  1.000

In the paper's own reading — the context length at which retrieval recall saturates differs by dataset. NQ saturates early, at 8k, while DocsQA and FinanceBench saturate at 96k and 128k respectively.

This curve is your answer.

If it saturates at small k (like NQ) — the answer is local. Retrieval already pulls all the evidence within 8k. Long-context adds nothing and only piles on distractors. Use RAG. Growing the context here is a pure loss.
If it keeps rising to 96k-128k (like FinanceBench) — the evidence is spread widely. This is the only regime where long-context can earn its keep. But there is a condition (Step 2).
If it never gets good — FinanceBench is still 0.916 at 128k. It never reaches 100%. Either retrieval is the bottleneck (a chunking/embedding problem), or it was never a retrieval problem in the first place. If the latter, try hybrid search and reranking first, and if that still doesn't work, the question may be demanding "synthesis" rather than "retrieval" — that is the point to consider Graph RAG.

And you must read the warning the Databricks paper attaches alongside it — retrieval accuracy increases monotonically, but that does not mean RAG accuracy increases monotonically. Grow the context and recall rises while generation gets worse. So the optimum is not at either end but somewhere in the middle. You can only find it by plotting the curve.

Measuring it is not hard. Build 50-100 questions whose gold document you know, vary k, and count the fraction where the gold document lands in the top k. This is a pure retrieval metric that needs no LLM call, so it is cheap and fast.

Step 2: Check that the model holds up at that length

Getting "rises to 96k" from Step 1 does not mean you should feed in 96k. In the Databricks measurements, most models had already bent at 16k-32k. And as NoLiMa showed, when the question and the document share no vocabulary, most models collapse at 32K. Do your questions use the same words as the documents? Usually not.

Step 3: Look at the update frequency

If the corpus changes often, fine-tuning drops out of the running (retrain on every change). Long-context also loses its caching benefit (change the prefix and the cache is invalidated, and the default TTL is on the order of minutes). For a frequently changing corpus, RAG is the default.

Step 4: Knowledge or behavior?

"Answer in our voice" / "only in this JSON schema" / "understand these domain abbreviations" → fine-tuning. "What does this document say" → retrieval. Do not mix the two. That is exactly what the Ovadia and Gekhman results are telling you.

Step 5: Budget by turn count

How many times do you throw questions at the same context? Borrowing the structure of "Beyond the Context Window" — few turns and long-context is cheaper, many turns and retrieval is cheaper. At 100k context the break-even was about 10 turns. One question per document may favor long-context; thousands of queries against the same corpus and retrieval wins overwhelmingly.

When Not to Use Each

This is usually the most quotable part.

Don't use fine-tuning for fact injection. It learns slowly and hallucinates more once it has (Gekhman et al.). A project to fine-tune on an internal wiki is almost always the wrong tool choice.
Don't use long-context just because the window is a million. Window size is capacity, not capability. Per NoLiMa, at 32K, 11 of 13 models fell below half their own baseline.
Don't lean on passing needle-in-a-haystack as evidence. That test allows a lexical-matching shortcut.
Don't grow the context when recall saturates at small k. You only add distractors and walk into lost-in-the-middle.
Don't carry a public-benchmark RAG-vs-LC ratio over to your corpus. Roughly two-thirds of NQ questions are solvable without context. Your documents are not.
And the numbers in this post are not your corpus's answer either. They all came from someone else's corpus, someone else's models.

What No One Has Answered

This is the part to state honestly. Below are the questions that, as far as I could verify, published research has not answered.

No study compares all three on one common benchmark. RAG vs LC (Self-Route, 2501.01880) and RAG vs fine-tuning (Ovadia, the agriculture paper) each exist, but they use different models, datasets, and metrics. So a calculation like "RAG is 12 points below LC and fine-tuning adds 6 p.p., so RAG+fine-tuning would…" does not hold. I did not do it in this post either.
No study re-measures RAG vs LC on 2026 frontier models with caching factored in. Self-Route uses 2024 models and a token-based cost model. A caching-aware cost model was built by 2603.04814, but its target is conversational memory, not document RAG.
Even the agent-workload effect of caching was only measured recently. 2601.06007 states that, to its authors' knowledge, no prior work had quantified these savings on multi-turn agentic tasks.
Controlled comparisons of knowledge injection across fine-tuning methods are lacking. Unsupervised continual pre-training (Ovadia) and supervised Q&A tuning (agriculture) have never been compared on the same data and the same evaluation. So "does supervised Q&A tuning achieve knowledge injection" is still open — the agriculture paper's evidence has overlapping error bars and LLM-generated labels.

The very existence of these gaps matters in practice. If someone asserts "research shows RAG is X% better than fine-tuning," ask what fine-tuning that X measured, and on what data.

Closing

The common answer to this question is useless not because it is wrong but because it is undecidable. "RAG if it's fresh information" is correct, but there is no way to know whether my problem falls under it.

Keep only what has been measured and the picture settles like this. Fine-tuning is not a knowledge-injection tool — this has been confirmed repeatedly across papers, and the case study that looks like the opposite measured a different intervention. Long-context breaks down along the three axes of position, length, and vocabulary, and the real limit of most models is an order of magnitude smaller than the advertised window size. RAG and long-context give the same answer on most queries to begin with — 63% were exactly identical per Self-Route. And the benchmarks all these numbers ride on were made of questions where one-third to two-thirds are solvable without any context.

So the practical answer is not to read more benchmarks but to plot one curve yourself. Your corpus, 50 of your questions, recall as you vary k. If it saturates at small k, use RAG; if it rises all the way, consider long-context but confirm the model holds up at that length; if it never rises, look again at the problem definition, not retrieval.

The question comes first, not the tool. And the nature of that question is written only in your data, not in someone else's paper.