Did Context Engineering Replace Prompt Engineering — What's Measured and What Isn't

Introduction — Answering the Question First
When Did the Term Appear, and From Whom?
One Measurement That Shows They're Different Jobs
The Context Window and the Effective Context Are Different Numbers
So Did 2026 Models Fix This?
Just How Bad Is Lost-in-the-Middle?
So Should You Just Organize the Context Neatly? — Where the Measurement Went the Other Way
Is "Attention Budget" a Measured Concept?
Is Compaction Really Low-Loss?
So What Should You Do? — A Decision Rule
When Context Engineering Is Not the Answer
Closing
References

Introduction — Answering the Question First

You'll often see the claim these days that "context engineering has replaced prompt engineering." The short answer: it's not replacement, it's inclusion. And that's not my opinion — it's what the people who popularized the term wrote themselves. When Karpathy defined context engineering, he listed its components as "task descriptions and explanations, few-shot examples, RAG, tools, state and history, compacting" — few-shot examples and task descriptions are just prompt engineering. Anthropic went further, writing that it "sees context engineering as the natural progression of prompt engineering." No primary source said "dead." That sentence was born in a headline.

But stopping there gets you only halfway. Inclusion is just terminology; the real question is "so is there anything new to learn?" That one you can answer by measurement — and the answer is "yes." The cleanest experiment is one Chroma ran in July 2025. It held the same question, the same task, and the same wording fixed and changed only the amount of surrounding context (about 300 tokens containing just the relevant material vs. about 113k tokens that also included irrelevant material). Performance dropped across all 18 models. If not a single character of the prompt changed and the numbers still moved, that's an axis prompt engineering can't touch. That's the value this term earned.

This post exists to strip away the hype and separate what has been measured from what gets repeated without measurement. Strategies for using the context window itself I already covered in LLM Utilization Strategies in the 1M Context Window Era, and practical patterns for designing agent memory in Context Engineering — How to Design Memory for AI Agents. Here I audit the evidence itself that those posts took as given.

When Did the Term Appear, and From Whom?

Laid out chronologically:

June 12, 2025 — Cognition's Walden Yan publishes "Don't Build Multi-Agents." He casts context engineering as the "next step" beyond prompt engineering and defines it as "doing this automatically inside a dynamic system." And he pins it as the "number one job" of an engineer building agents.
June 19, 2025 — Shopify CEO Tobi Lütke writes on X that he "prefers the term context engineering over prompt engineering." His definition: "the art of providing all the context necessary for the task to be plausibly solvable by the LLM."
June 25, 2025 — Karpathy amplifies it. That's the sentence quoted above.
June 27, 2025 — Simon Willison, citing the two tweets, writes that the term "seems likely to stick." His observation is interesting — prompt engineering was misunderstood by the public as "a fancy name for typing something into a chatbot," whereas with context engineering the meaning the public infers is likely closer to the intended one. In other words, much of the motivation for the term swap was connotation, not technique.
July 17, 2025 — exactly one year ago today, a survey sweeping over 1,400 papers (arXiv 2507.13334) goes up, giving the term an academic veneer.
September 29, 2025 — Anthropic posts "Effective context engineering for AI agents" on its engineering blog. This is the moment a vendor officially adopts the term.

One thing worth noting here. Cognition's post came out a week before the tweets, and three sources (Cognition, Karpathy, Anthropic) each independently produced a definition with the same structure — a superordinate concept that contains prompt engineering as a subset. No one called it replacement.

One Measurement That Shows They're Different Jobs

Enough of the term's lineage; let's go to the evidence. There's one experiment that shows "context engineering is a different job" by measurement, not assertion — Chroma's technical report "Context Rot: How Increasing Input Tokens Impacts LLM Performance" (Hong, Troynikov, Huber, July 14, 2025).

The core design is this. It uses a conversational QA benchmark called LongMemEval, but poses the same question in two ways.

Focused condition:  question + only the relevant parts needed to answer   -> ~300 tokens on average
Full condition:     question + relevant parts + all the irrelevant text   -> ~113k tokens on average

Question identical. Task identical. Prompt wording identical.
What changed: what surrounds the relevant information, and how much of it.

The result, in Chroma's own words, is that "all models perform significantly better on focused prompts than on full prompts." The models evaluated numbered 18 in total, including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3.

Why is this decisive? Because the experiment held every prompt-engineering variable fixed. The wording, the examples, the instructions — all unchanged. Not one of the knobs prompt engineering can touch was touched, and the numbers still moved. So the claim that "context engineering is a rebranding of prompt engineering" is refuted by this single experiment. Conversely, "prompt engineering is dead" is also wrong — that performance is good under the focused condition means the prompt still works. The two axes are orthogonal.

There are honest caveats to add. The authors selected the knowledge-update, temporal-reasoning, and multi-session categories from LongMemEval_s, and filtered out 38 ambiguous or unanswerable prompts, leaving 306. Grading is LLM-as-judge, and the authors state they calibrated this judging against human judgment (manually labeling about 600 LongMemEval outputs). Since Chroma is a vector-DB company, you should factor in that it has an interest in the conclusion "reduce context with retrieval." That said, the codebase is public and reproducible.

The Context Window and the Effective Context Are Different Numbers

A number written as "1M context" is the limit at which input is not rejected, not the limit at which performance holds. The thing that measured this head-on is RULER (Hsieh et al., arXiv 2404.06654, COLM 2024).

RULER's methodology is clear. It measures the average score across 13 tasks and, taking Llama2-7B's 85.6% at 4K as the baseline, defines the maximum length that stays above this line as the "effective length." The choice of baseline is itself arbitrary, but what matters is that the paper says so and fixes it. Here is part of the results for 17 models.

Model	Claimed length	Effective length	4K	32K	128K
Gemini-1.5-Pro	1M	>128K	96.7	95.9	94.4
GPT-4	128K	64K	96.6	93.2	81.2
Llama3.1 (70B)	128K	64K	96.5	94.8	66.6
Qwen2 (72B)	128K	32K	96.9	94.1	53.7
Yi (34B)	200K	32K	93.3	87.5	77.3
GradientAI/Llama3 (70B)	1M	16K	95.1	85.4	72.1
LWM (7B)	1M	`<4K`	82.3	69.1	65.0

The paper's sentence is this — "nearly all models fall below the baseline before reaching their claimed context length." LWM claims 1M yet can't even clear Llama2-7B at 4K. In other words, claimed length does not predict effective length, and doesn't even preserve the ordering — GradientAI/Llama3, which claims 1M (effective 16K), sits far below GPT-4, which claims 128K (effective 64K).

A caveat that must be attached here: these are 2024 models. So "is it still true?" is the natural question, and the answer is the next section.

So Did 2026 Models Fix This?

They didn't. And the most recent thing to measure it is ATLAS (arXiv 2605.28079, submitted May 27, 2026). It evaluated 26 models on a fixed grid from 8K to 1M across 6,438 instances.

Two of the findings ATLAS reports from its own measurements land directly on practice.

First, rankings flip with length. The authors aggregated scores up to 128K and up to 1M separately; 7 models moved by two or more ranks, and individual rank gaps widened to as much as 12 places. Even the reported No. 1 splits — at 128K, Gemini-3.1-Pro-Preview leads; at 1M, Claude-Opus-4.6 does. Which means the model you picked off a 128K benchmark is not guaranteed to be the best at 1M.

Second, being good at retrieval doesn't mean being good at use. In the abstract's words, "strong retrieval performance may not transfer to downstream use," and the primitive-operations layer and the applied-workload layer share only 61% of the cross-model variance. Don't flip that arithmetic into "so the rest is wrong" — what this figure says is that the two layers' cross-model performance variation points, in substantial part, in different directions, so you can't use a single NIAH-style retrieval score to stand in for the ranking of applied performance. The authors' conclusion: "report by capability and by length, not as a single headline score."

ATLAS is a preprint that has not yet been peer-reviewed, and its numbers are author-measured. That said, it published its methodology (fixed grid, AUC-based scoring, uncertainty propagation) and made its sub-components auditable, which puts it among the efforts that raised their own verifiability.

Here comes the real meta-point. The same phenomenon has reproduced across four generations.

2023  Lost in the Middle    GPT-3.5-Turbo, Claude-1.3 generation
2024  RULER                 GPT-4, Llama3.1, Qwen2 generation
2025  NoLiMa / Context Rot  GPT-4o, GPT-4.1, Claude 4, Gemini 2.5 generation
2026  ATLAS                 Gemini-3.1-Pro, Claude-Opus-4.6 generation

If it were a one-generation flaw, you could wait for the next model to fix it. Four generations running, and it's not something to wait out but something to design around. That is the single, sufficient reason context engineering is more than a buzzword.

Just How Bad Is Lost-in-the-Middle?

The most cited, and most often mangled, paper. The actual claim of "Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., arXiv 2307.03172, published in TACL) is this.

Place the document containing the answer at the beginning or end of the context and performance is highest; place it in the middle and it drops sharply. The bias toward the start is called primacy bias, the bias toward the end recency bias, and together they trace a U-shaped curve.

The effect size matters, and here are the paper's explicit numbers. GPT-3.5-Turbo's multi-document QA performance can drop by more than 20%, and in the worst case its performance in the 20- and 30-document settings is lower than giving it no documents at all (closed-book, 56.1%). For reference, the oracle setting that provides only the single answer document scores 88.3%.

It's easy to miss the weight of that sentence, so read it again. You put the document containing the answer into the context, and if its position is bad, that's worse than not putting it in at all. It is the most compact counterexample there is to why "just dump everything in and let the model find it" is dangerous.

There's a less-cited but more painful result the paper also reports — extended-context models often showed the same performance as their base models, and the authors write of this that "extended-context models are not necessarily better at using their input context." Enlarging the context window and using the context well are different things.

Caveat: these are 2023 models (GPT-3.5-Turbo, Claude-1.3). So you can't carry these numbers straight over to 2026 models. But the phenomenon itself has reproduced — NoLiMa (Modarressi et al., arXiv 2502.05167) removed the lexical overlap between the needle and the question so that literal matching couldn't be used, then re-evaluated 13 models claiming 128K or more; at 32K, 11 models fell below 50% of their short-context (<1K) baseline. Even top-ranked GPT-4o dropped from 99.3% to 69.7%. The authors add that neither CoT prompting nor added reasoning ability prevented this degradation — meaning prompts don't fix it.

So Should You Just Organize the Context Neatly? — Where the Measurement Went the Other Way

Here a piece of conventional wisdom breaks. Chroma compared the structure of the haystack (the irrelevant filler text) under two conditions — a set of essays with their original logical flow vs. the same sentences randomly shuffled to destroy local coherence. Common sense says the well-organized version should do better.

The result was the opposite. Across all 18 models, the shuffled haystack performed better. Chroma's interpretation: you'd think a needle lodged inside a coherent essay would stand out by breaking the logical flow, but in practice it works the opposite way.

This result has to be handled carefully. It is an experiment about the structure of the surrounding filler text, not a prescription to "shuffle your context." And Chroma itself draws the line in its limitations section — it can't explain why. But the direction is clear. "Organize the context to be easy for humans to read and the model will read it well too" is not a verified proposition. If one point where intuition flips has already been measured, then the rest of your intuitions are just intuitions until measured too.

Is "Attention Budget" a Measured Concept?

From here on is probably the most quotable part of the post. The phenomena above usually come with an explanation attached, and the status of that explanation differs from that of a measurement.

Anthropic's post frames the mechanism like this — because a transformer has every token attend to every other token, n tokens produce n² pairwise relationships, and as length grows the model's ability to capture these relationships is "stretched thin." And because the model learns its attention patterns from a training distribution where short sequences are more common, it has less experience with long-range dependencies. This is where the phrase "attention budget" comes from — a metaphor in which every new token spends a little of that budget.

It's plausible, and probably points the right way. To their credit, Anthropic hedges appropriately — they write that it's "a gradient of performance, not a hard cliff," and state explicitly that models remain capable even at long context. But honestly, this is an explanation, not a measurement. The post contains no ablation that isolates the n² term as the cause. "Attention budget" is a metaphor for tying our observations into a story, not a quantity you can read off an instrument.

And here's the decisive part. Chroma — the side that actually measured this phenomenon — writes in its limitations section that it does not explain the mechanism of the degradation and does not have a definitive answer for why it happens. It adds that uncovering that would require mechanistic-interpretability research and is beyond the scope of the report.

To summarize:

Measured             : as input grows, performance degrades non-uniformly   (Chroma, NoLiMa, RULER, ATLAS)
Explained            : n^2 attention, training distribution, "attention budget"   (vendor blogs, no ablation)
The measurers' stance: "we don't know why either"                          (Chroma limitations section, explicit)

The phenomenon is robust; the mechanism is unresolved. This distinction matters in practice too, because if you take the mechanism story as fact you'll want to deduce prescriptions from it ("it's n², so cutting tokens should improve things quadratically"). No measurement supports that deduction.

Is Compaction Really Low-Loss?

The standard prescription when context overflows the window is compaction — summarize the conversation and start a new window from that summary. Anthropic's post even describes how Claude Code does it: it passes the message history to the model to summarize, preserving architectural decisions, unresolved bugs, and implementation details while discarding redundant tool output, and passes along the five most recently accessed files.

And it writes that compaction compresses the contents of the context window "with high fidelity" so the agent can continue "with minimal performance degradation."

No measurement backing that sentence is public. How high the fidelity is, what percentage the degradation is, on what eval set it was measured — none of it. The same post does honestly admit compaction's risk ("overly aggressive compaction can lose subtle context whose importance only becomes clear later"). But there is no number telling you where that tradeoff sits.

Cognition is the same. Walden Yan recommends keeping a separate model that compresses the history into key information, writes that it's "hard to get right," and says Cognition actually fine-tuned a small model for it. Again, no numbers. And he honestly adds — you'll still hit a limit in the end.

So what public numbers are there? The closest ones are what Anthropic released when it announced its context-management features on September 29, 2025.

Measurement	Reported value	Condition
Memory tool + context editing	39% improvement over baseline	internal eval set for agentic search
Context editing alone	29% improvement	same
Token consumption	84% reduction	100-turn web search evaluation

These numbers are vendor-measured, and the eval set is not public. The number of tasks, the composition of the baseline, the variance — none are given. Reproduction is impossible in principle.

More important is that these are not compaction's numbers. Context editing deletes old tool calls and results from the window, whereas compaction summarizes the conversation. The mechanisms differ. Yet in practical discussion the two often get blurred, and I see "39% improvement" cited as if it were evidence for compaction. It isn't.

So the precise statement is this. The whole industry calls compaction essential, yet no public figure measures compaction's summary fidelity. This is not an accusation but a map — go in knowing this is a blank, and instead of borrowing someone else's numbers you'll measure before-and-after compression on your own eval set. That's the only honest path.

So What Should You Do? — A Decision Rule

I won't end with "it depends." There's a sequence that follows directly from the evidence above.

Step 1. Measure the focused/full gap first. Just replicate Chroma's experimental design. Pick 30–50 of your eval questions and run each in (a) a version with only the relevant information hand-picked by a human, and (b) a version with the full context your actual pipeline supplies. The difference between these two numbers is the ceiling on what context engineering can earn in your system.

Step 2. If the gap is small, stop. If focused and full are similar, the model is using your context well. In that case you need no compaction, no memory layer, no graph. Go back to prompt and task design. Stopping at this step is the right answer for most teams.

Step 3. If the gap is large, start with retrieval. The biggest lever is always "don't put irrelevant things in to begin with." Neither lost-in-the-middle nor context rot acts on tokens you never inserted.

Step 4. Measure the effective context on your own task. A vendor's claimed length is an input limit, not a performance limit. Do a miniature of what RULER did — fix the task, grow only the length, and draw the curve. As ATLAS concludes, you need a curve, not a single score.

Step 5. Compaction last, and only after measuring. Run before-and-after compression on the same eval set and see for yourself how much you lose. Since no public reference figure exists, this is mandatory, not optional.

When Context Engineering Is Not the Answer

Don't trust any post that leaves this part out.

When the context is short. The effect sizes in these papers mostly show up above 32K. For a one-shot classification, extraction, or summarization over a few thousand tokens, context rot is not your problem. Prompt engineering is the right tool.
When you have no eval set. Context engineering without measurement is just taste. It's no accident that all five steps above begin with "measure." Building the eval set is the first task, not changing the context structure.
When you haven't picked a model yet. As ATLAS's rank flips show, the best model differs by length band. Picking a better model for your length band may have a bigger effect than refining your context pipeline.
When the problem is actually on the generation side. This is the asymmetry the survey of over 1,400 papers (arXiv 2507.13334) points to — models are excellent at understanding complex context but show a clear limit in generating correspondingly sophisticated long-form output. If the output is the problem, no amount of careful input curation will solve it.
When multi-turn itself is the cause. "LLMs Get Lost In Multi-Turn Conversation" (Laban et al., arXiv 2505.06120) reports, from over 200,000 simulated conversations, that multi-turn performance is on average 39% lower than single-turn across six generation tasks (this 39% is a separate figure with no relation to Anthropic's 39% above). The authors' decomposition is interesting: most of the degradation was not a loss of capability but an increase in unreliability. The model jumps to conclusions on an early turn, races prematurely to a final answer, and then over-relies on it. In one line: it can't find its way back from a wrong turn. This may be a problem you solve not by filling the context better but by restarting the conversation.

Closing

The answer to the terminology question is anticlimactic. Context engineering did not replace prompt engineering — it includes it. The people who coined it wrote as much, and the sentence "it's dead" appears in no primary source.

The interesting answer comes next. Even as a superset, does it deserve a new name? — yes. The evidence is exactly one thing, and that one thing is enough: with every prompt variable held fixed, changing only the surrounding context still moves performance (Chroma's focused vs. full). That axis did not exist in prompt engineering's vocabulary, and so it needed a name.

And the evidence actually standing under that name is thinner than you'd think. What's robust is the phenomenon — as length grows, performance degrades non-uniformly; claimed length is not effective length; the middle vanishes; and all of this has reproduced across four generations from 2023 to 2026. The mechanism ("attention budget"), by contrast, is a metaphor, and the effect sizes of the prescriptions (compaction especially) mostly have no public numbers.

The guidance this map gives a practitioner is simple. Trust the phenomenon and build for it, but don't deduce prescriptions from the mechanism story, and don't borrow anyone else's prescription numbers. Measuring the single focused/full gap decides, within 30 minutes, whether this whole discussion even applies to your system. For most teams the answer is "this isn't our problem," and confirming that costs far less than building a new context pipeline.