How AI Agent Memory Is Actually Built — Four Designs, and What the Benchmarks Really Prove

• Introduction — "Memory" Is Not One Thing
• What Each of the Four Designs Does
• Where Shipped Products and Research Papers Diverge
• Which Design Is More Accurate — Reading the Most-Cited Table for Yourself
• Why Full-Context Won — I Opened the Benchmark Myself
• What Happens When Vendors' Numbers Don't Match — The Mem0-vs-Zep Record
• What About a Harder Benchmark — LongMemEval
• What Does Anthropic's 39% Measure?
• What Went Unmeasured — The Failure Modes
• So What Should You Use — A Decision Rule
• When Not to Use It
• Closing
• References

Introduction — "Memory" Is Not One Thing

"We gave our AI agent memory" tells you almost nothing. The sentence can point to at least four different designs, each with its own costs and its own failure modes. Jotting notes into a file; summarizing a conversation once it grows long and discarding the front of it; extracting facts, embedding them, and pulling them back for each question; accumulating entities and relationships into a graph — all four go by the name "memory," yet they are not the same thing. The word is overloaded enough that a January 2026 survey (arXiv:2602.06052) remarks that memory alone drew hundreds of papers in a single year.

So the first question this post answers is this: what does each design actually do, and has there ever been real measured evidence that one is better than another? The short version: what each design does is written plainly in the product docs. The public evidence for "which one is more accurate," on the other hand, is far weaker than you would expect. Read the table in the most-cited comparison paper for yourself and the top scorer turns out to be no memory system at all — it is "just paste the entire conversation into the prompt." And once you open that benchmark and count its tokens, you can see why. This post walks through that, step by step.

This post does not repeat the design patterns covered in Context Engineering. What it does instead is an audit of the evidence — which number was measured by whom, under what conditions, and against what baseline.

What Each of the Four Designs Does

Let me fix the terms first. The four below are not mutually exclusive, and real products usually mix them.

1. Scratchpad / file memory. The agent writes notes to a file and reads them back later. Anthropic's memory tool has exactly this shape — per the official docs, it creates, reads, updates, and deletes files under a /memories directory. The important part is that this is entirely client-side. In the docs' framing, Claude only requests the file operations; your application executes them against your own storage. The docs call this pattern just-in-time context retrieval — instead of loading everything up front, you write down what you learned and read it back when you need it.

2. Summarization / compaction. When a conversation grows long, the front of it gets swapped out for a summary. This is Anthropic's server-side compaction. Per the docs, once input tokens reach a threshold you configure, the API summarizes the conversation into a compaction block and, on subsequent requests, automatically drops every content block ahead of that block. The beta header is compact-2026-01-12 and the strategy name is compact_20260112. The reason the docs give is the interesting part — the point is not merely to stay under the limit, but to keep the active context small because response quality degrades as a conversation grows long.

3. Vector retrieval. Extract facts from the conversation, embed them, and pull back the similar ones for each question. Mem0's default configuration is in this family. Technically it is the same thing as the retrieval pipeline in Production RAG Patterns; the only difference is that the target is a conversation rather than a document.

4. Knowledge graph. Accumulate entities and relationships into a graph. Zep's Graphiti is the representative example; it attaches temporal information to edges to handle "from when until when was this true." How you extract the graph and what you treat as an entity is a large topic in its own right, covered separately in Actually Building a Knowledge Graph and Modeling Domain Knowledge as an Ontology. The corpus-querying side of the story is in Graph RAG.

One note on the research lineage: MemGPT (arXiv:2310.08560, October 2023), often cited as the starting point of this field, borrowed the idea from an operating system's tiered memory and proposed "virtual context management." The idea is to move data between fast and slow memory so that it looks as if you had a larger memory. The file memory and compaction in today's products sit on this lineage.

Where Shipped Products and Research Papers Diverge

Here comes the first important distinction.

What shipped products actually push is designs 1 and 2. This may be surprising. In Anthropic's docs, the context editing page guides you like this: for most use cases, server-side compaction is the primary strategy for long conversations, and context editing's finer strategies are for specific situations where you need more granular control over what gets removed. The compaction docs say the same thing — it is the recommended strategy for long-running conversations and agent workflows.

In other words, the "memory" that commercial platforms recommend by default is neither vector nor graph but summarize-and-discard plus write-to-a-file — the two simplest options.

What research and startups compete over is designs 3 and 4. Systems like Mem0, Zep, LangMem, and A-Mem live here, and all the arguments happen here too. So how solid is the evidence that this side is better?

Which Design Is More Accurate — Reading the Most-Cited Table for Yourself

The most widely cited evidence on this question is the Mem0 paper (Chhikara et al., arXiv:2504.19413, submitted April 28, 2025, ECAI 2025). The headline in the abstract goes like this — a 26% relative improvement over OpenAI on the LLM-as-a-Judge metric, roughly 2% higher than the default configuration when graph memory is used, and 91% lower p95 latency plus over 90% token-cost savings versus full-context.

But here is the paper's Table 2, copied as-is. Everything is author-measured, on the LoCoMo benchmark, and J is the LLM-as-a-Judge score.

The headline numbers check out arithmetically. I verified them myself: 66.88 divided by 52.90 is 1.264, so the 26% relative improvement over OpenAI holds; 1 minus (1.440 divided by 17.117) is 91.6%, so the 91% lower p95 versus full-context holds too. 68.44 minus 66.88 is 1.56pp, so "roughly 2% higher" holds as well. Nothing was fabricated.

The problem is the top row of that same table. Full-context wins at 72.90%. A method that just pastes the whole conversation into the prompt — one you could not even call a memory system — beat every memory system. The paper does not hide this: the body states that full-context, dropping in roughly 26,000 tokens whole, still produces the highest J score (about 73%), and it explicitly says Mem0 graph trails only the full-context approach, which is computationally hard to sustain.

So, put precisely: the 26% headline is measured against OpenAI's 52.90% — the weakest comparison in the table. The strongest baseline in the table was beaten by no memory system at all, and the abstract does not mention that fact. Nothing is false, but the framing is selective. The real value the paper sells is not accuracy but cost — that it reaches about 92% of the accuracy of the 26,031-token method with just 1,764 tokens. That is a genuinely useful result, and honestly, that should be the conclusion.

Look a little further at the table and something else stands out. LangMem's memory is 127 tokens and its search p95 is 59.82 seconds. A 60-second search is not a sane configuration in any production. It is a signal of how hard it is to run someone else's system at its optimal settings — and that signal leads straight into the next story.

Why Full-Context Won — I Opened the Benchmark Myself

To understand why full-context won, you have to look at what LoCoMo is.

LoCoMo is originally a conversational-memory benchmark (Maharana et al., arXiv:2402.17753, February 27, 2024). The dataset its Table 1 describes is this — 50 conversations, an average of 304.9 turns per conversation, an average of 19.3 sessions, and an average of 9,209.2 tokens. The questions fall into five categories: single-hop, multi-hop, temporal, commonsense/world knowledge, and adversarial.

But the Mem0 paper says it fed 26,031 tokens into full-context. 9,209 and 26,031 differ by almost a factor of three. To find out which one is right, I downloaded the public dataset and counted it myself.

The file actually sitting in the public repository (snap-research/locomo) is locomo10.json. Just as the name says, it holds 10 conversations — not the 50 the paper describes. Counting only the conversation body (utterance text and image captions) with tiktoken's cl100k_base gives this:

locomo10.json  (measured directly, tiktoken cl100k_base, utterance text + BLIP captions)
  conversations:       10           (paper Table 1 describes 50)
  tokens per convo:    min 12,218 / max 23,406 / mean 20,034 / median 21,387
                                    (paper Table 1 says mean 9,209.2)
  sessions per convo:  mean 27.2 (min 19, max 32)   (paper says mean 19.3)
  turns per convo:     mean 588.2                    (paper says mean 304.9)

  total questions:     1,986
    cat.1 single-hop      282
    cat.2 multi-hop       321
    cat.3 temporal         96
    cat.4 open-domain     841
    cat.5 adversarial     446   <- 444 of 446 have no `answer` field
                                   (only an `adversarial_answer` field)

So the public file everyone benchmarks against is roughly twice as long as the dataset the paper describes. My measured 20,034 tokens lines up with Zep's claimed range of "16,000–26,000 tokens" and points the same direction as Mem0's 26,031 tokens (Mem0's side most likely included more scaffolding, such as timestamps and observation fields). It does not line up with the abstract's "average 9K tokens."

This is why full-context won. When a single conversation is 20K tokens, it drops into a modern model's context window without any strain. This is not a test that pressures long-term memory; it is a test you pass by just dumping everything in. Zep's point on its own blog is exactly right — if feeding in everything beats a purpose-built memory system, then the benchmark is not measuring the memory pressure a real agent faces.

Category 5 is worth a look too. 444 of its 446 questions have no answer field at all (they carry an adversarial_answer instead). So both Mem0 and Zep excluded this category from evaluation. That means 22.5% of all questions are effectively unusable — and this exclusion is the cause of the accident in the next section.

What Happens When Vendors' Numbers Don't Match — The Mem0-vs-Zep Record

The event that best shows the gap between "what was measured" and "what was claimed" in this field is preserved publicly, from May 2025. In chronological order:

April 28, 2025. The Mem0 paper goes up. It reports Zep at 65.99% and claims SOTA.

May 6, 2025. Zep rebuts on its blog (Daniel Chalef, Preston Rasmussen). The gist: Mem0 implemented Zep wrong and then measured it. It cites three specifics — it assigned the user role to both conversation participants, making a single user's identity appear to change with every message; it appended timestamps to the message body instead of Zep's created_at field; and it ran retrieval sequentially rather than in parallel, inflating the latency. On that basis it claims Zep is at 84% when implemented correctly.

May 9, 2025. Mem0's CTO Deshraj opens an issue on Zep's repository (getzep/zep-papers issue #5). The core of the rebuttal is arithmetic — Zep's 84%, he says, counts the answers from category 5 (the one everyone agreed to exclude) in the numerator while dropping them from the denominator. Correct for that and it is not 84% but 58.44%. The claim is that roughly 25.56pp was inflated.

May 12, 2025. Zep's Chalef concedes the calculation error. But the corrected value was not 58.44% but 75.14% ± 0.17 (average over 10 runs). And he did not retract the rest of the critique — the claim that Mem0's experimental setup and its very choice of LoCoMo were wrong. Zep also attached a correction notice to the blog post, and the wording is still there today.

May 19, 2025. The issue is closed "due to inactivity."

The important thing here is that the two camps' numbers never once matched. Mem0 says Zep is 58.44%; Zep, even after the correction, says 75.14%. Same benchmark, same system, a 17pp gap. This post does not adjudicate which is right — because the evidence to adjudicate it is not public.

Whether the arithmetic is even coherent can be checked. By my count, categories 1–4 are 1,540 questions and category 5 is 446. Put category 5's answers in the numerator while keeping the denominator at 1,540, and inflating by 25.56pp requires about 394 of category 5's questions (88% of 446) to be scored correct. The mechanism and the magnitude Mem0 claimed do not contradict each other. But this is a check of the claim's internal consistency, not proof that either score is right.

The lesson a practitioner should take from this episode is not "who is the villain." It is that running a competitor's product at its optimal settings and measuring it is extremely hard even in good faith, so a number a vendor reports about someone else's product is weak evidence by default. LangMem's 60-second search from earlier is a signal in the same category. Zep's 65.99%, and the 75.14% Zep measured with its own hands, each carry their own stake.

What About a Harder Benchmark — LongMemEval

If LoCoMo is too easy, there is a harder one: LongMemEval (Wu et al., arXiv:2410.10813, October 2024 v1, March 2025 v2, ICLR 2025).

The design is decidedly more aggressive. There are 500 questions and two standard settings — LongMemEval_S at about 115k tokens per problem, and LongMemEval_M at about 1.5 million tokens across 500 sessions. That is a different order of magnitude from LoCoMo's 20K tokens. And it explicitly tests abilities LoCoMo lacked: information extraction, multi-session reasoning, temporal reasoning, knowledge updates (situations where a fact changes), and abstention (saying you do not know when you do not).

Here is what the paper reports. The abstract frames it as a 30% accuracy drop for commercial chat assistants and long-context LLMs, but the body writes it more precisely as a 30%–60% performance drop. And the sentence about commercial systems carries a caveat you must preserve — in manual evaluation, the latest commercial systems reached only 30%–70% accuracy, and that was in a setting much simpler than LongMemEval_S. In other words, these are not commercial-system scores at the real difficulty.

Zep, in its own paper (arXiv:2501.13956, January 2025), states that it prefers LongMemEval and reports up to 18.5% accuracy improvement and 90% latency reduction. The qualifier "up to" is attached, and it is author-measured. The same paper reports 94.8% versus 93.4% against MemGPT on the DMR benchmark, and calling that SOTA on the basis of a 1.4pp gap is thin on its own.

Here is something I have to say honestly. An independent, third-party comparison that runs Mem0, Zep, file memory, and compaction side by side under identical conditions on LongMemEval is not public, as far as I could confirm. There are only numbers each vendor measured in its own pipeline with its own prompts. Without a common benchmark you cannot line up vendor A's self-measurement against vendor B's and compare them — which is why this post does not build such a table.

What Does Anthropic's 39% Measure?

To be fair, the commercial-side numbers deserve the same scrutiny.

The figures Anthropic published when it announced context editing and the memory tool on September 29, 2025, are these — using the memory tool and context editing together gives a 39% performance gain over baseline, and context editing alone gives a 29% gain. And in a 100-turn web-search evaluation, it says context editing carried to completion a workflow that would otherwise have failed from context exhaustion, while cutting token consumption by 84%.

The conditions attached to these numbers, transcribed as-is: First, they are measured on an internal evaluation set for agentic search. It is vendor-measured and cannot be reproduced externally. Second, the baseline's absolute score is not published. You cannot tell whether the 39% gain is 40 to 55.6 points or 70 to 97 points. Third, the three numbers each measure different things — 39% and 29% are accuracy-family, while 84% is token consumption.

And one more thing — a mismatch that opened up over time. That 39% is a September 2025 measurement of the context editing + memory tool combination. But what Anthropic's docs now recommend as the primary strategy for long-running conversations is server-side compaction, a later feature whose beta header is compact-2026-01-12. In other words, the widely cited 39% is not a measurement of the currently recommended default. This does not mean Anthropic did anything wrong; it means the number ages more slowly than the product moves. When you cite a vendor figure, you have to write down which configuration, at which point in time, it measured.

What Went Unmeasured — The Failure Modes

There are failures that never appear in an accuracy table. These mostly exist not as benchmarks but only as warning notes in the docs.

Memory poisoning and prompt injection. If an agent writes what it read into memory and later trusts and reads that memory back, a single poisoned input persists across sessions. Anthropic's docs explicitly warn about path-traversal attacks for the memory tool — a path like /memories/../../secrets.env can reach a file outside the memory directory, so you have to validate every path in every command. Because your application is what executes, the defense is yours too.

Writing down sensitive information. In that same doc's words, Claude usually refuses to write sensitive information into memory files. If you want a stronger guarantee, it advises you to add your own validation that filters sensitive data before the handler writes the file. You must not upgrade the qualifier "usually" into a "does not."

Knowledge updates. When a user changes jobs, the old employer information becomes wrong. LoCoMo has no question that tests this at all (Zep's point, consistent with the category distribution I confirmed). LongMemEval names knowledge updates as one of its five abilities — which is why it is a better test.

Unbounded growth and expiry. Anthropic's docs recommend setting a file-size cap and periodically deleting memory files that have not been accessed in a long time. That means you need a policy for "what to forget" — and there is no public evidence that anything solves this part well automatically.

There is essentially no public quantitative comparison for these failures. So when you choose a memory system, looking only at the accuracy table may be looking at the wrong axis.

So What Should You Use — A Decision Rule

Given the state of the evidence, here is a decision rule. I will not end on "it depends."

1. If the conversation or task fits in the context window, do not add a memory system. Just put it all in. The Mem0 paper's own table supports this conclusion — on the 20K-token LoCoMo, full-context beat every memory system at 72.90%. This is a weakness of the benchmark, but it is also a conclusion that applies directly if your situation resembles that benchmark.

2. If it does not fit, start with compaction and files. The fact that these two are what commercial platforms recommend by default is itself a signal. Server-side compaction works without any client-side summarization code, and with file memory you control the storage. Both have a small implementation surface and can be debugged when they fail (you just open the file). Vector and graph are the tools you reach for after you have confirmed a failure here.

3. If cross-session fact retrieval turns out to be a real failure point, vector. What it sells here is not accuracy but cost. Read Mem0's table honestly and the system's value lies in reaching about 92% of the accuracy of the 26,031-token method with 1,764 tokens. If that is the trade-off you need, it is a good choice; if not, it is not.

4. If you must handle relationships that change over time, graph. Questions like "from when until when was this true" and "which team is this person on now." But the indexing cost and update burden laid out in Graph RAG come along unchanged.

5. Whatever you choose, evaluate on your own queries. This is not a platitude but a conclusion that follows directly from what this post confirmed. One of the public benchmarks has a mismatch between the paper's statistics and the public file; on top of it two vendors argued over 17pp and never reached agreement; and the commercial figures are irreproducible internal evaluations. Choosing an architecture by trusting someone else's leaderboard in this state is a choice without evidence. A self-eval set of 20 to 50 questions is more useful to you than every number in this post.

When Not to Use It

The part to write honestly.

• The conversation is short. Most chatbot sessions are a few thousand tokens. There is no reason to bolt on memory infrastructure.
• A wrong memory costs more than a right one. In domains like healthcare, law, and finance, where mistaken personalization does damage, not remembering is the safer choice. As seen above, there is no quantitative evidence on poisoning and update failures.
• You handle regulated data. Memory, by definition, persists personal data. If you have not designed for the right to erasure and a retention policy, do not add it.
• You have not confirmed a failure yet. Adding memory "because I will probably need it later" is a choice the evidence in this post does not support.

Closing

To sum up: "agent memory" is a blur of at least four designs, and what each does is written clearly in the product docs — file scratchpad, compaction, vector retrieval, knowledge graph. What commercial platforms recommend by default is the first two, the two simplest.

The public evidence for "which one is more accurate," by contrast, is weak. In the most-cited comparison's table the winner was not a memory system but full-context (72.90% versus 68.44%), and opening that benchmark myself showed that a single conversation averages 20K tokens, so it never pressured long-term memory in the first place. On top of that, two vendors rebutted each other's measurements, and even after one conceded a calculation error the two numbers never met. A harder benchmark exists, but an independent third-party comparison on it is not public.

This does not mean the field is fake. The problem of managing context is real, and compaction and file memory have actually shipped and are running. It is only that the sentence "adding memory makes things better" does not yet have a public measurement attached that supports it. So start from your failures, not from a leaderboard. Confirm what is not working first, add the simplest thing first, and measure with your own queries. Failure comes first, not the tool.

References

• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., arXiv 2504.19413) — the J scores and latency figures in Table 2
• Evaluating Very Long-Term Conversational Memory of LLM Agents (Maharana et al., arXiv 2402.17753) — the original LoCoMo paper
• snap-research/locomo — the public LoCoMo dataset — locomo10.json, the token-measurement target of this post
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (Wu et al., arXiv 2410.10813)
• Zep: A Temporal Knowledge Graph Architecture for Agent Memory (Rasmussen et al., arXiv 2501.13956)
• MemGPT: Towards LLMs as Operating Systems (Packer et al., arXiv 2310.08560)
• Lies, Damn Lies, & Statistics: Is Mem0 Really SOTA in Agent Memory? (Zep blog) — includes the correction notice
• getzep/zep-papers issue #5 — Mem0's rebuttal and Zep's correction
• Managing context on the Claude Developer Platform (Anthropic, 2025-09-29) — the source of the 39% / 29% / 84% figures
• Anthropic docs — Memory tool
• Anthropic docs — Compaction
• Anthropic docs — Context editing
• Rethinking Memory Mechanisms of Foundation Agents in the Second Half: A Survey (arXiv 2602.06052)
• Context Engineering — How to Design Memory for AI Agents (related post)
• What Graph RAG Is (related post)