Two of the more careful papers on agent memory published in the last year point in opposite directions, and the useful thing about them is that neither is wrong. One is the strongest case anyone has made for buying a memory layer. The other is the strongest case against. Read together, they tell you the one number that should drive the decision — and it is not on any vendor's slide.
Here is the whole finding, up front, because it is the article: a memory layer wins cost and latency by roughly an order of magnitude, and loses raw accuracy by roughly 35 points. Everything below is just the sourcing and the "so which one do I pick."
The case for the layer: an order of magnitude on cost and speed#
The Mem0 paper (ECAI 2025) is the cleanest version of the pro-memory argument, and its numbers are concrete. Against a full-context baseline on the LoCoMo benchmark — the setup where you just paste the entire prior conversation into the window — a memory layer holds a working footprint of about 7,000 tokens, under 10% of the full context. That compression drops p95 latency from 17.12 seconds to 1.44 seconds, and cuts total token usage by roughly 90%.
None of that is subtle. If your agent re-reads a long history on every turn, you are paying for those tokens every turn and waiting on them every turn, and a memory layer makes both bills roughly ten times smaller. For a product with long-lived conversations — a support agent, a coding assistant that remembers your repo, anything with returning users — that is not a rounding error. It is the difference between a viable unit economics and a demo.
A memory layer is not primarily an accuracy technology. It is a cost-and-latency technology that happens to keep accuracy in the same neighborhood — most of the time.
The case against: 35 points of accuracy, gone in the compression#
Now the other paper. "Beyond the Context Window" runs the comparison the vendor charts avoid: it pits fact- and memory-based systems against a plain long-context baseline on the same benchmarks, and scores accuracy, not tokens. The result is blunt. The long-context baseline beats the memory systems by 35.2 points on LoCoMo and 33.4 points on LongMemEval.
The mechanism is exactly what you would fear. A memory pipeline compresses a conversation of roughly 105,000 tokens down to something like 1,000–3,000 retrieved tokens before the model ever sees it. That is a 30-to-100x reduction, and it is wonderful for your token bill — but the tokens it throws away are not random. Multi-hop questions, the ones whose answer is stitched from two facts mentioned an hour apart, are precisely the questions whose supporting detail gets dropped in compression. Full context keeps everything, so it keeps the two facts, so it answers. The memory layer kept a summary of one of them and guesses.
This is the number nobody puts on the slide, and it is the one that should drive your decision: the layer is not free. Its price is paid in accuracy on the hard questions, and on these benchmarks that price is about 35 points.
Why "just use a bigger window" is not the escape hatch#
The obvious reaction is to skip the whole argument and buy your way out with a giant context window — never compress, never retrieve, just pass everything. It works right up until it doesn't, and the reason is context rot. Chroma's 2025 study tested 18 frontier models and found all 18 degrade as input length grows — not off a cliff at some documented limit, but gradually and non-uniformly, even on trivial retrieval tasks. So full context wins accuracy while the history is a comfortable size, and then quietly stops winning as the window fills, at the same time its cost and latency climb. The window is a budget you spend, not a problem you solve. Anthropic's own context-engineering guidance frames the window as a finite "attention budget" for exactly this reason.
That is why the two papers don't actually contradict each other. Full context is the accuracy leader in the regime where the history fits and stays sharp. The memory layer is the cost-and-latency leader in the regime where it doesn't. The decision is just: which regime are you in?
The decision, reduced to three questions#
Skip the leaderboard cells — they are mostly vendor-run and mutually inconsistent, to the point where one product's own tool scores 94 on a benchmark a rival's harness scores it 49 on. Trust the direction both independent analyses agree on, and ask three things:
- Does the history still fit, sharp, in the window? If a full conversation is comfortably inside your model's effective range — not its maximum, its effective range before context rot bites — then full context is your accuracy baseline and you need a real reason to give up 35 points to beat it. Usually you don't.
- Are the questions multi-hop or detail-sensitive? If answers depend on stitching specifics that a summary would drop, compression is where your accuracy goes to die. Lean full context, or a retrieval layer tuned for recall over token savings.
- Is cost or latency the actual constraint? If you are re-reading a 100K-token history every turn and the token bill or the 17-second wait is what's killing the product, the layer's order-of-magnitude win is real and worth the accuracy hit — and you'll want to think hard about where that memory physically lives and how you keep the live window clear while it does.
The honest default for most builders is the boring one: start with full context, measure, and add a memory layer only when history outgrows the window or the cost line turns red — which is the same duration-first way of sorting memory that keeps you from buying machinery the task never needed. The 35-point gap is not a reason to avoid memory layers. It is a reason to know exactly what you are trading for the ten-times cost cut — and to make that trade on purpose, not because a chart with the axis conveniently missing told you the layer was strictly better. It isn't. It's a trade.
