The agent-memory market sells itself in a single number, and that number is a LoCoMo score. Open any vendor's benchmark post from the last year and you will find single-hop accuracy in the mid-90s, a leaderboard that creeps up a point per release, and the strong implication that memory is a solved layer you can now buy. It is a tidy story. It is also, on close reading, an answer to a question almost nobody deploying an agent is asking.
What LoCoMo actually measures#
LoCoMo — Long-term Conversational Memory, from Maharana et al. at ACL 2024 — is a set of 50 synthetic dialogues, each running 19 to 35 sessions and hundreds of turns, paired with questions that ask whether a model can recall and reason over something stated earlier in the transcript. That is a genuine and hard problem. But notice its shape: the memory system is handed a fixed conversation and then quizzed about it. Nothing the system remembers changes what happens next, because nothing happens next. It is a reading-comprehension exam with a very long passage.
This matters because it is not how a working agent uses memory. A deployed agent writes to memory while acting — it books the wrong flight, gets an error, and has to remember the constraint it just learned — and then reads that memory to choose its next move. Recall of a planted fact and use of self-authored memory under changing conditions are different skills. LoCoMo only tests the first.
The leaderboard is also just wrong#
Before we even get to that gap, the numbers on top of LoCoMo do not survive an audit. Penfield Labs went through the answer key and found 99 of 1,540 questions have gold answers that are wrong or unanswerable — a 6.4% error rate, which puts the true ceiling at roughly 93.6%. Several published scores sit above that ceiling, which is only possible by getting corrupted questions "right" in the corrupted direction.
Worse is the judge. Most LoCoMo harnesses grade with an LLM, and the prompts diverge: the original paper uses token-overlap F1, while some vendor harnesses instruct the judge to "be generous with grading." In Penfield's stress test, the standard GPT-4o-mini judge accepted 62.81% of answers that were deliberately wrong but on-topic. A benchmark whose grader rubber-stamps two-thirds of plausible nonsense is not measuring what its leaderboard claims, and cross-vendor comparisons on it — where each group runs its own judge — are close to noise.
A LoCoMo score tells you a system can retrieve a fact from a transcript. It tells you almost nothing about whether memory will help your agent take the next action — and for a working agent, that second thing is the only thing.
MemoryArena changes the exam#
The more interesting result this year is not another point on LoCoMo — it is what happens when you change the exam entirely. MemoryArena (He et al., arXiv 2602.16313, February 2026, with authors from the UCSD and MIT orbit including Julian McAuley, Yejin Choi, and Alex Pentland) runs memory inside what it calls Memory-Agent-Environment loops, across four domains: web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning. The defining feature is interdependent subtasks — the agent must distill earlier actions and feedback into memory, then rely on that memory to solve later steps. Memory is a component of acting, not an oracle to be quizzed.
Run the systems that near-saturate LoCoMo through MemoryArena and they fall to roughly 40–60%. That is not a modest regression; it is a forty-point cliff. And it is the single non-obvious fact in this whole debate: the correlation between "great at LoCoMo" and "great at agentic memory" is weak enough that the first number does not predict the second. The leaderboard everyone optimizes against is measuring a skill orthogonal to the one they are shipping.
Why the gap is structural, not fixable with a bigger model#
You might expect a stronger base model to close the gap. It helps at the margin, but the failure mode MemoryArena exposes is not raw comprehension — it is that acting corrupts memory in ways passive recall never surfaces. When an agent writes its own notes, it writes ambiguous ones, stale ones, and contradictory ones, and then it has to act on them. On a static transcript those pathologies never appear, because the "memory" is a clean human-written text the system only reads. This is why the field is building tests LoCoMo can't be patched into: BEAM was designed so that no current memory architecture saturates it, precisely because a saturable benchmark stops discriminating between systems the moment they all ace it.
What to actually do#
If you are choosing a memory system, the practical takeaway is short. Do not buy on a LoCoMo number, and be especially wary of a number graded by the vendor's own generous judge. Evaluate memory in the loop: assemble a handful of your own multi-step tasks where a later step genuinely depends on something the agent had to remember from an earlier one, fix one grading rubric, and measure success at the task, not recall of a fact. When you read anyone's benchmark, the first question is not "what did it score" but "did memory have to feed an action, or just answer a question." Agent memory is not solved. It looked solved because we were all grading the wrong exam.
