Most memory benchmarks answer one question: can the model find the needle? LongMemEval-V2 — the 2026 successor to the ICLR 2025 benchmark that set the terms for this whole conversation — changes the question. It asks whether your memory system can turn a pile of an agent's past runs into the kind of experience a good colleague has. And then it does something the older benchmarks did not: it times you.
If you remember one thing, make it this: V2 scores accuracy and latency together, so "retrieve more context" is no longer a free way to win.
From transcripts to trajectories#
The original LongMemEval tested a chat assistant. You gave it a long conversation, buried a fact somewhere in it, and asked whether the model could still recall that fact thousands of turns later. That was the right test for 2025, when the thing we called "memory" was mostly a long chat history you had to search.
V2 replaces the transcript with the agent's own work. Its haystacks are long histories of multimodal web-agent trajectories — the actual step-by-step records of an agent doing tasks across web and enterprise environments — with 451 manually curated questions posed over them. The scale is the tell: up to 500 trajectories per haystack and up to 115 million tokens in the largest ones. No context window holds that. The benchmark is, by construction, about what your memory layer keeps and retrieves, because the raw history cannot be kept in the prompt.
The five things memory is actually for#
The reason this matters is hidden in what V2 chooses to grade. It scores five abilities: static state recall (a fact that never changes), dynamic state tracking (a fact that changes over time), workflow knowledge (how a task is really done in this environment), environment gotchas (the quirks that bite you), and premise awareness (whether a question is even built on something that was established).
Read that list and notice that only the first is "look up a fact." The other four are experience — the difference between a new hire who can search the wiki and a colleague who knows how things actually work here. A plain vector store retrieving semantically similar chunks is good at the first ability and structurally bad at the rest, because dynamic state and workflow are relationships across time, not similar-looking passages. This is the same wall we hit when we asked where an agent's long-term memory should live: the hard part was never storage, it was deciding what to keep and how to reconstruct state.
The part everyone will underrate: it times you#
Here is the design decision that reorders the leaderboard. V2 grades answer accuracy and query latency together, and the public board ranks systems by a single LAFS — Latency-Accuracy Frontier Score, measured as gain over reference baselines rather than accuracy in isolation.
Once latency is on the axis, the cheapest trick in memory benchmarking — retrieve a bigger slice and let a strong model sort it out — turns into a penalty instead of a strategy.
For two years the honest way to inflate a memory score was to widen retrieval: pull more history, stuff more into the window, let the model find the answer in the pile. It works, and it is exactly what you cannot afford in production, where every extra retrieved token is latency and cost on the critical path. By putting latency on the frontier, V2 makes the benchmark reward the thing you actually deploy for — a correct answer from a small, well-chosen slice — and punishes the thing that looks great in a paper and dies under load. It is the benchmark finally agreeing with the token-cost math that founders already live with.
What to do with it#
Three concrete moves.
First, stop trusting single-number memory claims. A vendor's "92% on the memory benchmark" is a V1-shaped statement. Ask where they sit on V2's latency-accuracy frontier and on which tier — the board has small and medium tiers precisely so you can compare without processing the 115M-token monsters every run.
Second, run the baselines yourself. The repo ships six baseline memory modules — no-retrieval, RAG query-to-slice variants, and AgentRunbook — with reproduction scripts. Point them at your own agent's trajectories before you believe any leaderboard; the fastest way to understand a benchmark is to make its baselines produce numbers on your data. If you have never done this, our walkthrough on how to read an agent-memory benchmark and the LoCoMo vs LongMemEval vs BEAM comparison are the two priors worth holding first.
Third, rewrite your memory layer's spec. Its job is not "recall a fact from a transcript." It is: reconstruct state, workflow, and gotchas from an agent's own history, and do it in few enough retrieved tokens that the answer is fast. LongMemEval-V2 is the first public benchmark that grades all of that at once. That is why it is worth paying attention to even if you never submit to the leaderboard — it is a clearer statement of the spec than most teams have written for themselves.
