For about two years, "agent memory" has meant one thing: an LLM sits between your conversation and your database and decides what's worth keeping. Mem0 extracts facts. Zep folds them into a temporal knowledge graph. Letta runs an OS-style scheduler that pages summaries in and out of context. The shared premise is that raw history is too big and too noisy to keep, so the intelligent move is to compress it well — to have a model read what happened and write down the gist.
The system currently sitting at the top of LongMemEval, the standard long-term memory benchmark, does none of that. MemPalace stores your conversation history as verbatim text, indexes it, and retrieves it with plain semantic search. No extraction. No summarization. No LLM at write time or read time. Its raw retrieval score is 96.6% Recall@5 with, in the maintainers' words, "no API key, no cloud, and no LLM at any stage." A held-out hybrid configuration reaches 98.4%, and adding an optional LLM reranker pushes it past 99%. The repository is MIT-licensed, mostly Python, and has picked up a five-figure star count and a sprawling ecosystem of ports — Node, TypeScript, Ruby, even a Zig rewrite — in a few months.
The obvious read is "local-first tool beats the SaaS incumbents." The more interesting read is that MemPalace is a falsification of the field's core assumption.
The extraction bet, stated plainly#
Here is what the extraction paradigm actually claims: the bottleneck in agent memory is deciding what to remember. If a model reads the transcript and pulls out "the user prefers Postgres, ships on Fridays, and hates em-dashes," you've turned a 40,000-token history into a handful of durable facts. Retrieval becomes trivial because there's almost nothing left to retrieve. Mem0's 2026 report leans into exactly this framing, touting a token-efficient algorithm whose biggest gains are on temporal queries (+29.6 points) and multi-hop reasoning (+23.1) — i.e., the cases where good extraction should pay off most.
But extraction has a cost that rarely shows up in the marketing. Every write is an LLM call: latency, dollars, and a fresh opportunity to hallucinate a "fact" that was never said. Worse, it's lossy in a direction you can't predict at write time. The model has to guess, during the conversation, which details a future question will hinge on. LongMemEval is built to punish that guess. Its questions span single-session fact retrieval, multi-session aggregation, knowledge-update tracking, and temporal reasoning — plus abstention cases where the honest answer is "that never came up." When the discriminating detail is a throwaway line the summarizer decided wasn't worth keeping, the fact was gone before retrieval ever ran.
Extraction moves the hard decision to write time, when you know the least about what you'll be asked.
MemPalace's answer is to not decide. Keep everything; get very good at finding it later. Its "palace" metaphor — people and projects become wings, topics become rooms, original text lives in drawers — is really just a structured index over verbatim storage, backed by ChromaDB, SQLite, Qdrant, or pgvector depending on how you deploy it. The intelligence isn't in a model that curates. It's in the indexing and the retrieval.
The catch worth being honest about#
This is where a lot of the breathless coverage stops, and where it shouldn't. That 96.6% is Recall@5 — a retrieval metric, not end-to-end answer accuracy. It says the right passage is in the top five results, not that the agent got the question right. (If you're ever unsure which number a memory vendor is quoting, that distinction is the first thing to check in any memory benchmark.) LongMemEval deliberately measures both stages, and the gap between them is the whole game: on the QA side, even strong commercial systems land in the 30–70% range, and long-context models drop 30–60% as histories grow.
Verbatim storage doesn't erase that gap — it relocates it. You've saved yourself a write-time LLM by agreeing to pour raw drawers into the model's context at read time. The reasoning still has to happen; you've just moved it downstream, where it shows up as bigger prompts, higher token bills, and the same long-context degradation everyone else fights. A critical analysis on arXiv makes the sharper version of this point: the spatial-palace framing is a UX metaphor, not a retrieval algorithm, and the scores come from the semantic search underneath, not the rooms-and-drawers story wrapped around it. That's fair. It's also the point. Strip the metaphor and MemPalace's real claim is unglamorous: agent memory is an information-retrieval problem, and the LLM-in-the-write-path was a solution to a problem retrieval already had answers for.
What this changes for anyone building#
Don't rip out Mem0, Zep, or Letta because a local tool posted a good recall number. Do notice what the number is telling you. If your agent keeps failing on "you told me this three sessions ago" questions, the fix probably isn't a smarter extractor — it's not throwing the sentence away in the first place. Storage is cheap; the detail you didn't keep is the one the user will ask about.
The verbatim camp isn't obviously right for every workload — high-volume, privacy-constrained, or strictly token-budgeted agents may genuinely need to forget. But it has done something useful for a field that got comfortable: it made "we compress history with an LLM because obviously you have to" say its premise out loud, at which point the premise stopped looking obvious. The next benchmark cycle won't be extraction versus verbatim. It'll be about where you're willing to pay — write time or read time — and MemPalace's contribution is forcing everyone to answer that on purpose instead of by default.
