The short version: "agent memory" is three different jobs, and most agents that feel forgetful are using the wrong one. Working memory is what's in the context window right now. Session memory is state that survives within one task or thread. Long-term memory is what persists across sessions — the user's preferences, facts the agent learned last week, episodes it can recall. Get the tier wrong and you either pay for a database you didn't need, or you widen a context window hoping it will remember, and it can't.
If you build alone, this is a spending decision as much as an engineering one. Each tier has a different cost curve and a different failure mode. (The framing isn't ours alone — the viral Google agentic-engineering course devotes an early segment to "agent memory: short, persistent, long," which is exactly the split worth getting right.) Here's how to tell the three apart and pick.
The canonical map (so we're honest about terms)#
The most-cited academic frame is CoALA — Cognitive Architectures for Language Agents (2023). It's a two-way split: working memory versus long-term memory, with long-term further divided into episodic (specific past events), semantic (general facts), and procedural (skills and routines). LangGraph, LangMem, and Letta all echo it.
The "short / persistent / long" framing in this piece is an engineering lens on top of CoALA, not a competing taxonomy — I'm splitting out the persistent-within-a-task case because in practice that's where the most decisions get made wrong. Treat the three tiers as a checklist, not gospel.
1. Working memory — and why a big window isn't memory#
Working memory is everything inside the model's context on this call: the system prompt, the conversation so far, retrieved chunks, tool outputs. It's fast, it's simple, and it disappears the moment the call ends.
The trap is treating a large context window as memory. Windows are big now — Claude around 200K tokens, Gemini up to ~2M — so it's tempting to just keep appending. But a big window buys you more working memory in one call; it buys you zero persistence. It also degrades as you fill it: cost and latency scale with tokens, and models recall the middle of a long context worse than the ends — the "lost in the middle" effect. Anthropic's own guidance is telling: pair context editing (auto-clear stale tool calls) with a memory tool (durable notes on disk) — i.e. keep active context small, push durable knowledge out of it.
Context is where memory gets used, not where it lives. If the answer to "will the agent remember this tomorrow?" is "as long as we resend it," that isn't memory — it's a bill.
Use working memory when the relevant history fits and is cheap to resend. It's the wrong tier when the user would be annoyed to repeat something next session. That's your signal to go persistent.
2. Session / persistent memory — the tier founders skip#
This is state that must survive within a task but not necessarily forever: a running scratchpad across a multi-step agent loop, the checkpoint of a long job so a crash or a pause doesn't restart it, the conversation state of an in-progress thread.
In LangGraph terms this is a checkpointer — thread-scoped state, distinct from the cross-thread store that holds long-term memory. The reason this tier gets skipped is that it looks like working memory until something interrupts the run; then you discover you were reconstructing state by replaying the entire history on every step, which is slow and expensive and eventually overflows the window.
Use it when a run has more than a couple of steps, can be paused, or can fail partway. It's the wrong tier when you reach for it to remember things between users or between days — that's long-term's job, below.
3. Long-term memory — and the retrieval-vs-graph fork#
Long-term memory persists across sessions. This is where you personalize ("this user prefers terse answers"), where you accumulate ("last month we decided X"), and where the interesting tool choices live. There are two shapes, and the fork matters:
- Retrieval-backed (vector) memory — Mem0, a vector DB, or file-based notes like Claude's memory tool (GA, client-side,
memory_20250818, Claude 4+). Use when knowledge exceeds the window or must persist, and similarity search over unstructured facts is good enough. This is the default; start here. - Knowledge-graph memory — Graphiti / Zep or the MCP memory server. Use only when relationships and time matter: multi-hop reasoning across connected facts, or "what was true when." Graphiti, for instance, closes a fact's validity window when the fact changes instead of overwriting it — so you can still ask about the past. Graphs cost more to build and maintain; don't start there.
If you'd rather the agent manage its own tiers OS-style — paging between an in-context "core" and external storage — that's Letta (formerly MemGPT). Powerful, but you're adopting its model of the world.
For the deeper tool bake-off, see our Mem0 vs Zep vs Letta comparison, and for keeping a long-running agent inside its window, Context Editing vs Compaction vs the Memory Tool.
Measuring it (read the benchmarks skeptically)#
If you want numbers, the two to know are LoCoMo (ACL 2024 — recall and reasoning over very long multi-session dialogues) and LongMemEval (ICLR 2025 — five abilities including temporal reasoning and knowledge updates, which found commercial assistants drop ~30% on sustained-interaction memory). One warning: vendor-reported scores don't agree, and a lab quoting its own LoCoMo/LongMemEval figures is making a claim, not reporting an independent result. We wrote a whole guide on how to read an agent-memory benchmark precisely because the number wars are misleading.
The decision, in one line#
Start in working memory. Add session memory the moment a run has steps that can fail. Add a persistent store the moment a user should not have to repeat themselves — vector first, graph only if facts change over time and you need to reason about when. The failure mode to avoid isn't forgetting; it's building all three when you needed one.
