The short version: an agent-memory system that works in the demo and rots in production is failing in one of exactly four ways — unbounded growth, stale retrieval, no forgetting, or poisoning — and each has a specific fix. Wiring the three memory layers (the short-term window, the episodic store, and the long-term profile) is the easy part. Keeping them healthy over weeks of real traffic is where agents fall over. Here's how to diagnose each failure and the fix that actually holds.
1. Unbounded growth — the agent gets slower and dumber over time#
The symptom: cost and latency climb week over week, and accuracy quietly drops. The instinct that causes it is treating memory as a bucket you pour every turn into.
The root cause: storing everything adds noise. Past a point, the extra context degrades retrieval rather than helping it — the relevant memory gets buried under near-duplicate junk, and a long-running agent destabilizes as its recall gets noisier. It's also just expensive: every persisted turn is tokens you re-read on future queries.
The fix — bound every layer:
- Trim or summarize the context window each turn so short-term memory never overflows (LangChain). Summarize-then-delete the originals so they stop costing tokens.
- Cap what you persist. Not every turn earns a durable memory. Write the salient fact, drop the chatter.
- Extract facts, don't dump transcripts. A distilled fact is a fraction of the tokens of the exchange that produced it — and it retrieves cleaner.
- TTL the long tail. Episodic entries that haven't been retrieved in N days are usually noise.
We put hard numbers on how fast this cost compounds in how many tokens an agent memory layer uses — from 7K to 3.26M per query.
2. Stale / irrelevant retrieval — the right memory, for the wrong moment#
The symptom: the agent confidently recalls something true but irrelevant — you ask about one Python service and it drags in a memory about an unrelated Python pipeline.
The root cause: vector search ranks by semantic similarity, not conversational relevance. Two things can be embedding-close and situationally unrelated. And the naive fix — "just retrieve the most recent" — is wrong in the other direction, because sometimes you want the relevant memory, not the latest.
The fix — rank on more than similarity:
- Hybrid retrieval: combine dense vector search with keyword/BM25 so exact-term matches aren't lost to fuzzy semantics.
- Re-rank on a composite score that weights recency and relevance together. The Generative Agents paper's score — recency × importance × relevance — is still the template a decade of memory systems copy (arXiv 2304.03442).
- Filter by a recency window before scoring to shrink the candidate set to memories that could plausibly matter now.
Retrieval quality, not storage, is where memory systems live or die. You can store perfectly and still recall garbage if you rank on similarity alone.
3. No forgetting — stale assumptions leak into new tasks#
The symptom: the agent brings last week's context into today's unrelated task, or "helpfully" surfaces an old preference that no longer holds. Every serious agent-memory system is really a forgetting system, and this is what happens when you skip that half.
The root cause: you built the write path and never built the delete path. Memory only accumulates.
The fix — design forgetting first, on three fronts:
- TTL on long-tail entries to bound storage.
- Recency decay on retrieval scores (exponential or LRU-style) to bound interference, so old memories fade out of results instead of competing with fresh ones.
- Active supersession on write: when a new fact contradicts an old one, overwrite it — UPSERT into a keyed profile — rather than letting both sit in the store as a contradiction the agent will later trip over.
Deciding which of two conflicting memories wins is its own design call; we walked the deterministic-vs-LLM options in agent memory conflict resolution, and the consolidation mechanics across mem0, Zep, and the memory tool in how AI agents decide what to forget.
4. Memory poisoning — a false fact that fires every session#
The symptom: the agent states something false with total confidence, session after session, and clearing the context window doesn't fix it.
The root cause: the payload is in the store, not the window. Two ways it gets there. An attacker plants it — this is prompt injection that never resets, which OWASP tracks as ASI06. Or no attacker at all: a model that repeatedly conditions on its own past generations can stabilize a hallucination into a self-consistent, durable, wrong belief.
The fix — trust writes less than reads:
- Scope per user and namespace so one tenant's (or one bad turn's) memory can't contaminate another's.
- Validate before persisting. For high-stakes facts, prefer verbatim source chunks with attribution over free-form extracted claims the model wrote.
- Supersede, don't append, so a corrected fact replaces the poisoned one instead of arguing with it.
- Reconcile contradictions periodically as a background pass.
The through-line: build the delete path#
All four failures share a root: teams build the write path with care and treat the delete path as an afterthought. An agent-memory system that only ever writes will rot — it grows unbounded, retrieves stale, never forgets, and can't shed a poisoned fact. Design forgetting with the same rigor you design storage, and memory stops being the thing that breaks your agent in week three.
If you're still choosing where the layers live, start with the three kinds of agent memory and the backend comparison; if you want to trust the benchmark numbers behind any of these claims, read how to read an agent-memory benchmark first.
