The one-line version: once your agent's memory store is big, the bottleneck moves from what to keep to what to surface for this prompt. Three signals compete for that call — relevance (vector similarity to the query), recency (how fresh the memory is), and importance (how durable or consequential it is) — and each one, used alone, fails in a specific, repeatable way. The working answer, straight out of the Generative Agents memory stream, is a weighted composite of all three, tuned to the failure you're actually hitting.

This is the read-side companion to how agents decide what to forget (the write-side) and to the architectural question of a memory layer vs. RAG. Here we're one level down: given a clean store, what gets loaded now.

Relevance: right topic, wrong time#

Vector relevance — cosine similarity between the query embedding and each memory embedding — is the default in Mem0 and nearly every memory layer, because it's the one signal that answers "is this memory about what the user just asked?" It's necessary. It is not sufficient.

Two failure modes show up immediately. First, staleness: if the user said "deploy to us-east" last month and "actually, eu-west" today, both memories score high against a deploy query, and similarity alone can hand the agent the superseded one. Second, context-blindness: relevance has no idea what the last three turns were about, so it can't prefer the fact that continues the current thread over an equally-similar one from an unrelated session.

Recency: last-said isn't most-important#

Recency fixes the time problem. Generative Agents models it as exponential decay — roughly 0.995 ^ (hours since the memory was last accessed) — so fresh memories float up. This is what keeps a conversation continuous: "what were we just doing?" pulls the right thing.

But recency alone inverts the failure. Now the last thing said outranks everything, and a load-bearing constraint the user gave an hour ago loses to a throwaway remark from thirty seconds ago. Recency is a great tie-breaker and a terrible sole ranker.

Relevance knows the topic but not the time. Recency knows the time but not the stakes. Importance knows the stakes but not the topic. No single signal is a ranker — the ranker is how you weigh all three.

Importance: protect the durable facts#

Importance (or salience) is the signal that stops durable facts from being buried under chatter. Generative Agents assigns it at write time: the model rates each memory's poignancy 1–10 and stores the score, so it costs nothing on every subsequent read. Zep encodes a related idea structurally — a fact is a graph edge with a validity window, so "still true and consequential" is a property of the edge, not a re-computation.

Alone, importance is static: it has no idea what this query is about, so a high-importance fact about billing surfaces even when the user is asking about theming. Useful as a floor, useless as the whole function.

The composite, in code#

Normalize each signal to [0,1] and take a weighted sum. That's the entire trick:

def score(mem, query_emb, now, w=(1.0, 1.0, 1.0)):
    w_rel, w_rec, w_imp = w
    relevance = cosine(query_emb, mem.embedding)          # 0..1
    recency   = 0.995 ** hours_since(mem.last_access, now) # 0..1, exp decay
    importance = mem.importance / 10.0                     # LLM-rated 1..10 at write
    return w_rel * relevance + w_rec * recency + w_imp * importance

top = sorted(store, key=lambda m: score(m, q_emb, now), reverse=True)[:N]

Generative Agents uses equal weights. In production you tune them to the failure in front of you: raise w_rel for factual-recall agents (support, coding assistants), raise w_rec for chat continuity, raise w_imp when durable facts keep getting buried. Keep each term normalized or the biggest raw number silently wins.

Retrieve wide, let the model choose#

The last mistake is over-tight ranking. The scorer's job is not to drop the memory the agent needs — not to crown a single winner. Pull a comfortable top-N (5–15 memories) by the composite and hand them all to the model; deciding which one actually applies to the turn is the model's strength, not the ranker's. If your retrieval returns exactly one memory and it's wrong, you didn't have a ranking problem, you had a recall problem — and the fix is a wider net scored by all three signals, not a cleverer single number.

Relevance to find the topic, recency to stay in the moment, importance to protect what matters. Weight them for your agent, retrieve a few, and let the model take it from there.