Short answer: Cache the output of a tool call keyed on the tool name plus its normalized arguments, give each cacheable tool its own short TTL, and only ever do it for tools you've explicitly marked idempotent and read-only. Writes, side effects, and volatile or user-specific data either bypass the cache or fold identity into the key.

An agent loop is a re-run machine. The model calls web_search("current mcp spec"), reasons, calls it again three steps later, then the user re-asks tomorrow and it runs a fourth time. Each call is latency you pay, money you spend, and rate-limit budget you burn — for an answer you already had. The fix is memoization: remember what a tool returned for a given input and hand it back instead of calling again.

This is not prompt caching#

The tool loop works like this: the model emits a tool call with a name and an input object, your code runs it, and you feed the result back. Provider-side prompt/KV caching reuses model tokens for a repeated prompt prefix — it lowers the input cost of a single model call. That is a different layer. Here you are caching the result of your tool — the search hits, the API body, the rows — so the tool itself never executes twice. The two stack cleanly. If you also want to shrink what the tool definitions cost every turn, that's the token layer's job: cache agent tool definitions to cut token cost.

Build the key from name + normalized arguments#

A cache is only as good as its key. The same call must always produce the same key, and semantically identical calls should too. Two rules:

  1. Include the tool name. get_user(42) and delete_user(42) share arguments but not meaning.
  2. Serialize arguments canonically. JSON with sorted keys so {"a":1,"b":2} and {"b":2,"a":1} hash to one entry. This is the same principle behind an HTTP cache key, which is built from the method plus the target URI.

For free-text arguments (search queries), normalize before hashing — trim whitespace, lowercase — so "MCP spec" and " mcp spec " don't become two misses. Don't over-normalize; stripping meaningful tokens turns a hit into a wrong answer.

Cache by whitelist, never by blacklist#

The cache cannot tell whether a tool is safe to reuse — you must. Do it as an opt-in whitelist: a tool is cached only if you've registered it as idempotent and read-only, with its own TTL. Everything else — every unknown tool, every write — bypasses. A blacklist fails open: the day someone adds charge_card and forgets to list it, you cache a payment.

The line matches HTTP's own vocabulary in RFC 9110: safe methods (GET, HEAD) don't change server state and are cacheable by default; idempotent methods produce the same state whether called once or many times. Your read-only tools are the GETs of your agent. Your writes are the POSTs.

A minimal, correct wrapper#

TTL is checked at read time, not write time — store the timestamp with the value and compare age on every get, exactly as Redis evaluates key expiry.

import json, time, hashlib
from typing import Any, Callable

# Whitelist: only these tools are cacheable, each with its own TTL (seconds).
# A tool NOT in this dict is never cached — writes and unknowns fail closed.
CACHEABLE_TTL = {
    "web_search":     300,   # 5 min  — retrieval, key on the normalized query
    "get_weather":    600,   # 10 min — idempotent read
    "get_user_prefs":  60,   # 1 min  — user-scoped: user_id lives IN args
}

def cache_key(tool_name: str, args: dict) -> str:
    # Canonical JSON: sorted keys, so argument order never splits a hit.
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(f"{tool_name}\x00{canonical}".encode()).hexdigest()

class ToolCache:
    def __init__(self) -> None:
        self._store: dict[str, tuple[float, Any]] = {}

    def get(self, key: str, ttl: float) -> Any | None:
        hit = self._store.get(key)
        if hit is None:
            return None
        stored_at, value = hit
        if time.monotonic() - stored_at > ttl:   # TTL checked on read
            del self._store[key]
            return None
        return value

    def set(self, key: str, value: Any) -> None:
        self._store[key] = (time.monotonic(), value)

def run_tool(name: str, args: dict,
             execute: Callable[[str, dict], Any],
             cache: ToolCache) -> Any:
    ttl = CACHEABLE_TTL.get(name)
    if ttl is None:                 # not whitelisted -> run it, never cache
        return execute(name, args)
    key = cache_key(name, args)
    cached = cache.get(key, ttl)
    if cached is not None:
        return cached               # cache hit: the tool does not run
    result = execute(name, args)
    cache.set(key, result)
    return result

Swap the in-process dict for Redis in production and TTL becomes the platform's job: SET key value EX ttl and let Redis expire it — no read-time age math, and the cache survives across sessions and processes. The key-building logic stays identical.

For user-scoped reads, the user id must be part of the key. The clean way is what the whitelist above assumes: the id is already an argument (get_user_prefs(user_id=...)), so it's inside the canonical JSON automatically. If your framework passes identity out-of-band, fold it in explicitly — a cache keyed on args alone will serve one user another user's data.

When not to cache#

Four hazards, in rough order of how badly they bite:

The decision collapses to three questions, and a "no" to any one means don't cache: is the call safe (no state change), is the result stable for the TTL window, and is it stateless across users (or is identity in the key)?

Measure before you trust it#

A cache that returns stale or cross-user data is worse than no cache — it fails silently. Log hit rate per tool and spot-check that hits carry the right TTL, and while you're accounting for what each tool costs you, it's worth learning to measure MCP tool context cost so you know which repeated calls were worth catching in the first place. Start with the boring wins — idempotent GETs and repeated searches inside one loop — and expand the whitelist only when you can name the TTL out loud.