For two years the standard RAG diagram had one arrow into the retriever and one arrow out: a query goes in, a top-k blob of chunks comes back, and the model writes its answer over whatever it was handed. The retriever ran once, blind, before the model had thought about anything. If it grabbed the wrong five chunks, the model was stuck defending them.
LlamaIndex's new legal-kb reference app — a working TanStack Start web app, not a library — quietly deletes that arrow. Instead of a search() function, the agent gets a small toolbox that will look familiar to anyone who has watched a coding agent work: retrieve (hybrid semantic search), findFiles, readFile, and grepFile. LlamaIndex calls the pattern a Retrieval Harness, and the framing is the whole point. Retrieval stops being a pipeline stage that happens to the model and becomes a set of primitives the model drives.
The knowledge base is a filesystem now#
Watch how legal-kb answers a real question. Ask "What notice is needed to terminate the MSA?" and the agent doesn't fire one vector search and pray. It lists the files, runs retrieve to find the candidate documents semantically, then greps for the exact termination clause and reads the surrounding page. It answers with a citation to the specific page — and legal-kb renders that citation as a page screenshot with a bounding box drawn over the cited text.
A single-shot retriever gives you the five chunks it guessed you needed. A harness lets the model go find the one it can prove it needed.
That sequence — semantic locate, then lexical pin-point, then read in context — is exactly what a good engineer does in an unfamiliar repo, and exactly what coding agents converged on over the last year. The industry's most capable code agents largely stopped pre-indexing repositories into a vector store and started just running grep and read over the tree, because embeddings are lossy about the thing that matters most in a precise document: the exact string. "Terminate for convenience" and "terminate for cause" live in nearly the same embedding neighborhood and mean opposite things to a client who's about to get sued. grep doesn't care about neighborhoods. It cares about the literal clause.
LlamaIndex is now making the same argument about knowledge bases that the coding-agent world already made about codebases: the best retrieval interface for an agent is the one it can navigate, not the one that navigates for it.
What Index v2 is actually doing underneath#
The tools are the visible half. The other half is that they need something worth navigating. legal-kb runs on Index v2 — the LlamaParse Platform — which handles the unglamorous plumbing that makes filesystem-style tools viable over messy source documents: a persistent pipeline that parses uploads, indexes them, keeps them in sync, and maintains per-file version control. Upload a file and it's parsed and indexed in the background; the harness tools then read a clean, versioned, addressable corpus rather than raw PDFs.
That per-file versioning is the detail worth dwelling on. A grepFile tool is only trustworthy if "the file" has a stable identity across edits — otherwise a citation you return today points at a clause that moved yesterday. Treating the knowledge base like a filesystem forces you to treat it like a version-controlled filesystem, which is a real cost LlamaIndex has absorbed into the platform so the agent-facing tools can stay dumb and reliable.
The catch nobody puts on the slide#
There's an honest tension here, and it's worth naming before you go rip out your RAG pipeline. Single-shot retrieval is one call. A harness is a loop: findFiles, then retrieve, then two greps, then a readFile, then maybe another retrieve because the first pass missed. Every one of those is a round-trip and a chunk of context, which means agentic retrieval trades a fixed, cheap, predictable cost for a variable, larger, smarter one. On a straightforward lookup you'll spend more tokens and more latency to arrive at the same answer a plain top-k would have gotten. The pattern earns its keep on the hard questions — multi-clause, cross-document, "find the exception buried in the addendum" — where single-shot retrieval simply fails and no amount of reranking saves it.
So the real decision the Retrieval Harness forces isn't "vectors or grep." It's whether your workload has enough genuinely hard retrieval in it to justify letting the model spend tokens hunting. For a legal knowledge base, where the wrong clause is a liability and the right one is buried, the answer is obviously yes — which is exactly why LlamaIndex shipped the legal reference app first. For a FAQ bot, the answer is just as obviously no.
What to take from it#
You don't need LlamaParse to steal the idea. The transferable lesson from legal-kb is that "retrieval" is not one operation, and pretending it is — collapsing find, locate, and read into a single opaque search() — is what made classic RAG so brittle. Give your agent a hybrid retriever for semantic locate and a literal string search for exact pin-point and a read tool for context, and let it choose the order. The model is often a better retrieval planner than the fixed pipeline you'd have hand-written in front of it.
The single-arrow RAG diagram was always a simplification we tolerated because the models weren't good enough to do better. They are now. The retriever was never supposed to be the smart part.
