Ask a vector database to find the string auth_token= inside ten million log lines and you will discover a strange hole in the toolbox. The obvious tool, full-text search, comes up empty — not slow, empty. And the tool that would work, LIKE '%auth_token=%', reads every row because nothing indexes it. LanceDB's July release fills that hole with a primitive most people last saw in a bioinformatics class: the FM-Index.

Full-text search is word search, and that is the whole problem#

The instinct, when you want to filter text, is to reach for BM25 or a full-text index. That instinct is wrong here, and it is worth being precise about why, because the failure is structural, not a tuning problem.

A full-text index tokenizes. It runs your text through an analyzer that splits it into terms — words, roughly — lowercases them, maybe stems them, and builds a posting list from tokens. When you query, it matches tokens and ranks documents by relevance. This is exactly what you want for "find me documents about retrieval augmentation." It is exactly what you do not want for foo.bar.baz, 9f8c1e2a, /var/log/agent/run-4417.jsonl, or TypeError: cannot read. None of those are words. The fragment you are hunting lives inside a token, and a token index cannot see inside a token. No analyzer, no fuzzy setting, no AND/OR rescues it — the information was thrown away at index time.

A tokenizer's job is to forget where words end. Substring search is the one query that needs to remember.

So you fall back to contains() / LIKE '%needle%', and it works, and it scans the entire column, because a substring predicate has no index to stand on. Correct and unusable at scale — the worst quadrant.

What LanceDB shipped#

In LanceDB 0.34.0 (Python; Node/Rust 0.31.0), landed via PR #3532, contains(col, 'needle') gets a real index behind it: the FM-Index, a scalar index for Utf8, LargeUtf8, Binary, and LargeBinary columns. You build it like any other scalar index — lancedb.index.Fm() in Python, Index.fm() in TypeScript — and containment queries stop scanning. (The same 0.34.0 release also shipped table branches, which make RAG evals reproducible by turning the corpus into a git-style repo — a different feature aimed at a different problem, but the same "vector store is becoming versioned data infrastructure" arc.)

"FM-index" is not a LanceDB coinage. It is a compressed full-text index built on the Burrows-Wheeler transform — the same family of suffix-style structures that lets a genome aligner find a short read anywhere in three billion base pairs without a linear scan. It indexes the raw byte sequence, not tokens. That single design choice is why it can match an arbitrary substring: there is no word boundary in its worldview, so a fragment mid-token, across punctuation, or spanning whitespace is just… a sequence of bytes it already knows the positions of.

The reframe: it is a scalar index, not a vector index#

Here is the part that is easy to skate past. FM-Index is not a new kind of vector search. It sits in the scalar index family, next to BTREE (ranges), BITMAP (low-cardinality equality), and LABEL_LIST (array membership). It has nothing to do with embeddings.

That placement is the actual story. A "vector database" in 2026 is no longer a thing that only does approximate nearest neighbors; it is a multi-index retrieval engine where the vector index is one column type among several. And for the agents doing the most retrieval right now — coding assistants grepping a repo, ops agents grepping logs and traces — the query that decides whether the retrieval is usable is frequently the exact-match filter, not the semantic one. "Find code near this meaning" gets you a shortlist; "and it must literally contain SIGKILL / this commit sha / this request id" is what pins the answer. Before this, that second clause was either a slow scan or a lossy hybrid-search hack. Now it is an index.

Where it belongs, and where it doesn't#

Do not index every text column with it. A suffix-style structure costs more to build and store than a BTREE, and it earns that cost only on columns you actually infix-search. The high-value targets are obvious once you look for them: source columns in a code-retrieval store, message/stack-trace columns in a log table, id and path columns where you match prefixes and fragments. Keep learned-sparse and dense retrieval for meaning, keep FTS for prose, and reach for FM-Index for the one query they both quietly fail: does this exact fragment appear, anywhere, and can you find it without reading the whole table.

It is a small feature. It is also the difference between an agent that can grep its own memory and one that pretends LIKE '%' will scale. If you build on LanceDB, add it to the columns your agents actually search by fragment — and delete the full-table scan you have been apologizing for.