BM25 vs vector search: why local RAG needs both
BM25 finds exact terms, vectors find meaning, and on a local RAG most cannot-find-it bugs are a missing lexical half. When each wins, and how to fuse them.
Most local RAG tutorials hand you an embedding model and a vector database and call it search. Then you point it at your own files and it misses things a plain keyword grep would have caught: an exact error code, a client's name, a function you know is in there. The reason is almost never the embedding model. It is that the setup threw away the lexical half of the problem. Searching your own documents wants two different tools, BM25 for the words that have to match exactly and vectors for the meaning when the words do not line up, and the real engineering is knowing which one is carrying a given query.
We build a local-first assistant at recal, so on-device retrieval is a wall we hit and fixed rather than one we read about. The short version of what we learned: the lexical half does far more work than the tutorials imply, and the moment you drop it you lose a whole class of searches without any error to tell you why.
Key takeaways
- BM25 (keyword, lexical) and vector search (semantic) fail in opposite ways, so a local RAG running only one silently misses a whole class of queries.
- Most "vector search cannot find it" reports are missing-lexical-half problems: the query hinges on an exact term that embeddings blur.
- Vectors earn their keep on vocabulary mismatch, when the user's words and the document's words differ but mean the same thing.
- Hybrid search runs both and merges the two rankings; reciprocal rank fusion (RRF) is the boring default that works without tuning score scales.
- On device this is cheap. BM25 is sub-millisecond over something like SQLite FTS5, and the vector half is the part that costs you, mostly at write time, not query time.
What is hybrid search, and what does it combine?
Hybrid search combines two retrievers that look for different things. The lexical retriever, usually BM25, scores documents by the query terms they literally contain, weighted so that rare words count more and long documents do not win just for being long. The dense retriever embeds the query and each document into vectors and ranks by cosine similarity, so it can match on meaning even when no words overlap. Hybrid search runs both, then fuses the two ranked lists into one. That is the whole idea: not a smarter single model, but two cheap models with opposite blind spots covering for each other.
BM25 vs vector search: how each one fails
BM25 is a lexical, bag-of-words method. It is excellent at exactly the things embeddings are bad at: rare tokens, identifiers, error strings, product SKUs, a surname, a flag like --no-cache. If the term is in the document, BM25 finds it and ranks it near the top. Where it falls down is paraphrase. Ask "how do I stop it saving to disk" when the doc says "disable persistence" and BM25 has nothing to match on, because it does not know those phrases are the same.
Vector search is the mirror image. Embeddings capture meaning, so the paraphrase query lands on the right passage even with zero shared words. That is the magic everyone demos. The failure mode is precision. Embeddings blur exact terminology, so a rare identifier gets pulled toward its neighbors in vector space and a confident, semantically-nearby but wrong passage outranks the one that literally contains your term. On a personal or technical corpus, where the thing you are searching for is often a specific name or token, that blur is the difference between finding the file and not.
Is BM25 semantic search or keyword search?
BM25 is keyword (lexical) search, not semantic search. It has no model of meaning; it counts term matches with a clever weighting (term frequency saturation plus a length penalty, the descendants of TF-IDF). People sometimes call good BM25 results "semantic" because the ranking feels smart, but there is no embedding involved. That distinction matters when you design a stack: BM25 gives you exactness for free and knows nothing about synonyms, so if synonyms matter you add a semantic retriever rather than expecting BM25 to grow one.
When should you use hybrid search, and when is BM25 enough?
Honestly, more often than the vector-first tutorials suggest, BM25 alone is enough. For documentation search, a code or CLI "grep but ranked" tool, or a keyword router in front of an agent, lexical retrieval gets you most of the way with almost no infrastructure and no model download. Reach for hybrid when vocabulary mismatch is a real part of your queries: consumer-facing wording over formal documents, notes written by different people over time, or a corpus where users ask in plain language about text written in jargon. The rule we settled on is to make BM25 the spine and add vectors as the fallback for the lay-to-jargon gap, not the other way around. Starting vector-first and bolting keywords on later is how you end up debugging why an exact-match query fails.
What is reciprocal rank fusion (RRF)?
Once you have two ranked lists, you have to merge them, and the naive move (add the BM25 score to the cosine score) breaks because the two scores live on different, unbounded scales. Reciprocal rank fusion sidesteps that entirely. It ignores the raw scores and uses only the rank a document reached in each list, scoring it as the sum of 1 / (k + rank) across retrievers, with a small constant k (60 is the common default). A document that ranks near the top in either list gets a strong combined score, and one that both retrievers like rises above either alone. RRF wins as a default because it needs no score normalization and no per-corpus weight tuning; you can always add a weighted or learned fusion later if you measure a reason to.
Does ChromaDB, pgvector, or Qdrant support hybrid search?
It varies, and this is worth checking against current docs because the space moves fast. As of 2026, Qdrant and Weaviate support hybrid (sparse plus dense) natively, with a built-in fusion step. pgvector is dense-only by design, so hybrid on Postgres means pairing it with Postgres full-text search (tsvector/tsquery) and fusing the two yourself. Chroma has been adding full-text and filtering, but a full BM25 hybrid there has typically meant bringing your own lexical index. On device, you do not need any of them: SQLite ships FTS5 for BM25, and a vector extension or a small companion index covers the dense side, which is the shape we use so that search keeps working with no network and no service to run.
The three approaches at a glance
| Approach | What it matches | Strong at | Weak at | On-device cost |
|---|---|---|---|---|
| BM25 (lexical) | Exact query terms | Rare tokens, IDs, names, code | Paraphrase, synonyms | Tiny; sub-ms over FTS5 |
| Vector (dense) | Meaning of the query | Vocabulary mismatch, intent | Exact rare terms, precision | Embedding at write time |
| Hybrid (BM25 + vector, RRF) | Both, fused by rank | Broad coverage, fewer blind spots | A little more plumbing | Both, dominated by write-time embedding |
Frequently asked questions
BM25 vs vector search: which is better for RAG?
Neither on its own for a general corpus. BM25 is better when queries hinge on exact terms, identifiers, or code, and vectors are better when queries are paraphrases of what the documents actually say. For most real RAG, the better answer is hybrid, because your queries are a mix and the two methods fail in opposite places. If you must pick one to start, start with BM25, since it is cheaper and its failures are easier to reason about.
Is BM25 still relevant in 2026?
Yes, arguably more than ever. As people run RAG over private and technical corpora where exact terms matter, the lexical half has become the part that keeps retrieval honest. Modern hybrid pipelines lean on BM25 for precision and use embeddings to cover paraphrase; the strong result is the combination, not either alone.
What does hybrid search combine?
A lexical retriever (BM25 or another keyword index) and a dense retriever (vector embeddings), merged into one ranking. The lexical side guarantees exact-term matches surface; the dense side catches meaning when the words differ. Reciprocal rank fusion is the usual, tuning-free way to merge them.
Does hybrid search need a GPU?
No. BM25 needs no GPU at all. The vector side needs to compute embeddings, which is the only heavy step, and on a modern Mac a small local embedding model runs that comfortably on the CPU or the neural engine. The expensive moment is indexing (embedding your documents once), not the query.
Where recal sits
recal is a local-first personal assistant, and search over your own machine is a core part of it, so we ship hybrid retrieval on device rather than as a cloud call. The design follows the same lesson as the rest of this post: BM25 over a local full-text index is the spine, because so much of what you look for on your own machine is an exact name, path, or token, and the vector half is the fallback for when your wording and the document's wording drift apart. Keeping both local means search works offline and nothing leaves your machine to answer a query. If you want the wider landscape, our roundup of the best local RAG apps for Mac covers the tools, and what poisoning a RAG store taught us about agent memory goes deep on the retrieval-versus-authority split that matters once retrieval starts driving actions.
Written by the recal team with AI assistance and human review. Library and database behaviors described here reflect each project's public documentation as of July 2026; this space moves quickly, so verify current support before you commit. We build a local-first assistant, and we called out our own bias where it exists.