Best Local Reranker Models for RAG in 2026

A reranker is the cheapest large win available to most local RAG stacks, and also the easiest thing to bolt on for no benefit at all. It sits after retrieval, takes the handful of chunks your vector search returned, and reorders them by actually reading each one against the query. Done right it lifts answer quality more than upgrading your embedding model. Done wrong it burns latency reordering a candidate list that never contained the right answer in the first place.

We run retrieval fully on-device building recal, a local-first assistant, so the picks below come from living with these models rather than reading a leaderboard. None of them is recal; recal is not a reranker. The comparison is on the merits, and the numbers are from each model's own Hugging Face page, checked on 1 August 2026.

Key takeaways

• For most local RAG in 2026, start with bge-reranker-v2-m3. It is multilingual, permissively licensed, and by far the most used open reranker at roughly 18.9 million downloads.
• If you need speed above all and your content is English, ms-marco-MiniLM-L6-v2 is still the workhorse at 22.7M parameters and roughly 86.8 million downloads.
• If you want the current accuracy ceiling and have the memory, Qwen3-Reranker at 4B or 8B is the strongest open option.
• Check the license before you ship. jina-reranker-v2 is released under CC BY-NC 4.0, which means no commercial use, and that is easy to miss when a model is otherwise excellent.
• A reranker reorders; it cannot retrieve. If the right chunk is not in your candidate pool, no reranker will find it, so measure pool recall before you tune ranking.

What is a reranker, and how is it different from an embedding model?

An embedding model encodes your query and your documents separately, ahead of time, into vectors you compare with cosine similarity. That separation is what makes vector search fast: the document side is precomputed. A reranker is a cross-encoder, which means it reads the query and one document together in a single pass and outputs a relevance score for that specific pair. Because it sees both sides at once, it can pick up on the kind of relationship a dot product between two independently made vectors cannot represent.

The cost is that nothing can be precomputed. Scoring 50 candidates means 50 forward passes at query time, so a reranker is orders of magnitude slower per document than a vector lookup. That is exactly why the standard shape is retrieve wide and cheap, then rerank narrow and expensive: pull 50 to 100 candidates with vector or hybrid search, then rerank down to the 3 to 5 you actually put in the prompt.

Do I actually need a reranker?

Usually yes, and it is normally a better investment than a larger embedding model. But there is one failure mode worth naming first, because it is the reason people conclude rerankers "do nothing" and rip them out.

A reranker can only reorder the candidates retrieval already handed it. If your top-50 pool does not contain the chunk that answers the question, reranking is rearranging deck chairs. So before you tune ranking, measure the thing underneath it: for a set of real questions where you know the right chunk, check how often that chunk appears anywhere in the candidate pool. That number is your ceiling. If pool recall is 60 percent, a perfect reranker still fails 40 percent of the time, and the fix is retrieval (better chunking, hybrid search, a wider k) rather than a better cross-encoder.

The corollary is happier: if pool recall is high but answers are still mediocre, a reranker is very likely the highest-value thing you can add, and a small one will do.

The dashed line in that diagram is the part people skip. It is set by retrieval, and everything the reranker does happens underneath it.

The best local reranker models in 2026

bge-reranker-v2-m3 (the sensible default)

The model most local stacks should reach for first. At 568M parameters it is small enough to run comfortably on a laptop, it is multilingual rather than English-only, and it is Apache 2.0, so there are no commercial surprises. Its adoption is the strongest signal available: roughly 18.9 million downloads and 1,118 likes on Hugging Face, which makes it the most used open reranker by a wide margin, with correspondingly good tooling support and plenty of people who have hit your problem before you. The honest catch is latency. It is roughly 25 times the parameter count of the MiniLM baseline, so on CPU-only hardware you will feel it once your candidate pool gets wide. Keep k modest, around 20 to 50, and it stays quick enough for interactive use.

ms-marco-MiniLM-L6-v2 (fastest, English only)

The classic cross-encoder baseline and still the right answer when latency dominates. At 22.7M parameters it is tiny, it runs on any CPU, and at roughly 86.8 million downloads it is the most downloaded model on this list by some distance, which tells you how many pipelines quietly depend on it. Apache 2.0. The limitation is in the name: it was trained on MS MARCO, an English passage-ranking dataset, so it is English-only in practice and will not help on a multilingual corpus. It is also an older architecture than everything else here, so on hard queries the newer models beat it on quality. Reach for it when you are reranking a wide pool on modest hardware, or when a heavier model has pushed your query latency somewhere users notice.

Qwen3-Reranker (the accuracy ceiling)

Released in three sizes, 0.6B, 4B and 8B, all Apache 2.0. These are LLM-style rerankers rather than the older encoder architecture, and they support instruction-style inputs, so you can describe the task rather than relying on the model's fixed notion of relevance. Adoption is already substantial for a newer family: roughly 2.74 million downloads for the 0.6B and 2.64 million for the 4B. The 0.6B is the interesting one for local use, since at 596M parameters it is comparable in size to bge-reranker-v2-m3 while bringing the newer approach. The 4B and 8B are strongest on quality but want a GPU or generous unified memory, and quantization is close to mandatory before they are pleasant on a laptop. Start at 0.6B and only move up if you can measure the difference on your own documents.

mxbai-rerank-base-v2 (a solid Apache-2.0 middle)

Mixedbread's v2 reranker sits between the tiny baseline and the heavier multilingual models at 494M parameters, under Apache 2.0. Downloads are more modest at roughly 142,000, which reflects how much newer it is rather than a quality verdict, though it does mean fewer community answers when something breaks. It is a reasonable pick if you want something current and permissively licensed without committing to the Qwen family, and Mixedbread publishes a larger variant if the base size underdelivers on your corpus. As with everything on this list, the useful test is not the leaderboard but a hundred of your own queries.

jina-reranker-v2-base-multilingual (strong, but check the license)

Technically one of the nicer options: 278M parameters, multilingual, fast for its class, and with a good reputation for long inputs. Roughly 886,000 downloads. The catch is not technical at all. It ships under CC BY-NC 4.0, a non-commercial license, so you cannot use it in a product you sell without a separate commercial agreement with Jina. That is easy to miss when you are benchmarking on quality alone, and it is genuinely painful to discover after you have built around it. Fine for research, personal projects, and internal evaluation. If you are shipping something commercial, use an Apache-2.0 or MIT model instead, or buy the license deliberately.

bge-reranker-base and bge-reranker-v2-gemma (the rest of the BAAI family)

bge-reranker-base is the older, smaller sibling at 278M parameters under MIT, with roughly 4.7 million downloads. It is still perfectly usable and its MIT license is the most permissive here, but v2-m3 supersedes it for most purposes. At the other end, bge-reranker-v2-gemma is a 2.5B-parameter LLM-based reranker, Apache 2.0, aimed at multilingual and English tasks where you want more capability than v2-m3 can give. Downloads are much lower at roughly 32,000, which suits its niche: reach for it when you have measured that v2-m3 is your bottleneck and you have the hardware to spend.

Local reranker models compared

Model	Params	License	HF downloads	Best for
ms-marco-MiniLM-L6-v2	22.7M	Apache 2.0	~86.8M	Speed, English, wide pools
bge-reranker-base	278M	MIT	~4.7M	Permissive license, light multilingual
jina-reranker-v2-base-multilingual	278M	CC BY-NC 4.0	~886K	Non-commercial use only
mxbai-rerank-base-v2	494M	Apache 2.0	~142K	Current, permissive middle ground
bge-reranker-v2-m3	568M	Apache 2.0	~18.9M	The default for most local RAG
Qwen3-Reranker-0.6B	596M	Apache 2.0	~2.74M	Newer approach at laptop size
bge-reranker-v2-gemma	2.5B	Apache 2.0	~32K	Heavier multilingual capability
Qwen3-Reranker-4B	4.0B	Apache 2.0	~2.64M	Accuracy with a GPU budget
Qwen3-Reranker-8B	8.2B	Apache 2.0	~567K	Maximum open accuracy

Parameter counts are from each model's safetensors metadata; download figures are Hugging Face lifetime totals read on 1 August 2026 and will drift.

Reranker versus BM25: which should you add first?

They solve different problems, and the order matters. BM25 is a retrieval method: it widens or changes what lands in your candidate pool by matching on terms rather than semantics. A reranker is a ranking method: it reorders whatever is already in the pool. If your failure is that the right chunk never gets retrieved, which is common when queries contain exact identifiers, error codes, or rare product names that embeddings smear together, BM25 alongside vector search fixes that and a reranker cannot. We wrote about that split in BM25 vs vector search.

If your pool is good and the ordering is bad, the reranker is the fix. In practice most mature local stacks end up with both: hybrid retrieval to build a pool with high recall, then a cross-encoder to sort it. Add whichever addresses your measured failure, not whichever is more interesting to build.

How to run a reranker locally

The lightest path is sentence-transformers, which exposes cross-encoders in a few lines and covers the MiniLM and BGE families directly. For a quantized setup, several of these models are published in GGUF, so they run through llama.cpp-based tooling alongside your generation model, which matters if you are already managing one runtime and do not want a second. On Apple silicon the same architecture choice applies as it does for generation, and we covered that trade-off in MLX vs llama.cpp.

Whatever you use, wire the measurement in before you tune. Log the candidate pool and the post-rerank order for a fixed set of real queries, and diff them when you change a model. Reranking is unusually easy to evaluate compared to the rest of a RAG pipeline, because you can see directly whether the right chunk moved up.

FAQ

What is the best local reranker model? For most people, bge-reranker-v2-m3: multilingual, 568M parameters, Apache 2.0, and the most widely used open reranker at roughly 18.9 million downloads. Choose ms-marco-MiniLM-L6-v2 instead if latency dominates and your content is English, or Qwen3-Reranker-4B or 8B if you want maximum accuracy and have the hardware.

Is a reranker better than a bigger embedding model? Usually, and for less money. Adding a small cross-encoder over your top-k results typically improves answer quality more than moving to a larger embedder, because it fixes ordering rather than representation. See our local embedding model guide for the other half of that decision.

Can I run a reranker with Ollama? Several of these models are available in GGUF, which lets llama.cpp-based tooling load them. Support for reranking endpoints varies by runner and version, so check your runner's current model library and API rather than assuming, then point your pipeline at the local endpoint.

How many candidates should I rerank? Start at 50 and tune down. Wider pools raise your recall ceiling but cost linearly, since every candidate is a forward pass. If latency hurts, cut k before you cut model quality, and check whether hybrid retrieval lets you hit the same recall with a smaller pool.

Do rerankers work offline? Yes. Every model here downloads once and runs fully on your own machine with no network call at query time, which is the whole point if you are building something local-first. The API-only options, Cohere Rerank among them, are the exception and send your query and documents to a vendor.

Which local rerankers are safe to use commercially? The Apache 2.0 and MIT models on this list: MiniLM, the BGE family, mxbai and Qwen3-Reranker. jina-reranker-v2-base-multilingual is CC BY-NC 4.0 and is not free for commercial use. Always read the current model card, since licenses do change between versions.

---

Written with AI assistance and reviewed by the recal team. We build recal, a local-first assistant for macOS that keeps your data on your machine, so retrieval that runs fully on-device is something we use and care about. The model comparisons above are on the merits and independent of recal. Parameter counts, download totals and licenses were read from Hugging Face on 1 August 2026 and change over time, so verify current numbers on each model's card before committing.