Reranking

Reranking follows a first-stage retrieval pass that quickly returns a broad candidate set from a vector index. A cross-encoder model then scores each candidate against the original query, and the top results replace the original ranking. This two-stage design balances speed and accuracy.

That second pass is where most quality gains come from in production search. First-stage retrieval favors speed; it returns plausible matches using fast approximate nearest-neighbor search. Reranking tolerates fewer candidates because it processes them with a model that reads the query and document together, capturing nuance that single-vector similarity misses.

Think of it like this. Think of a librarian who first pulls every book mentioning travel, then reads each candidate against your actual question and shelves the best matches at the front.

How it works

First-stage retrieval fetches a candidate set, typically twenty to a hundred documents, using fast vector search or BM25. Each query-document pair is fed to a cross-encoder that outputs a relevance score. Documents are reordered by score and the top results are returned.

Common mistakes

"Reranking replaces retrieval." It is a second stage that depends on solid first-stage recall. "Any cross-encoder works." Domain-specific rerankers outperform general ones on specialized content. "More candidates is better." Reranking cost scales linearly with candidate count.

Trade-offs

Improves ranking quality significantly, but adds latency and compute cost per query. Best used when first-stage recall is already acceptable but precision needs a lift, especially with diverse or ambiguous queries.